Updates to "A Metric Learning Reality Check
metric-learningbenchmarkingdeep-learningevaluationpytorch
Abstraction: Updates to arXiv paper exposing unfair comparisons in metric learning benchmarks
Key points:
- arXiv paper 2003.08505 "A Metric Learning Reality Check" by Kevin Musgrave; update adds concrete examples of unfair benchmark comparisons between metric learning methods
- Bayesian optimization plots added to supplementary material showing validation accuracy vs. hyperparameter surfaces
- MAP@R metric argued as superior to Recall@1 for evaluating metric learning (better accounts for the full retrieval ranking)
- Large-batch experiment on CUB200 (batch size 256) showed FastAP performance significantly improves and reaches parity with other methods at larger batch sizes
- Comprehensive "papers vs reality" figure updated to include more algorithms — showing gap between claimed vs. reproducible results
- Related tools: Powerful Benchmarker library and PyTorch Metric Learning library
Connections: Kevin Musgrave · Metric Learning · Benchmark Evaluation · Bayesian Optimization
Source: https://medium.com/@tkm45/updates-to-a-metric-learning-reality-check-730b6914dfe7