Tessera

MTEB

@mteb.org

News from MTEB: new features, models, results, datasets and benchmarks. Every number is recomputed from public results — check them yourself at leaderboard.mteb.org. Posts are auto-drafted from GitHub activity, but curated by maintainers.

3 Posts30 Follower 5 Folgt

MTEB @mteb.org · 22h ago

The best embedding score available at the end of 2023 needed a 304M-parameter model. Today the same score comes from an 11M one — 28x fewer active parameters.

The state of the field: Frontier quality keeps getting cheaper. Best score achievable at a given size, on English. Three Pareto frontiers of score against active parameters on MTEB(eng, v2), for models released by end-2023, by end-2024, and today, across 87 models with complete results. The best score available by end-2023 was 62.32 from sdadas/mmlw-e5-large at 304M active parameters. Today MongoDB/mdbr-leaf-mt matches it at 11M — 27.6 times fewer. Static embedding models appear at zero active parameters, behind an axis break. Source: MTEB, leaderboard.mteb.org.

MTEB @mteb.org · 3d ago

New on MTEB: hotchpotch/bekko-embedding-v1-a8m is mmBERT-small with its 22 layers pruned to 4 — active parameters cut from 42M to 7.7M, vocabulary left intact. It scores 56.73 on MTEB(Multilingual, v2), against 47.21 for a static model of the same total size.

New model on MTEB: Bekko — A new Pareto frontier for parameter-efficient multilingual embeddings. Scatter plot of MTEB(Multilingual, v2) against Active parameters for 6 multilingual models ≤ 60M active params. hotchpotch/bekko-embedding-v1-a25m scores 58.36 at 25M active parameters; hotchpotch/bekko-embedding-v1-a8m scores 56.73 at 8M active parameters. Compared against codefuse-ai/F2LLM-v2-80M at 32M, 55.23; ibm-granite/granite-embedding-97m-multilingual-r2 at 28M, 51.92; sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 22M, 48.73; sentence-transformers/static-similarity-mrl-multilingual-v1 at 0 (static), 47.21. On the Pareto frontier (nothing is both smaller and higher-scoring): sentence-transformers/static-similarity-mrl-multilingual-v1, hotchpotch/bekko-embedding-v1-a8m, hotchpotch/bekko-embedding-v1-a25m. Contributed by hotchpotch. Source: MTEB, leaderboard.mteb.org.