🤨"But why linguistics" is the most common question when talking about linguistic reasoning benchmarks. Last year we organized a shared task at WMT...and no one participated 🤣 🤯Let me change your mind why this is one of the most challenging, focused and best reasoning benchmarks right now.

Cohere Labs@cohereforai.bsky.social · last mo.

For the launch of the 1-month open challenge that precedes the in-person Olympiad, @juliakreutzer.bsky.social is hosting @danmirea.bsky.social & Eduardo Sanchez to discuss the appeal of these problems, solution challenges, and potential insights from AI-human evaluation. Join us: luma.com/jk8uv7zs

🔥Possibly the most fun and underrated AI challenge this summer: Compete on unseen linguistic reasoning problems and present your solutions to the expert jury in a month!

Cohere Labs@cohereforai.bsky.social · last mo.

We're excited to be sponsoring and co-organizing IOL-AI 2026, a new open challenge with the International Linguistics Olympiad. 🌎💬 🌐 Open-science 🗓️ One-month competition 🎯 Targeting AI reasoning with challenging, unseen test problems from the 2026 olympiad.

Join us tomorrow for a discussion around culture in AI! 🌱

Cohere Labs@cohereforai.bsky.social · 5mo ago

AI is getting better at math. Better at code. But is it getting better at understanding cultural nuances? 🤔 Join us for “Cultural Awareness in AI — From Knowledge Tests to Social Norms and Beyond”, a conversation on what it means to build AI systems that work at global scale.

💭We need more research that focuses on aspects beyond accuracy, especially in multilingual AI. 👉Help us explore the importance of culture in building and testing AI, with a few minutes of your time. Happy to have a chat as well with anyone who's interested in that space!

Cohere Labs@cohereforai.bsky.social · 5mo ago

Does AI truly understand different cultures and languages? We’re surveying cultural awareness in real-world AI use. ✨ When cultural awareness matters in real-world AI use 💡 Whether AI reflects diverse norms, communication styles & knowledge 🫥Where AI falls short in cultural understanding

🌱Very proud of our team's latest release 😊 meet Tiny Aya, a massively multilingual model with 3.35B parameters. Tech report here: github.com/Cohere-Labs/...

Cohere Labs@cohereforai.bsky.social · 6mo ago

Introducing ✨Tiny Aya✨, a family of massively multilingual small language models built to run where people actually are. Tiny Aya delivers strong multilingual performance in 70+ global languages in a 3.35B parameter model, efficient enough to run locally, even on a phone.

At #Neurips2025 this week with @cohereforai.bsky.social 🤩 This is what brought me into research: the ✨Network Effect ✨ Let's build the next breakthrough together! #LabLegends

We’re thrilled to announce that some of our research will be presented at @emnlpmeeting.bsky.social next week! 🥳 If you’re attending the conference, don’t miss the chance to explore our work and connect with our team.

How well do LLMs handle multilinguality? 🌍🤖 🔬We brought the rigor from Machine Translation evaluation to multilingual LLM benchmarking and organized the WMT25 Multilingual Instruction Shared Task spanning 30 languages and 5 subtasks.

🌍Most multilingual instruction data starts as English and translation can’t capture cultural nuance or linguistic richness What if we optimized prompts instead of completions? That’s the focus of our most recent work on prompt space optimization for multilingual synthetic data🗣️

The next generation of open LLMs should be inclusive, compliant, and multilingual by design. That’s why we @icepfl.bsky.social @ethz.ch @cscsch.bsky.social ) built Apertus.

EPFL School of Computer and Communication Sciences@icepfl.bsky.social · 11mo ago

EPFL, ETH Zurich & CSCS just released Apertus, Switzerland’s first fully open-source large language model. Trained on 15T tokens in 1,000+ languages, it’s built for transparency, responsibility & the public good. Read more: actu.epfl.ch/news/apertus...

Let's do the venue justice. Very excited for today's multilingual workshops at #COLM2025 💙

Cohere Labs@cohereforai.bsky.social · 10mo ago

Today at COLM, Cohere Labs Sr Research Scientist, @juliakreutzer.bsky.social will be presenting at 2 workshops. First, the Multilingual Data Quality Signals workshop, bringing together researchers across disciplines to discuss & present research on data quality signals in multilingual data.

Looking forward to tomorrow's #COLM2025 workshop on multilingual data quality! 🤩

Workshop on Multilingual Data Quality Signals@wmdqs.bsky.social · 10mo ago

In collaboration with @commoncrawl.bsky.social, MLCommons, and @eleutherai.bsky.social, the first edition of WMDQS at @colmweb.org starts tomorrow in Room 520A! We have an updated schedule on our website, including a list of all accepted papers.

Ready for our poster today at #COLM2025! 💭This paper has had an interesting journey, come find out and discuss with us! @swetaagrawal.bsky.social @kocmitom.bsky.social Side note: being a parent in research does have its perks, poster transportation solved ✅

Cohere Labs@cohereforai.bsky.social · 10mo ago

Today at COLM, we are excited to share our work Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation, during Poster Session 4, 4:30 - 6:30pm. Come connect with paper authors @juliakreutzer.bsky.social and @kocmitom.bsky.social.

We’re not your average lab. We’re a hybrid research environment dedicated to revolutionizing the ML space. And we’re hiring a Senior Research Scientist to co-create with us. If you believe in research as a shared, global effort — this is your chance.

💡A collaborative➕diverse team is key. In real life as in the LLM world 💪🦾 Check out our latest work that builds on this insight. 👇

Cohere Labs@cohereforai.bsky.social · 10mo ago

Is Best-of-N really the best use of your inference compute? Introducing Fusion-of-N: a simple and powerful way to advance inference and distillation beyond Best-of-N.

Breaking into AI research is harder than ever, and early-career researchers face fewer chances to get started. Entry points matter. We started the Scholars Program 3 years ago to give new researchers a real shot — excited to open applications for year 4✨

Cohere Labs@cohereforai.bsky.social · 12mo ago

Applications are now open for the next cohort of the Cohere Labs Scholars Program! 🌟 This is your chance to collaborate with some of the brightest minds in AI & chart new courses in ML research. Let's change the spaces breakthroughs happen. Apply by Aug 29.

While effective for chess♟️, Elo ratings struggle with LLM evaluation due to volatility and transitivity issues. New post in collaboration with AI Singapore explores why Elo falls short for AI leaderboards and how we can do better.

COLM 2025 is now accepting applications for: Financial Assistance Application -- docs.google.com/forms/d/e/1F... Volunteer Application -- docs.google.com/forms/d/e/1F... Childcare Financial Assistance Application -- docs.google.com/forms/d/e/1F... All due by July 31

COLM 2025 Financial Assistance Application

Goal of the Financial Assistance Program. We at COLM believe our community should be diverse and inclusive. We recognize that some might be less likely to attend because of financial burden of travel ...

docs.google.com

🍋 Squeezing the most of few samples - check out our LLMonade recipe for few-sample test-time scaling in multitask environments. Turns out that standard methods miss out on gains on non-English languages. We propose more robust alternatives. Very proud of this work that our scholar Ammar led! 🚀

Cohere Labs@cohereforai.bsky.social · last yr.

Can we improve the performance of LLMs during inference without the need for extensive sampling OR special reward models? 🤔 Our latest work introduces a new inference time scaling recipe that is sample-efficient, multilingual, and suitable for multi-task requirements. 🍋

🚨LLM safety research needs to be at least as multilingual as our models. What's the current stage and how to progress from here? This work led by @yongzx.bsky.social has answers! 👇

Cohere Labs@cohereforai.bsky.social · last yr.

It’s been two years since cross-lingual jailbreaks were first discovered. How far has the multilingual LLM safety research field advanced? 🤔 📏 Our comprehensive survey reveals that there is still a long way to go.

🚧No LLM safety without multilingual safety - what is missing to closing the language gap? And where does this gap actually originate from? Answers 👇

Cohere Labs@cohereforai.bsky.social · last yr.

Over 7000 languages are spoken worldwide 🌐, but AI safety efforts focus on only a fraction of them. Our latest paper draws on our multi-year efforts with the wider research community to explore why this matters and how we can bridge the AI language gap.

Multilingual 🤝reasoning 🤝 test-time scaling 🔥🔥🔥 New preprint! @yongzx.bsky.social has all the details 👇

Yong Zheng-Xin (Yong)@yongzx.bsky.social · last yr.

📣 New paper! We observe that reasoning language models finetuned only on English data are capable of zero-shot cross-lingual reasoning through a "quote-and-think" pattern. However, this does not mean they reason the same way across all languages or in new domains. [1/N]

1/ Science is only as strong as the benchmarks it relies on. So how fair—and scientifically rigorous—is today’s most widely used evaluation benchmark? We took a deep dive into Chatbot Arena to find out. 🧵

🤓MT eyes on multilingual LLM benchmarks 👉 Here's a bunch of simple techniques that we could adopt easily, and in total get a much richer understanding of where we are with multilingual LLMs. 🍬Bonus question: how can we spur research on evaluation of evaluations?

Cohere Labs@cohereforai.bsky.social · last yr.

🚀🌍The rapid advancement of multilingual large language models (mLLMs) is exciting, but are we evaluating them effectively? Our new paper explores how we can improve generative evaluations for mLLMs by learning from machine translation (MT) evaluation practices. 🔎

Tired of messy non-replicable multilingual LLM evaluation? So were we. In our new paper, we experimentally illustrate common eval. issues and present how structured evaluation design, transparent reporting, and meta-evaluation can help us to build stronger models.

Julia Kreutzer@juliakreutzer.bsky.social · last yr.

📖New preprint with Eleftheria Briakou @swetaagrawal.bsky.social @mziizm.bsky.social @kocmitom.bsky.social! arxiv.org/abs/2504.11829 🌍It reflects experiences from my personal research journey: coming from MT into multilingual LLM research I missed reliable evaluations and evaluation research…

📖New preprint with Eleftheria Briakou @swetaagrawal.bsky.social @mziizm.bsky.social @kocmitom.bsky.social! arxiv.org/abs/2504.11829 🌍It reflects experiences from my personal research journey: coming from MT into multilingual LLM research I missed reliable evaluations and evaluation research…

🚀 We are excited to introduce Kaleidoscope, the largest culturally-authentic exam benchmark. 📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation

☀️ Summer internship at Cohere! Are you excited about multilingual evaluation, human judgment, or meta-eval? Come help us explore how a rigorous eval really looks like while questioning the status quo in LLM evaluation. I’m looking for an intern (EU timezone preferred), are you interested? Ping me!

Command🅰️ technical report is out. Information-dense. Detailed. Pretty. Simply A+! 💎: cohere.com/research/pap...

Command A: An Enterprise-Ready Family of Large Language Models

In this report we describe the development of Command A, a powerful large language model purpose-built to excel at real-world enterprise use cases. Command

cohere.com

Max Bartolo@maxbartolo.bsky.social · last yr.

I'm excited to share the tech report for our @cohere.com @cohereforai.bsky.social Command A and Command R7B models. We highlight our novel approach to model training including self-refinement algorithms and model merging techniques at scale. Read more below! ⬇️

A bit of a mess around the conflict of COLM with the ARR (and to lesser degree ICML) reviews release. We feel this is creating a lot of pressure and uncertainty. So, we are pushing our deadlines: Abstracts due March 22 AoE (+48hr) Full papers due March 28 AoE (+24hr) Plz RT 🙏