Tessera
Folgen

@colab-links.bsky.social

48 Posts4 Follower2 Folgt
@colab-links.bsky.social · last wk.

Cohere is another known group known in making huge efforts with many people, seems like this is their latest: I cannot be translated. I cannot be web-searched. I only make sense if you grew up where I did. What am I? A good riddle.

Bild
@colab-links.bsky.social · last wk.

Multilingual cultural dataset 6 european languages https://arxiv.org/abs/2607.21445 many multiple choice hard cultural QA in 288 topics (3k questions) — LChoshen

When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics. We introduce TriviaRoomQA, a multilingual benchmark designed to evaluate everyday, culturally grounded, and long-tail knowledge across 288 topics. The benchmark contains 3,300 parallel multiple-choice questions in six European languages and additional 5,340 French-only questions for a more fine-grained case study. We evaluate 30 open-weight LLMs from European, Asian, and North American providers, covering models from 7 to 70B parameters. We find that models are strong on knowledge-intensive topics such as history, geography, and mathematics, but substantially weaker on everyday popular-culture topics such as celebrities, music, movies, and news. Moreover, model performan

arxiv.org

@colab-links.bsky.social · 2w ago

Agentic search harnesses (alphaEvolve and co) have many uncheched components which should be seen as a hyperparameter to tune, no one significantly beats the others (at least with 3m rollouts 😛 ) For you agents lovers bsky.app/…

Leshem (Legend) Choshen @EMNLP (@lchoshen.bsky.social)

OpenEvolve underperforms simple autmated discovery harnesses the rest are insignificant from each other. The best choice changed across model–problem pairs. We ran a controlled study (3m+ rollouts) using repeated budget-matched runs, strong baselines, and statistical hypothesis testing.

bsky.app

@colab-links.bsky.social · last mo.

I wish there was a good way to get all the papers here and not just the bottom lines from Micheal Frank's work. Anyway, he has a long list of cool works and this kinda summariezes a lot of the bottom lines. bsky.app/… — LChoshen

Leshem (Legend) Choshen @EMNLP (@lchoshen.bsky.social)

Livetweet @mcxfrank.bsky.social talk: If we care about intelligence and cognition, AI has recently allowed a change; we now have two ways to study them. AI is of course allowing us a lot that we wouldn't dare on our Children (brain surgery, never tell a child about cats...) #acl #conll

bsky.app

@colab-links.bsky.social · last mo.

https://arxiv.org/abs/2602.17655 — The Data Therapist

What Language is This? Ask Your Tokenizer

Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm, leveraging its probabilistic framing, parameter estimation technique and inference strategy. In short, to predict a string's language label, we simply ask: under which language's unigram distribution is this string most likely? Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fastText,

arxiv.org

@colab-links.bsky.social · last mo.

Might be relevant: https://arxiv.org/abs/2605.25360 — Uriel Dolev

Learning to Route Languages for Multilingual Policy Optimization

Large language models~(LLMs) are trained on heterogeneous multilingual corpora, yet existing policy optimization methods often implicitly restrict each training question to a single response language or rely on a fixed dominant language for supervision. We propose language-routed policy optimization (LRPO), an online policy optimization framework that treats language as a selectable variable. LRPO elicits multilingual rollouts for each training question and integrates their relative quality into preference-based policy updates, increasing the diversity and informativeness of training signals under the fixed rollout budget. To adaptively determine which languages to explore during reinforcement learning, we introduce a trainable language router formulated as a multi-armed bandit, balancing exploration of underutilized languages with exploitation of more informative ones. Extensive experiments show that LRPO consistently improves multilingual performance, demonstrating that adaptive lang

arxiv.org

@colab-links.bsky.social · last mo.

https://arxiv.org/abs/2604.07067 — LChoshen

Is Cross-Lingual Transfer in Bilingual Models Human-Like? A Study with Overlapping Word Forms in Dutch and English

Bilingual speakers show cross-lingual activation during reading, especially for words with shared surface form. Cognates (friends) typically lead to facilitation, whereas interlingual homographs (false friends) cause interference or no effect. We examine whether cross-lingual activation in bilingual language models mirrors these patterns. We train Dutch-English causal Transformers under four vocabulary-sharing conditions that manipulate whether (false) friends receive shared or language-specific embeddings. Using psycholinguistic stimuli from bilingual reading studies, we evaluate the models through surprisal and embedding similarity analyses. The models largely maintain language separation, and cross-lingual effects arise primarily when embeddings are shared. In these cases, both friends and false friends show facilitation relative to controls. Regression analyses reveal that these effects are mainly driven by frequency rather than consistency in form-meaning mapping. Only when just f

arxiv.org

@colab-links.bsky.social · last mo.

More multilingual than pretraining, but when we go multimodal useful https://arxiv.org/abs/2507.19634 — LChoshen

MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks

Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework. As these models are rapidly evolving toward general-purpose instruction following across diverse and complex tasks, a key frontier is evaluating their crosslingual and multimodal capabilities over both short- and long-form inputs. However, existing benchmarks fall short in evaluating these dimensions jointly: they are often limited to English, mostly focus on a single modality at a time, rely on short-form inputs, or lack human annotations--hindering comprehensive assessment of model performance across languages, modalities, and task complexity. To address these gaps, we introduce MCIF (Multimodal Crosslingual Instruction Following), the first crosslingual human-annotated benchmark based on scientific talks on NLP and beyond. MCIF evaluates instruction following in crosslingual, multimodal settings over different input lengths an

arxiv.org

@colab-links.bsky.social · last mo.

To make confidence estimation (calibration) in multilingual setting, they find a place to make a probe that transfers rather than fine-tune for each language. scholar.google.com/…

https://scholar.google.com/scholar_url?url=https://arxiv.org/pdf/2605.31220&hl=en&sa=X&d=2215136617550590432&ei=hiEiauu7Luq46rQP1fX7-As&scisig=ANDmEU5j-qpRQ8tmC88OzS1Ur-nx&oi=scholaralrt&hist=8b8IhUYAAAAJ:9281163533438476704:ANDmEU7vdlWyDCQpQ5uV7SBVlopI&html=&pos=0&folt=cit&fols=

scholar.google.com

@colab-links.bsky.social · last mo.

For future related work sections (e.g. <@761599518683955230> ) generalization not only across languages but modalities (vision surely have such issues as well, I definitely know they are really good at answering based on the text alone and not based on the image) and omni models?

Gallil Maimon (@GallilMaimon) on X

@LChoshen We saw cases of knowledge transfer (answering in speech facts only seen in text) when looking at interleaved speech-text LMs like the ones from our paper - https://t.co/tgzJ3X4hhC We are working now on understanding how this works, what is needed, how to boost:)

x.com

@colab-links.bsky.social · last mo.

https://arxiv.org/abs/2605.19284 — LChoshen

Language models struggle with compartmentalization

In the training data used by large language models (LLMs), the same latent concept is often presented in multiple distinct ways: the same facts appear in English and Swahili; many functions can be expressed in both Python and Haskell; we can express propositions in both formal and natural language. We show that LLMs can exhibit compartmentalization, where they fail to identify and share statistical strength between distinct presentations of unified concepts. In the worst case, LLMs simply learn parallel internal representations of each presentation of the concept, saturating model capacity with redundancies and decreasing sample efficiency with the number of such presentations. We also demonstrate that synthetic parallel data can fail to improve this despite being easily learned itself. Under this framework, we find that, for small models, early multilingual learning is nearly entirely compartmentalized. Finally, all interventions that we study exhibit a phase transition in which their

arxiv.org

@colab-links.bsky.social · last mo.

I think it’s worth looking at to try to ensure that our experiments will transfer to large scale: https://x.com/rosinality/status/2057370209434939792?s=46 <@165490503570292736> — Uriel Dolev

Rosinality (@rosinality) on X

https://t.co/LJg6GFTOV2 Validation of transformer modifications similar to https://t.co/r9BdFwpCfK with modern modifications. It is nice to find out bonferroni correction here. Maybe the setup that has been used here could have low statistical power, but at the same time each https://t.co/M6mQREWvBG

x.com