Very proud of my student Lukáš Eigler presenting this at the ACL Student Research Workshop. TL;DR: you can validate NLP evaluation metrics with synthetic LLM judgments — the rankings track human ones almost perfectly. 📄 aclanthology.org/2026.acl-srw.125

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Lukáš Eigler, Jindřich Libovický, David Hurych. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop). 2026.

aclanthology.org

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · 4w ago

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation aclanthology.org/2026.acl-srw... by Lukáš Eigler, @jlibovicky.bsky.social & David Hurych Rankings from synthetic LLM-generated data almost perfectly match real human judgments. 🤖⚖️ at Student Research Workshop

Dušan made something like TensorBoard for tokenization 📊 Token length, entropy, vocab overlap, JS divergence, alignment scores: sliced by language family, script, region, speakers, data availability. 🔗 Code: github.com/ufal/TokCollate 🔗 Demo: quest.ms.mff.cuni.cz/tokcollate

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · last mo.

Dušan Variš presents joint work w/ @abyste.bsky.social & @jlibovicky.bsky.social: TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages aclanthology.org/2026.acl-dem... A dashboard for your tokenizer 👀 See which languages you left behind. Open-source, MIT.

AC observation this ARR cycle: I'm getting way more reviews and way earlier, than usual. I suspect it's the draconian desk-reject threats working, and not a sign of better reviewing culture. I worry that this gets read as 'threats work, do more of them, discipline the lazy researchers'.

📣 TokShop 2026 deadline extended! 🗓️ New submission deadline: Friday, June 26, 2026 (AoE) Research papers (up to 9 pages) and extended abstracts (up to 2 pages) are welcome. Submit: openreview.net/group?id=col... More info: tokenization-workshop.github.io

Is "Je to obrovský problém, tvrdí české ministerstvo obrany." Czech or Slovak? Both. Language ID systems must pick one: and that's just one way they break. Our new benchmark CHALIS exposes how fragile they really are. 📄 arxiv.org/abs/2606.06088 🤗 huggingface.co/datasets/michal-tichy/CHALIS

CHALIS: A Challenge Dataset for Language Identification in Difficult Scenarios

We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic no...

arxiv.org

Lukáš Eigler defended his thesis (co-supervised with David Hurych, @valeoai.bsky.social) 🎉 Congrats! #NLP metric validation needs 🐌💰 human judgment data. Our fix: generate synthetic data for metric validation. ✅ Tested on MT, QA, summarization. To appear #ACL2026 SRW: arxiv.org/abs/2603.09403

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalabl...

arxiv.org

I have to grade 200+ ML exams/year. To survive, I coded a bunch of tools: unique exam generation, handwriting recognition, and per-question stats. Check out the details on my blog. 📊 jlibovicky.github.io/2026/06/04/G...

Send your work to TokShop! tokenization-workshop.github.io

Tokenization Workshop (TokShop) @COLM2026@tokshop.bsky.social · 3mo ago

Announcing First Call for Papers: Second Tokenization Workshop 🔡 📣 ▶️ Non-archival submissions of two types: Research papers (up to 9 pages) ▶️ Extended abstracts (up to 2 pages) Submission deadline June 23, 2026 (AoE) Acceptance notification on July 24, 2026 (AoE) tokenization-workshop.github.io

TokShop is happening again!

Tokenization Workshop (TokShop) @COLM2026@tokshop.bsky.social · 3mo ago

TokShop will be at #COLM2026! 🗓️ October 9th, 2026 📍 San Francisco, USA More details and a call for papers coming soon.

Nice work by my student @gianlucavico.bsky.social on a topic close to home: crowdsourcing Piedmontese to test LLMs on non-standard orthography. New dataset covering tokenization, classification & translation.

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · 4mo ago

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography by @gianlucavico.bsky.social and @jlibovicky.bsky.social aclanthology.org/2026.vardial... New Piedmontese dataset tests tokenization, classification & translation! 🗣️

On the Credibility of Evaluating LLMs using Survey Questions by @jlibovicky.bsky.social aclanthology.org/2026.mme-mai... Survey-based LLM value evals are unreliable — prompting & decoding choices skew results drastically.

So proud of my new PhD student Adnan Al Ali for presenting their master's thesis work at EACL! 🎓 A great contribution to understanding bias in AI text detectors across languages.

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · 4mo ago

Different Time, Different Language: Revisiting the Bias Against Non-Native Speakers in GPT Detectors by Adnan Al Ali, @jindrahelcl.bsky.social and @jlibovicky.bsky.social aclanthology.org/2026.eacl-sr... AI text detectors are suspected of biasing against non-native speakers, but not in Czech 🇨🇿

I'm at #EACL2026 in Rabat 🇲🇦. Find me and talk to me about tokenization or multilingual model eval. Also, check out our work on how to eval morphological plausibility of your tokenizer if you don't have gold segmentation data, but you happen to have morphosyntactic features 👇

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · 4mo ago

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features by @abyste.bsky.social & @jlibovicky.bsky.social aclanthology.org/2026.finding... TL;DR: Morpho-syntax features replace gold segmentation data for tokenization eval.

Thanks for organizing this! I am definitely coming, and everyone interested in tokenization at #EACL2025 should too. 🫵

Ilker Kesen@ilkerkesen.bsky.social · 5mo ago

📢I'm organizing a BoF session at #EACL2026 called Tokenization & Beyond, aiming to gather researchers exploring tokenization and alternatives such as byte-level and pixel-based approaches. Sign up using the form if you're interested! #NLProc @eaclmeeting.bsky.social

3rd run teaching ML to 250+ bachelor students (with great materials originaly by @straka-milan.bsky.social). Core philosophy: explain the math, implement algorithms from scratch, Kaggle-style competitions, all auto-graded. ufal.mff.cuni.cz/courses/npfl... But look what LLMs did to the course 👇

Spent time making AI-generated images of Bayes' Rule, Laplace Smoothing, Markov Chains & Shannon Entropy for class today 🎨🤖 Even though the images are objectively hilarious, none of the 50 students in the room laughed. Or even smiled. 💀

I reviewed papers evaluating LLM values using sociology questionnaires. Different methods, different results. Didn't trust them, so I tested it myself. Methodology matters. Short answers vs CoT, squared err vs KL div.: each changes which populations an LLM "aligns" with. www.arxiv.org/pdf/2602.04033

We have updated the pre-print on CUS-QA, benchmark for regional knowledge about Czechia, Slovakia and Ukraine arxiv.org/abs/2507.22752 Now, there are results of retrieval-augmented generation and more detailed analysis of model performance depending on the topic of the question or visual context.

CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset

We introduce CUS-QA, a benchmark for evaluation of open-ended regional question answering that encompasses both textual and visual modalities. We also provide strong baselines using state-of-the-art l...

arxiv.org

Jindřich Libovický@jlibovicky.bsky.social · 12mo ago

🧵 We're releasing CUS-QA - a new benchmark for testing LLMs on regional knowledge! Find out what your model knows about Czechia 🇨🇿, Slovakia 🇸🇰, and Ukraine 🇺🇦! 👉 Textual and visual questions, answers, and human judgment on model outputs! huggingface.co/datasets/ufa... www.arxiv.org/abs/2507.22752

We (= mostly @abyste.bsky.social) developed a way to evaluate how morphological a #tokenization is w/o gold segmentation labels. arxiv.org/abs/2601.18536 The key: align subword tokens with morphological features from UniMorph using IBM Model 1. To appear in EACL 2026 Findings.

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentati...

arxiv.org

Happy holidays! 🎄🎅🤩🎁

Attenzione! 🇮🇹 Know Piedmontese or Neapolitan speakers? @gianlucavico.bsky.social is collecting crowd-sourced translations to evaluate LLM performance on these regional languages. Partecipate!

Gianluca Vico@gianlucavico.bsky.social · 9mo ago

We’re collecting crowd-sourced translations in Piedmontese and Neapolitan. 🎯 Goal: see how well LLMs understand these languages. 👉 Participate here (in IT🇮🇹): - Piedmontese: quest.ms.mff.cuni.cz/crowd-transl... - Neapolitan: quest.ms.mff.cuni.cz/crowd-transl... Anyone can join, no need to be fluent!

With @andrei-a-manea.bsky.social, we posted a survey on multilingual vision-language models 👉 arxiv.org/pdf/2509.22123 We reviewed 31 models+21 benchmarks. There's a tension between language neutrality (same results across languages) & cultural awareness (context matters differently across cultures)

So proud of my PhD student @andrei-a-manea.bsky.social for his first first-author publication! 🎉 He presented this work last week at TSD. Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders arxiv.org/pdf/2504.21681

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · 11mo ago

Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders by @andrei-a-manea.bsky.social & @jlibovicky.bsky.social TL;DR: Explore how parallel datasets improve cross-lingual transfer in vision-language models. arxiv.org/abs/2504.21681

🧵 We're releasing CUS-QA - a new benchmark for testing LLMs on regional knowledge! Find out what your model knows about Czechia 🇨🇿, Slovakia 🇸🇰, and Ukraine 🇺🇦! 👉 Textual and visual questions, answers, and human judgment on model outputs! huggingface.co/datasets/ufa... www.arxiv.org/abs/2507.22752

ufal/cus-qa · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

Stay tuned, we will release the dataset soon...

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · last yr.

CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset arxiv.org/abs/2507.22752 by @jlibovicky.bsky.social , ‪@jindrahelcl.bsky.social, @andrei-a-manea.bsky.social Question that foreigners don't know the answer to + human judgment on question generation

We need to have poster fights at the end of every conference.

Just presented MAGBIG, a new dataset and evaluation methodology for gender bias in multilingual text-to-image generation. Grammatical gender matters when studying these biases across languages! Thanks to Felix Friedrich, @kathaem.bsky.social and all co-authors - it was fun to work on this together!

Institute of Formal and Applied Linguistics@ufal.mff.cuni.cz · last yr.

Multilingual Text-to-Image Generation Magnifies Gender Stereotypes aclanthology.org/2025.acl-lon... by Felix Friedrich, @kathaem.bsky.social, Patrick Schramowski, @mbrackaiml.bsky.social , @jlibovicky.bsky.social, @kerstingaiml.bsky.social, Alex Fraser

This week I am at #ACL2025NLP in Vienna 🎡🇦🇹. Find me 🕵️ or message 💌 me if you want to chat about multilinguality or tokenization. Stop 🛑 by our poster on gender bias in text-to-image generation on Monday aclanthology.org/2025.acl-lon...

TokShop @ #ICML2025 got way more submissions than expected! 📈 We could really use a few more reviewers to help out. If you have the capacity to review a #tokenization paper by Saturday, please fill out this form: forms.gle/32A6sQHQrMSb... 🙏

TokShop 2025

Registering interest in all things tokenization at TokShop @ ICML 2025 (July 18) Consider joining the Google group for future updates! https://groups.google.com/g/tokshop

forms.gle

📣 Call for Paper Alert: TokShop @ ICML 2025 TokShop explores tokenization across all data modalities. Topics include: subword NLP techniques, multimodal approaches, multilingual challenges, post-training modification, alternative representations, and statistical perspectives.

ICML 2025 Workshop TokShop

Welcome to the OpenReview homepage for ICML 2025 Workshop TokShop

openreview.net