🤔 Can simple string-matching metrics like BLEU rival reward models for LLM alignment? 🔍 We show that given access to a reference, BLEU can match reward models in human preference agreement, and even train LLMs competitively with them using GRPO. 🫐 Introducing BLEUBERI:

🕵️‍♀️ agents are strong on many tasks, but are they good at interacting with the web? 🧸our BEARCUBS benchmark shows that they struggle on interactive tasks that seem trivial to humans! 📄 check out the paper for how to build robust evaluations & directions for future agent research

Yixiao Song@yixiaosong.bsky.social · last yr.

Introducing 🐻 BEARCUBS 🐻, a “small but mighty” dataset of 111 QA pairs designed to assess computer-using web agents in multimodal interactions on the live web! ✅ Humans achieve 85% accuracy ❌ OpenAI Operator: 24% ❌ Anthropic Computer Use: 14% ❌ Convergence AI Proxy: 13%

Is the needle-in-a-haystack test still meaningful given the giant green heatmaps in modern LLM papers? We create ONERULER 💍, a multilingual long-context benchmark that allows for nonexistent needles. Turns out NIAH isn't so easy after all! Our analysis across 26 languages 🧵👇

current models struggle with complex long-range reasoning tasks 📚 how can we reliably create synthetic training data? 💽 check out CLIPPER, a pipeline that generates data conditioning on compressed forms of long input documents!

Chau Minh Pham@chautmpham.bsky.social · last yr.

⚠️Current methods for generating instruction-following data fall short for long-range reasoning tasks like narrative claim verification. We present CLIPPER ✂️, a compression-based pipeline that produces grounded instructions for ~$0.5 each, 34x cheaper than human annotations.

People often claim they know when ChatGPT wrote something, but are they as accurate as they think? Turns out that while general population is unreliable, those who frequently use ChatGPT for writing tasks can spot even "humanized" AI-generated text with near-perfect accuracy 🎯

Great blog post (by a 15-author team!) on their release of ModernBERT, the continuing relevance of encoder-only models, and how they relate to, say, GPT-4/llama. Accessible enough that I might use this as an undergrad reading.

Finally, a Replacement for BERT: Introducing ModernBERT

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

🚨I too am on the job market‼️🤯 I'm searching for faculty positions/postdocs in multilingual/multicultural NLP, vision+language models, and eval for genAI! I'll be at #NeurIPS2024 presenting our work on meta-evaluation for text-to-image faithfulness! Let's chat there! Papers in🧵, see more: saxon.me

🐠 what monday feels like..

private closed-source evals are the future 🫣

i knew something like this had to exist but why did i only discover it now?? no more suffering from looking at my 10+ open arxiv tabs not knowing which one is which...

I noticed a lot of starter packs skewed towards faculty/industry, so I made one of just NLP & ML students: go.bsky.app/vju2ux Students do different research, go on the job market, and recruit other students. Ping me and I'll add you!

such a creative way of using long-context models! this sounds like a super hard evaluation task, but gemini is already so good at it...

Lynn Cherny@arnicas.bsky.social · 2y ago

Steve Johnson on using book text to build a text adventure in NotebookLM. “the game relies on three elements: the original text from my book; a large language model (… Gemini Pro 1.5); and a 400-word prompt that I wrote giving the model instructions on how to host the game” thelongcontext.com

Mat is not on 🦋—posting on his behalf! It's time to revisit common assumptions in IR! Embeddings have improved drastically, but mainstream IR evals have stagnated since MSMARCO + BEIR. We ask: on private or tricky IR tasks, are rerankers better? Surely, reranking many docs is best?

The soul-searching journey for figuring out what research area is right for you is tricky since so many papers are cool. I tell my early career students that they should try to differentiate papers that they'd like to read 📖, implement 🔨, *and* write 📝 from papers that they'd only like to read 📖.

#EMNLP2024 was fun🍹now brainstorming ideas for #EMNLP2025 🙇🏻‍♀️

airbnb >>> hotel for conferences #EMNLP2024

✨I am on the faculty job market in the 2024-2025 cycle!✨ My research centers on advancing Responsible AI, specifically enhancing factuality, robustness, and transparency in AI systems. If you have relevant positions, let me know! lasharavichander.github.io Please share/RT!

Long-form text generation with multiple stylistic and semantic constraints remains largely unexplored. We present Suri 🦙: a dataset of 20K long-form texts & LLM-generated, backtranslated instructions with complex constraints. 📎 arxiv.org/abs/2406.19371

I really wanted to run NEW #nocha benchmark claims on #o1 but it won't behave 😠 - 6k reasoning tokens is often not enough to get an ans and more means being able to process only short books - OpenAI adds sth to the prompt: ~8k extra tokens-> less room for book+reason+generation!

🌊Heading to #EMNLP2024 tmr, presenting PostMark on Tue. morning! 🔗 arxiv.org/abs/2406.14517 Aside from this, I'd love to chat about: • long-context training • realistic & hard eval • synthetic data • tbh any cool projects people are working on Also, I'm on the lookout for a summer 2025 internship!

Looking forward to catching up with old and new friends at #EMNLP2024! I’m on the academic job market so please reach out if you would like to chat 🙏 And come talk to me, @rnv.bsky.social and @iaugenstein.bsky.social on Thu (Nov 14) at poster session G from 2-3:30PM about LLM tropes!

Dustin Wright@dustinbwright.com · 2y ago

At #EMNLP2024 we will present our paper on LLM values and opinions! We introduce tropes: repeated and consistent phrases which LLMs generate to argue for political stances. Read the paper to learn more! arxiv.org/abs/2406.19238 Work done Uni Copenhagen + Pioneer Center for AI

Starter packs are a great way to find people. But I followed a few tech/AI starter packs, and now have a sizeable gender skew in who I'm following. To counteract, I started collecting this list. Who else should I be following & add? 👇 go.bsky.app/LaGDpqg

thxx for the acknowledgment 🥰 also check out our followup work NoCha (novelchallenge.github.io), a benchmark that asks LLMs to verify claims about new fictional books! the leaderboard is updated with very new models

Mark J. Nelson@mm-jj-nn.bsky.social · 2y ago

Impressive attempt to evaluate how well LLMs can summarize novels. They use 26 recent books (so summaries won't be in the training set), extract 3158 claims from the LLM summaries, and have humans evaluate the claims' accuracy.

we released our open multimodal language model Molmo today 🥳 🍝 secret sauce? really really high quality set of image+text pairs, which we'll release openly 🕹️ try it out: molmo.allenai.org 📎 read more about it: molmo.allenai.org/blog 🤗 download models: huggingface.co/collections/...