💡 We compare prompting (zero and multi-shot + explanations) and inference-time interventions (ActAdd, REFT and SAEs). Following SpARE (@yuzhaouoe.bsky.social @alessiodevoto.bsky.social), we propose ✨ contrastive SAE steering ✨ with mutual info to personalize literary MT by tuning latent features 4/

MMLU-Redux just touched down at #NAACL2025! 🎉 Wish I could be there for our "Are We Done with MMLU?" poster today (9:00-10:30am in Hall 3, Poster Session 7), but visa drama said nope 😅 If anyone's swinging by, give our research some love! Hit me up if you check it out! 👋

My amazing collaborators will present several works at ICLR and NAACL later this month -- please catch up with them if you're attending! I tried to summarise our recent work in a blog post: neuralnoise.com/2025/march-r...

Please share it within your circles! edin.ac/3DDQK1o

🚀 New Paper Alert! 🚀 We introduce Q-Filters, a training-free method for efficient KV Cache compression! It is compatible with FlashAttention and can compress along generation which is particularly useful for reasoning models ⚡ TLDR: we make Streaming-LLM smarter using the geometry of attention

Live from the CoLoRAI workshop at AAAI (april-tools.github.io/colorai/) Nadav Cohen is now giving his talk on "What Makes Data Suitable for Deep Learning?" Tools from quantum physics are shown to be useful in building more expressive deep learning models by changing the data distribution.

2018: Saliency maps give plausible interpretations of random weights, triggering skepticism and catalyzing the mechinterp cultural movement, which now advocates for SAEs. 2025: SAEs give plausible interpretations of random weights, triggering skepticism and ...

Graphical tensor notation for interpretability www.lesswrong.com/posts/BQKKQi...

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. sakana.ai/ai-cuda-engi... The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Examples:

It's 2025, and I’ve finally updated my Python setup guide to use uv + venv instead of conda + pip! Here's my go-to recommendation for uv + venv in Python projects for faster installs, better dependency management: github.com/rasbt/LLMs-f... (Any additional suggestions?)

Cool research on how models memorize data 📝 : The 'Manifold Memorization Hypothesis' by Brendan Ross, Hamidreza Kamkariet al. suggests memorization occurs when the model's learned manifold matches the true data manifold but with too small 'local intrinsic dimensionality'.

Massive activations & weights in LLMs, two cool works 🤓: - The Super Weight: finds performance can be totally degraded when pruning a *single* weight - Mengxia Yu et al. - Massive Activations in LLM:finds some (crucial) activations have very high norm irrespective of context - Mingjie Sun et al.

On the last day before the Spring Festival holiday in China, DeepSeek released a NEW work on @hf.co 🤯 Janus-Pro🔥 autoregressive framework that unifies multimodal understanding and generation huggingface.co/deepseek-ai/... ✨ 1B / 7B ✨ MIT License

deepseek-ai/Janus-Pro-1B · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

Not only that, but much of the science community here is already stronger and larger than it was on X. On Twitter, my feed of scientists who study climate-related topics topped out at 3300. Here, we’re at 4500 already and it’s still growing. Pin here: bsky.app/profile/did:...

Nature@nature.com · 2y ago

Roughly 6,000 readers answered our poll, with many declaring that Bluesky was nicer, kinder and less antagonistic to science than X https://go.nature.com/42tH8Ai

*MoE Graph Transformers for Interpretable Particle Collision Detection* by @alessiodevoto.bsky.social @sgiagu.bsky.social et al. We propose a MoE graph transformer for particle collision analysis, with many nice interpretability insights (e.g., expert specialization). arxiv.org/abs/2501.03432

deepseek GGUF just dropped, if you have 207GB disk/40GB RAM for the smallest version huggingface.co/collections/...

Deepseek V3 (All Versions) - a unsloth Collection

Deepseek V3 - available in bf16, original, and GGUF formats, with support for 2, 3, 4, 5, 6 and 8-bit quantized versions.

huggingface.co

LLMs inner representations🔬 Llamas Work in English: LLMs default to English-based concept representations, regardless of input language @wendlerc.bsky.social et al Semantic Hub: Multimodal models create a single shared semantic space, structured by their primary language @zhaofengwu.bsky.social et a

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

In Vision & Audio transformers, not all tokens need the same compute resources! We propose “modular learners” to control compute at token-level granularity (MHA & MLP): hard tokens get more, easy ones get less! w/ @sscardapane.bsky.social @neuralnoise.com @bartoszWojcik Soon #AAAI25 Link 👇

Josh Tenenbaum on scaling up vs growing up and the path to human-like reasoning #NeurIPS2024

*Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference* with @alessiodevoto.bsky.social @neuralnoise.com Happy to share our work on distilling efficient transformers with dynamic modules' activation was accepted at #AAAI2025. 🔥 arxiv.org/abs/2312.10193

*Sparse Crosscoders for Cross-Layer Features and Model Diffing* by @colah.bsky.social @anthropic.com Investigates stability & dynamics of "interpretable features" with cross-layers SAEs. Can also be used to investigate differences in fine-tuned models. transformer-circuits.pub/2024/crossco...

Very cool work! 👏🚀 Unfortunately, errors in the original dataset will propagate to all new languages 😕 We investigated the issue of existing errors in the original MMLU in arxiv.org/abs/2406.04127 @aryopg.bsky.social @neuralnoise.com

Sara Hooker@sarahooker.bsky.social · 2y ago

Is MMLU Western-centric? 🤔 As part of a massive cross-institutional collaboration: 🗽Find MMLU is heavily overfit to western culture 🔍 Professional annotation of cultural sensitivity data 🌍 Release improved Global-MMLU 42 languages 📜 Paper: arxiv.org/pdf/2412.03304 📂 Data: hf.co/datasets/Coh...

Super Cool work from Cohere for AI! 🎉 However, this highlights a concern raised by our MMLU-Redux team (arxiv.org/abs/2406.04127): **error propagation to many languages**. Issues in MMLU (e.g., "rapid intervention to solve ebola") seem to persist in many languages. Let's solve the root cause first?

Sara Hooker@sarahooker.bsky.social · 2y ago

Is MMLU Western-centric? 🤔 As part of a massive cross-institutional collaboration: 🗽Find MMLU is heavily overfit to western culture 🔍 Professional annotation of cultural sensitivity data 🌍 Release improved Global-MMLU 42 languages 📜 Paper: arxiv.org/pdf/2412.03304 📂 Data: hf.co/datasets/Coh...

Cool take on straight-through-estimator (to backpropagate through discrete ops): during back-propagation, keep gradient's relative angle, not absolute direction. The authors call this the "rotation trick". From "Restructuring Vector Quantization With The Rotation Trick" @ChristopherFifty et al.

Wish I knew about this much earlier: In VS Code (or Cursor 😉) you can install a 'data wrangler' extension to inspect the values of your PyTorch Tensors in a nice UI with summary statistics. Just right-click on a tensor in your variables view!

Why don’t they just make the whole app out of L1 cache samwho.dev/numbers/

Latency Numbers Every Programmer Should Know

An interactive exploration of how long things take.

samwho.dev

It's Sunday morning so taking a minute for a nerdy thread (on math, tokenizers and LLMs) of the work of our intern Garreth By adding a few lines of code to the base Llama 3 tokenizer, he got a free boost in arithmetic performance 😮 [thread]