Tessera
Folgen

Pete Shaw

@ptshaw.bsky.social

Research Scientist at Google DeepMind. Mostly work on ML, NLP, and BioML. Based in Seattle. http://ptshaw.com

9 Posts1592 Follower356 Folgt
Pete Shaw@ptshaw.bsky.social · 12mo ago

Excited to share a new paper that aims to narrow the conceptual gap between the idealized notion of Kolmogorov complexity and practical complexity measures for neural networks.

Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
Ahmad Beirami@abeirami.bsky.social · 2y ago

Excited to share 𝐈𝐧𝐟𝐀𝐥𝐢𝐠𝐧! Alignment optimization objective implicitly assumes 𝘴𝘢𝘮𝘱𝘭𝘪𝘯𝘨 from the resulting aligned model. But we are increasingly using different and sometimes sophisticated inference-time compute algorithms. How to resolve this discrepancy?🧵

InfAlign: Inference-aware language model alignment Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, Ahmad Beirami
Jacob Eisenstein@jacobeisenstein.bsky.social · 2y ago

I’m pretty excited about this one! ALTA is A Language for Transformer Analysis. Because ALTA programs can be compiled to transformer weights, it provides constructive proofs of transformer expressivity. It also offers new analytic tools for *learnability*. arxiv.org/abs/2410.18077

ALTA: Compiler-Based Analysis of Transformers

We propose a new programming language called ALTA and a compiler that can map ALTA programs to Transformer weights. ALTA is inspired by RASP, a language proposed by Weiss et al. (2021), and Tracr (Lin...

arxiv.org