🔥What if web text isn’t the best place to start training LLMs? Our latest work shows that warming up models on procedural data (e.g. from formal languages & simple algorithms) speeds up subsequent pretraining on language, code, and math, on models up to 1.3B parameters⬇️🧵

Can vision transformers learn without images?🤔👀 Our latest work shows that pretraining ViTs on procedural symbolic data (eg sequences of balanced parentheses) makes subsequent standard training (eg on ImageNet) more data efficient! How is this possible?! ⬇️🧵

Coming up at ICML: 🤯Distribution shifts are still a huge challenge in ML. There's already a ton of algorithms to address specific conditions. So what if the challenge was just selecting the right algorithm for the right conditions?🤔🧵

OMG I can confirm this ... tested by @mbsariyildiz.bsky.social on our new upcoming work (vision/robotics). Thanks @damienteney.bsky.social the effect is real 😍 arxiv.org/abs/2505.20802

⬇️ "Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild" arxiv.org/abs/2503.10065

Andrei Bursuc@abursuc.bsky.social · last yr.

Thought-provoking work by @damienteney.bsky.social et al. looking for optimal inductive biases (activation functions) in the wild (beyond image classification: regression, tabular, algorithmitc, shortcut). ReLU works well on avg, but you can find completely different activations. #cvpr2025

Coming up this week: (oral @cvprconference.bsky.social) Do We Always Need the Simplicity Bias? We take another step to understand why/when neural nets generalize so well. ⬇️🧵

What does/will happen when the model learns (from experience or historical data) that accepted papers contain grand claims and bold numbers >SOTA?💡Hint: it's not "doing rigorous science".

Sung Kim@sungkim.bsky.social · last yr.

Intology releases Zochi, the Artificial Scientist with state-of-the-art contributions accepted in ICLR 2025 workshops. With a standardized automated reviewer, Zochi’s papers score an average of 7.67 compared to other publicly available papers generated by AI systems that score between 3 and 4.

arxiv.org/abs/2505.20802

Christian Wolf@chriswolfvision.bsky.social · last yr.

Leaner Transformers: More Heads, Less Depth H. Saratchandran, @damienteney.bsky.social , S. Lucey

Seen on the other place... 😕🤷 Any advice to get more ML and less bunnies/politics in my Bluesky feed?

This ⬇️ is also great advice for writing a paper! 👌 But start it one *month* before the deadline.

Christian Wolf@chriswolfvision.bsky.social · last yr.

Creating your slides at the last minute is a bad idea for multiple reasons, IMHO. Not only is generally bad to rush things, but having a first draft ready a week before your presentation will make you revisit them mentally while showering, cycling, driving, cooking ...

"Deep learning does not require rethinking generalization" If you enjoyed our work on inductive biases (eg the Neural Redshift arxiv.org/abs/2403.02241), you'll love this paper that rigorously articulates "soft inductive biases" & how they explain supposedly-mysterious behaviors of neural nets.

Andrew Gordon Wilson@andrewgwils.bsky.social · last yr.

My new paper "Deep Learning is Not So Mysterious or Different": arxiv.org/abs/2503.02113. Generalization behaviours in deep learning can be intuitively understood through a notion of soft inductive biases, and formally characterized with countable hypothesis bounds! 1/12

This is as useful as telling us what the authors had for breakfast on the day of the experiments. 🤷

Une chose me marque au sujet de l'IA, notamment générative. Que penser d'une innovation technico-scientifique dont la promotion dans les médias est plutôt du fait d'entrepreneurs, éditorialistes et politiciens, tandis que les scientifiques du secteur sont bien plus mesurés ?

"Attention" in attention layers. How about sum-product layers? Key-query products? ... Neural attention has little to do with human attention. And the intuitive baggage of the name probably constrains our thinking about how transformers work. (1/2)

Federico Adolfi@fedeadolfi.bsky.social · 2y ago

If you could fix one☝🏻 piece of terminology in your field, what would it be? I’ll go first👇🏻(replying to myself like it’s normal)

💡 Just learned about a useful short-hand notation for "expectation". Seems common for physicists but I can't remember coming across it before. With an example use-case below ⬇️

Reviewing ML papers? 💡 If you feel that experiments are missing, ask yourself: are the additional results likely to affect the central message of the paper/nullify its main claims? If not, it's probably a nice suggestion (eg additional comparisons, datasets) but not a reason for rejection by itself.

PSA: Can we use more bar charts in ML papers? I can't recall the last time I wanted to compare dozens of numbers in a table to two decimal places. A visualization makes it much clearer whether claimed differences are significant.

Writing tips! This should be mandatory reading for every PhD student 👇 We'll all benefit from it as readers.

Michael J. Black@michael-j-black.bsky.social · 2y ago

For those who missed this post on the-network-that-is-not-to-be-named, I made public my "secrets" for writing a good CVPR paper (or any scientific paper). I've compiled these tips of many years. It's long but hopefully it helps people write better papers. perceiving-systems.blog/en/post/writ...