Tessera
Folgen

Anirbit

@anirbit.bsky.social

Assistant Professor/Lecturer in ML @ The University of Manchester | https://anirbit-ai.github.io/ | working on the theory of neural nets and how they solve differential equations. #AI4SCIENCE

68 Posts151 Follower69 Folgt
Anirbit@anirbit.bsky.social · last yr.

With all renewed discussion about "Sparse AutoEncoders (#SAE)" as a way of doing #MechanisticInterpretability of #LLMs, I am resharing a part of my PhD where we proved years ago about how sparsity automatically emerges in autoencoding. arxiv.org/abs/1708.03735

Sparse Coding and Autoencoders

In "Dictionary Learning" one tries to recover incoherent matrices $A^* \in \mathbb{R}^{n \times h}$ (typically overcomplete and whose columns are assumed to be normalized) and sparse vectors $x^* \in ...

arxiv.org

Anirbit@anirbit.bsky.social · last yr.

Why does noisy gradient-descent train neural nets? This fundamental question in ML remains unclear. In our hugely revised draft my student @dkumar9.bsky.social gives the full proof that a form of noisy-GD, Langevin Monte-Carlo (#LMC), can learn arbitrary depth 2 nets. arxiv.org/abs/2503.10428

Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data

In this work, we will establish that the Langevin Monte-Carlo algorithm can learn depth-2 neural nets of any size and for any data and we give non-asymptotic convergence rates for it. We achieve this ...

arxiv.org

Anirbit@anirbit.bsky.social · last yr.

Some luck to be hosted by a Godel Prize winner, Prof. Sebastien Pokutta, and to present our work in their group 💥 Sebastien heads this "Zuse Institute Berlin (#ZIB) " which is an amazing oasis of applied mathematics bringing together experts from different institutes in Berlin.

BildBild
Anirbit@anirbit.bsky.social · last yr.

University of Manchester has a 1 year post-doc position that I am happy to support in our group if you are currently an #EPSRC funded PhD student - and have the required specialization for work in our group. Typicall we prefer candidates who have published in deep-learning theory or fluid theory.

Anirbit@anirbit.bsky.social · last yr.

So, the next time you train a deep-learning model, it's probably worthwhile to have a baseline for the only provable adaptive gradient deep-learning algorithm - our delta-GClip 🙂

Anirbit@anirbit.bsky.social · last yr.

Our "delta-GCLip" is the *only* known adaptive gradient algorithm that provably trains deep-nets AND is practically competitive. That's the message of our recently accepted #TMLR paper - and my 4th TMLR journal 🙂 openreview.net/pdf?id=ABT1X... #optimization #deeplearningtheory

Anirbit@anirbit.bsky.social · last yr.

Our insight is to introduce an intermediate form of gradient clipping that can leverage the PL* inequality of wide nets - something not known for standard clipping. Given our algorithm works for transformers maybe that points to some yet unkown algebraic property of them. #TMLR

BildBild
Anirbit@anirbit.bsky.social · last yr.

Our "delta-GCLip" is the *only* known adaptive gradient algorithm that provably trains deep-nets AND is practically competitive. That's the message of our recently accepted #TMLR paper - and my 4th TMLR journal 🙂 openreview.net/pdf?id=ABT1X... #optimization #deeplearningtheory

Anirbit@anirbit.bsky.social · last yr.

An updated version of our slides on necessary conditions for #SciML, - and more specially, "Machine Learning in Function Spaces/Infinite Dimensions". Its all about the 2 key inequalities on slides 27 and 33. Both come via similar proofs. github.com/Anirbit-AI/S...

GitHub - Anirbit-AI/Slides-from-Team-Anirbit: Slide Presentations of Our Works

Slide Presentations of Our Works. Contribute to Anirbit-AI/Slides-from-Team-Anirbit development by creating an account on GitHub.

github.com

Anirbit@anirbit.bsky.social · last yr.

It would be great to be able to see a compiles list of useful PDEs that #PINNs struggle to solve - and how would we measure success there. We know of edge-cases with simple PDEs, where PINNs struggle, but then often those aren't the cutting-edge of use-cases of PDEs.

Anirbit@anirbit.bsky.social · 2y ago

A revised version of our delta-GClip algorithm - which is probably the *only* deep-learning algorithm that provably trains deep-nets while using step-size scheduling - and competes/supersedes heuristics like Adam and even Adam+Clipping on transformers. arxiv.org/abs/2404.08624

Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks

We present and analyze a novel regularized form of the gradient clipping algorithm, proving that it converges to global minima of the loss surface of deep neural networks under the squared loss, provi...

arxiv.org

Anirbit@anirbit.bsky.social · 2y ago

He was the first person to interview me for a PhD position in applied maths and stats when I decided to shift my career focus in that direction. Years later when I became a faculty, he accepted my invite to fly to UK from Leipzig to give a talk and meet my students. Sayan is irreplaceable 😶

Maxim Raginsky@mraginsky.bsky.social · 2y ago

To say that I am shocked at this tragic news is an understatement. I got to know Sayan well during my time at Duke; he was deeply knowledgeable about all sorts of things and had a great sense of humor. He was the one and only Bayesian anarchist. www.mis.mpg.de/news/loss-sa...

Anirbit@anirbit.bsky.social · 2y ago

My PhD student @dkumar9.bsky.social has done an amazing job of blending multiple deep results to establish that Langevin Monte-Carlo algorithm provably learns 2-layer nets for any data and for any size. This is very rigorously a "beyond NTK" regime. More to come from Dibyakanti 🙂

Dibyakanti Kumar@dkumar9.bsky.social · 2y ago

Noisy gradient descent has attracted a lot of attention in the last few years as a mathematically tractable model of actual deep-learning algorithms. In my recent work with @anirbit.bsky.social and Samyak Jha (arxiv.org/abs/2503.10428), we prove noisy gradient descent learns neural nets.

Anirbit@anirbit.bsky.social · 2y ago

My first year PhD student Sébastien André-sloan presents at a #INFORMS conference in Toronto. Its a joint work with Matthew Colbrook at DAMTP, Cambridge. We prove a first-of-its-kind size requirement on neural nets for solving PDEs in the super-resolution setup - the natural setup for #PINNs.

Bild
Anirbit@anirbit.bsky.social · 2y ago

doi.org/10.1093/imai... Our first #IMA journal paper! 😊 We showed Gibbs' measures of neural losses can satisfy the Poincare inequality - and this also holds when the loss is non-Lipschitz. This opens up the first way to get convergence of SGD on such nets *without restrictions on data or size*. 💥

Global convergence of SGD on two layer neural nets

Abstract. In this note, we consider appropriately regularized $\ell _{2}-$empirical risk of depth $2$ nets with any number of gates and show bounds on how

doi.org

Anirbit@anirbit.bsky.social · 2y ago

second time speaking @ the holy land of statistics, the Indian Statistical Institute (ISI) 🙂 In 2022 I was @ the Bangalore campus. Same topic : theory of operator learning - the story continues 🔥 Both journal papers were published at #TMLR in 2024.

Bild
Anirbit@anirbit.bsky.social · 2y ago

Here are the slides that my PhD student Dibyakanti Kumar made for his talk @ the #CMStatistics conference talk yesterday on our journal, www.linkedin.com/posts/anirbi... #PINNs are a barely understood way of using nets, for solving PDEs - and we take a careful look at them. #AI4SCIENCE #SciML

Anirbit Mukherjee on LinkedIn: CMStatistics-Talk-Dibyakanti

Here are the slides that Dibyakanti Kumar had made for his talk @ #CMStatistics conference talk today on our journal paper, https://lnkd.in/gF4x7xQ6 I think…

linkedin.com