Tessera
Folgen

Leqi Liu

@leqiliu.bsky.social

AI/ML Researcher | Assistant Professor at UT Austin | Postdoc at Princeton PLI | PhD, Machine Learning Department, CMU. Research goal: Building controllable machine intelligence that serves humanity safely. leqiliu.github.io

24 Posts175 Follower7 Folgt
Leqi Liu@leqiliu.bsky.social · last yr.

New method to crack hard reasoning problems with LLM! No expert traces. No test-time hacks. Just: Self-explanation + RL-style training Result? Accuracy on MATH level-5 jumped from 2% → 23%.

For hard reasoning tasks, the chance of sampling a correct answer is low. Thus, sharpening the sampling distribution is not enough, and standard RL post-training fails.
Leqi Liu@leqiliu.bsky.social · last yr.

What if you could understand and control an LLM by studying its *smaller* sibling? Our new paper introduces the Linear Representation Transferability Hypothesis. We find that the internal representations of different-sized models can be translated into one another using a simple linear(affine) map.

Leqi Liu@leqiliu.bsky.social · 2y ago

Ever wondered why there are synchronized ups and downs for chosen and rejected log-probs during DPO (and most *POs: IPO, SimPO, CPO, R-DPO, DPOP, RRHF, SlicHF) training? Why do chosen logps decrease, and rejected logps sometimes increase? Our answer: Gradient Entanglement! arxiv.org/abs/2410.13828

Bild
Leqi Liu@leqiliu.bsky.social · 2y ago

How to **efficiently** build personalized language models **without** textual info on user preferences? Our Personalized-RLHF work: - light-weight user model - personalize all *PO alignment algorithms - strong performance on the largest personalized preference dataset arxiv.org/abs/2402.05133