Tobi

@mocutobi.bsky.social

Founder @ ??? Ex-Machine Learning Engineer @ FAANG. Researched DeepRL+Affective Computing @usc

this "claude made a c compiler thing" feels pretty dishonest and marketing-hype whenever you scroll to the very bottom to find these lines and the token cost lol i mean, sure its impressive (particularly that it was a fully offline environment), but seems like hype bait

The generated code is not very efficient. Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled.
The Rust code quality is reasonable, but is nowhere near the quality of what an expert Rust programmer might produce.

Also good thing to keep in mind, that afaik the current "subscription token economy" is heavily subsidized and I am not sure Anthropic/OpenAI are charging necessarily full amount (i.e the same price that would charge for API cost)

Tobi@mocutobi.bsky.social · 7mo ago

This is why I genuinely believe that in the model game, there is no moat, and there will always be tremendous pressure for open source and cheaper models. The majority of the world economy doesn't want to pay $21 (!!!) per million tokens, even if they can afford to

ByteDance Seed's ConceptMoE: moving beyond uniform token-level processing to adaptive concept-level computation in LLMs! Why waste equal compute on trivially predictable tokens? When you can merge similar tokens into concepts while preserving fine-grained processing for complex content.

Bild

I did some experiments with nanochat and the SYNTH dataset from Pleias. I wanted to compare how a model pretrained with a mixture of SYNTH and FineWeb would compare to the standard FineWeb mixture. Results: Pretraining on Fineweb alone looks good (see below), but the story changes after midtraining:

Pretraining results for training on SYNTH + FineWeb vs FineWeb alone. 

Four rows are displayed: dgx1-continued, dgx1-synth-contued, dgx1-synth, and dgx1.

Rows labeled *-synth are trained with 3:1 SYNTH:FineWeb mixture. Results with *-continued* are trained for 3B tokens, 1B otherwise. CORE metric:

nanochat + SYNTH, 3B tokens: 0.147 CORE Metric
nanochat, 3B tokens: 0.166
nanochat + SYNTH, 1B tokens: 0.123
nanotchat, 3B tokens: 0.139

Been running comparisons nanochat pretrained on FineWebEdu vs synthetic data and it’s not looking great for the synthetic data runs thus far. We’ll see what it looks like after mid training

When Bill Clinton left office, we had a budget surplus. After 8 years of disastrous foreign interventions and tax cuts, we had deficit and a recession. Now with Trump we’re repeating repeating the same mistakes made under Bush

If you assume that there is an infinite number of tech workers with six figure salaries who want to live in the Bay or NY then the NIMBYs are right and no amount of new housing will help. But there is not an infinite number of tech workers, and the ones here are being laid off.

People talk about artificial intelligence but don't understand what that technology actually is, what the term "AI" is referring to, or best use cases for all the different tools that exist. Nor do they understand that they've already been using AI assistance for YEARS before LLMs became public.

"In the past, Ono had condemned systemic racism & argued for the necessity of DEI [...] Ono emphasized his ideological evolution, which ultimately failed to convince the board" Board said hm you're *new to hatin niggas and we're lookin for someone *true to hatin niggas. good luck on the job market!

Josh Moody@byjoshmoody.bsky.social · last yr.

Basic details of today's vote here, with more to come: www.insidehighered.com/news/governa...

“The University of Cincinnati wants to make all students eligible for Davis’s scholarship, not just Black applicants. At least six other scholarships worth about $4 million established by Black doctors at the school are also under review, according to an email obtained by The Washington Post.”

Trump DEI policies threaten millions in scholarships raised by Black doctors

Trump’s campaign to DEI programs has put at risk six scholarships worth about $4 million established by Black doctors in Cincinnati.

washingtonpost.com

The elephant in the room on social housing is that we severely lack the state capacity required to administer it at the levels required to bring down rents. And increasing state capacity is what’s explicitly being argued for in the Abundance agenda

Ok trying the video upload feature--a Sephardic tune on the baroque lute, Siete Modos de Guisar las Berenjenas (seven ways to prepare eggplants, only two ways are discussed bc we're at 60 seconds).