How it feels to be worried about both

Dashiell@dashiells.bsky.social · last wk.

AI safety folks should be way more worried about concentration of power than the proliferation of powerful models

There are a bunch of ways the near future could go that require AI treaty verification tech. We can separate treaty verification tech into 3 levels, roughly ordered by strength: 1. Pragmatic 2. Enclaves 3. Math Hardware, security, and crypto folk should work on all of these!

Natural selection timelapse

Ha, neat! Adam Goucher conditionally disproved my absurd conjecture, in its most natural 1980's vintage big-endian form. So we have to modify it! "All but finitely many primes have the same little-endian SHA256* hash." *Assuming a good arbitrary-length extension. x.com/apgox/status...

Adam P. Goucher (@apgox) on X

@geoffreyirving Ignoring for the moment that SHA256 is only defined on inputs of fewer than 2^64 bits, a proof of the first Hardy-Littlewood conjecture would allow you to construct a bunch of nearby p...

x.com

Geoffrey Irving@girving.bsky.social · 2w ago

In case you need a control conjecture that the machines will never be able to disprove: "All but finitely many primes have the same SHA256 hash."

In case you need a control conjecture that the machines will never be able to disprove: "All but finitely many primes have the same SHA256 hash."

Riemann is still true tho.

One lacking area of alignment theory is how best to think about rationalization, the process of (1) guessing an answer and (2) justifying it after the fact. Ideally multiple teams at Resolution will touch on this question from different directions, using different tools.

We're excited to announce that Resolution has a $160M grant from Coefficient Giving: $108M unconditional, with a further $52M conditional on hiring and compute needs. We'll use it to grow teams across our research portfolio and invest heavily in research automation. 🧵 resolution.org/post/funding

We’re changing our name: Sequent is now Resolution. 🧵

Geoffrey Irving@girving.bsky.social · 2mo ago

We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵 sequent.org/launch

A while ago I wrote a minimal replacement for top called ltop using Claude Opus 4.6-4.8. At first I made it so that I could nicely filter the processes to just the ones I wanted, but then I noticed the binary was pretty small, and got curious if it could be smaller... 🧵 github.com/girving/ltop

We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵 sequent.org/launch

Don’t Worry — It Can’t Happen (also, the scientists who claim fission exists are just in the pocket of Big Science, and they’re literal nazis anyway, and also it’s just a stochastic reaction that peters out, and only physbros care about it, and it hasn’t ever happened before so it won’t)

Here is a metaphor for AGI definitions. Imagine you’re on a long drive from Los Angeles to the Bay Area (for me: undergrad to grad school). 🧵

A while ago I wrote a Claude skill to parse Google Docs into Markdown. It was easy, but took a few iterations to get right, as the first pass had weird glitches like \! instead of !. What else needs a couple more iterations? Literally Google Doc's own agent Markdown converter.

An inconvenient realisation today is that while I've written dozens of documents that refer to something like "pretraining on human-level data", now that phrase always has to be amended to "pretraining on (mostly) human-level data".

average MLeng: 'well, the adversary would have to know the details of how our defense works' the humble Kerckhoff axiom:

A bittersweet announcement! For family reasons, I will be leaving AISI soon to move back to the Bay Area. I will be starting a new nonprofit alignment research org (more to come). I will miss this place! Here are some reflections about my time at AISI. 🧵❤️| naml.us/post/reflect...

New paper arguing that AI automation of AI alignment research could fail due to AI mistakes, even if the AI agents are intent aligned (not trying to cause harm). Arguably this is obvious: AIs make mistakes all the time (as do humans). But it is useful to go into detail.🧵 arxiv.org/abs/2605.06390

Automated alignment is harder than you think

A leading proposal for aligning artificial superintelligence (ASI) is to use AI agents to automate an increasing fraction of alignment research as capabilities improve. We argue that, even when resear...

arxiv.org

The real question is whether Homer faked a Bronze Age Greek accent when doing his Iron Age performances.

Matthew Yglesias @mattyglesias.bsky.social · 3mo ago

Can’t believe the American accents in The Odyssey trailer. Characters should be speaking with English accents just like real Bronze Age Greeks.

The most delightful bug that Mythos found in Firefox is a NaN vulnerability. There are many bit representations of floating point not-a-number, one of which looks like a tagged representation of a pointer except on SPARC. You can ship it across the sandbox boundary, and BOOM.

My 6yo’s current favorite movie is Lord of the Rings, which she is unafraid of. But Crouching Tiger, Hidden Dragon drove her into the kitchen or under blankets. Paraphrased, she explained that the latter is morally ambiguous fighting where it’s unclear who the good guys and bad guys are. Fair, tbh.

Phew, guess we can all relax about Mythos then. You're all doing the same security best practices and overall level of software engineering quality on every one of your projects as curl, right? right?

daniel:// stenberg://@bagder.mastodon.social.ap.brid.gy · 3mo ago

#Mythos finds a #curl vulnerability yes, as in singular one. https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-vulnerability/

Arena allocators are pretty.

In a catastrophic typo, researchers ask superintelligence to optimize for CVE

Vibe coding parsers for untrusted data is a useful warning-sign-filled experience to have.

Any recommended tools for following some narrow area of research, to get notifications when interesting papers come out? By narrow I mean as specific as an arbitrary LLM prompt. (I realise one could code up such a system, but I am interested in what exists without me writing it.)

It's a shame: the 1.41e64 constant is _almost_ an LLM capability measure ("What quality expanders can the machines formalise?"), but alas it appears there's an enormous jump from what's already done (MGG) to Ramanujan graphs w/ deep number theory, with little in between.

Geoffrey Irving@girving.bsky.social · 5mo ago

As an exercise in learning recent Claude Code + Opus 4.6, I've formalised Seiferas's simplified construction of the Ajtai-Komlós-Szemerédi O(log n)-depth, O(n log n)-size sorting networks in Lean, using Margulis–Gabber–Galil expander graphs. github.com/girving/aks

I'm curious how long it would take someone to make an optimised SNARK system for Lean verification, based on Lean4Lean (arxiv.org/abs/2403.14064) and arkworks.rs. 🧵

studying chatgpt's busy beaver number: how long can it run and still halt. finished one prompt in slightly under 24 hrs. the response was just as unhinged as a human would sound after grinding that long

Your periodic reminder that software engineering is a mixture of easy-to-verify subtasks and hard-to-verify subtasks, and the fact that the machines are getting better at coding assistance should not be explained away as "they're only good on easy-to-verify stuff".