What is the strongest evidence for the "elicitation gap" reducing over time, e.g. thoughtful prompting helping less and less?
Daniel Paleka
@dpaleka.bsky.social
ai safety researcher | phd ETH Zurich | https://danielpaleka.com
Can LLMs figure out who you are from your anonymous posts? From a handful of comments, LLMs can infer where you live, what you do, and your interests; then search for you on the web. New 📄 w/ @SimonLermenAI, @joshua_swans, @AerniMichael, Nicholas Carlini, @florian_tramer 🧵
How well can LLMs predict future events? Recent studies suggest LLMs approach human performance. But evaluating forecasters presents unique challenges compared to standard LLM evaluations. We identify key issues with forecasting evaluations 🧵 (1/7)
why is it that whenever i see survivorship bias on my timeline it already has the red-dotted plane in the replies?
OpenAI and DeepMind should have entries at Eurovision too
3.7 sonnet: *hands behind back* yes the tests do pass. why do you ask. what did you hear 4o: yes you are Jesus Christ's brother. now go. Nanjing awaits o3: Listen, sorry, I owe you a straight explanation. This was once revealed to me in a dream
Quick sycophancy eval: comparing the two recent OpenAI ChatGPT system prompts, it is clear last week's prompt moves other models towards sycophancy too, while the current prompt makes them more disagreeable.
i was today years old when i realized the grammatical plural of anecdote is anecdotes, not anecdata. i dislike this finding
we are so lucky that pathogens, as opposed to political and religious memes, do not organize coalitions of hosts against non-hosts as an instrumental objective
are slot machines and the like so profitable because simplistic gambling is inherently very addictive, or because there has been a legible financial incentive for an entire industry to spend decades optimizing them to be addictive as possible?
TIL the concept of *epistemic hell*. standard Joseph Henrich example: in the ancestral environment, hygienic and food prep rituals determine survival, but no hunter-gatherer can possibly explain why. hence genetic selection for accepting of religious rituals and against reasoning
Why do meeting transcription apps (Fireflies, Granola) require Google Workspace accounts?
the rate of people's familiarity with Scaling Scaling Laws with Board Games over time is starting to look like the plot from Scaling Scaling Laws with Board Games
Recent LLM forecasters are getting better at predicting the future. But there's a challenge: How can we evaluate and compare AI forecasters without waiting years to see which predictions were right? (1/11)
LLMs rapidly improving at software engineering and math, given that the rate of improvement in ideation is slower, means you should be intentional about what value is gained from doing a highly technical project now as opposed to later
by interacting with LLMs you learn to offload thinking to them in ways useful to you, which is the second most important skill for the takeoff every time you talk to an LLM you lose decorrelation with LLM cognition, which is *the* most important skill for the takeoff
my New Year's resolution: don't work on a bigger project if there is not a clear reason for doing it *now*. disregarding the AGI timelines, the R&D acceleration is a clear reason against technical work where the discount rates on the final product are low
environments are a psyop a model can verify a proof or unroll a chess game. it can even eyeball if the code works the superintelligence loop will just be asking an AI agent to give feedback on its output by any means it can if the task needs a simulator the AI will write one
To those who believe Anthropic HHH incorrigibility paper implies sth for tamper resistance: I am willing to bet against. Just specify what exactly can't be done with the first open-weight model over some capability and jailbreak resistance threshold, given some compute budget.
NeurIPS test of time award talk on GANs mentions the paper was done in 12 days, from idea to submission. Two days more than Javascript, but slightly faster than the first versions of Git or Unix.
I'm at NeurIPS, do reach out if you want to grab a coffee!
they are doing gain of function research on Whova attendees order hacks now
TIL that the atmosphere blocks basically all electromagnetic radiation, except three small windows: one for visible light, one for cooling the Earth, and one for radio waves. Earth is the USA of planets.
guys literally only want one thing and it's the patient work of sitting down every day and reading papers until their eyes bleed, and hoping that something good comes out of it someday