Most models these days are perfectly competent agents, and my daily frustration is that they are all completely inept communicators. Remember when labs were trying to solve Diplomacy? It feels like they've completely forgotten that there's a human on the other end that needs to understand sometimes.

I consistently underestimate how difficult it will be to write anything with a GUI using a language model. You immediately end up inside the hot loop trying to articulate why everything feels bad and looks weird to a model with no intuition for those ideas.

o1-preview beat my family zero-shot at Codenames when paired with the weakest guesser. Clues weren’t amazing when picking word pairs for itself, but very impressive ability to relate words when given a collection and asked for a clue that relates them all.

jax.config.update(“jax_explain_cache_misses”, True) is an extremely useful tool for figuring out what parts of your Jax program are getting recompiled and why.

A research idea that I would love to see someone take further is “reverse” positional encodings that count down to an EOS token. The training setup is easy and it could offer an interesting way to bias the length+style of LM responses at runtime. Want ~100 token response? Count down from 100.

Had a great week at SC24! Very exciting to see the ways that diffusion and other ML algorithms are transforming scientific computing. The tooling community also seems to be making big strides to make HPC more accessible to a broader scientific community. Look forward to next year!