The two rules of software development: - Up-front thinking is Waterfall; we can only learn by shipping - There's no room on the roadmap for rework, so everything needs to be done right the first time Bosses love AI because they think it can give them up-front certainty on the cheap. But it can't.
The fake double diamond is dead. The real double diamond is immortal.
AI is not a substitute for divergent thinking, and the false certainty it gives will never save you from having to iterate.
productpicnic.beehiiv.com
![These mystifying or obfuscatory metaphors about mecha-
nism (Christin 2020; Petrick 2020) bring to mind: “Any suffi-
ciently advanced technology is indistinguishable from magic.”
(Clarke 1973, p. 21; also noticed by Penn 2022) For example,
these systems are described as: “powerful digital minds that
no one — not even their creators — can understand, predict,
or reliably control.” (by a mix of CEOs and founders from
Elon Musk and Steve Wozniak, to academic experts in AI,
like Yoshua Bengio and Stuart Russell, in Pause Giant AI Ex-
periments: An Open Letter, Bengio et al. 2023). Others repeat
this same mantra, with OpenAI’s Bills et al. (2023) claiming
that:
we do not understand how [LLMs] work. [...] Our
explanations also do not explain what causes be-
havior at a mechanistic level, which could cause
our understanding to generalize incorrectly. To
predict rare or out-of-distribution model behav-
iors, it seems possible that we will need a more
mechanistic understanding of models. (emphasis
added, n.p.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:vsyj5jtayxcq5dw5mtja4xtb/bafkreigmcu3sd3qekglmqhnwspjeo5gs4j2uvluhyj4fyhz3hp6ko6floa)
![Even critical scholars repeat this, such as Jobin and Katzen-
bach (2023):
we are living in a time when the infrastructures
and institutions of our everyday lives are be-
ing (re)built at the hands of techniques that al-
ready elude popular and professional understand-
ing. (emphasis added, p. 48)
Journalists like Perrigo (2024), also follow suit:
Today’s artificial intelligence is often described
as a “black box.” [T]he inner workings of the AI
models remain opaque, and efforts to peer inside
them to check exactly what is happening haven’t
progressed very far. Nobody truly understands
what [the internals] mean, or how they work.
(emphasis added, n.p.)
Relatedly, the media and the public are then — perhaps
rightly — alarmed and follow up with questions such as:
“Without a thorough (mechanistic) understanding of the early
antecedents of these powerful future LLMs, can we realis-
tically expect to manage these AIs[...]?” (emphasis added,](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:vsyj5jtayxcq5dw5mtja4xtb/bafkreigb64yg5x5qqtgtqiky7j232qtp3afv3svcba64mjjacqu7ll5uiq)
