Understanding how software works September 28, 2026

But Sebastian, how can you claim we are losing our understanding of how software works when you are building on top of layers and layers of software, firmware and hardware yourself? I bet you don't understand it all the way down, either!

That's right, but

  1. I rarely need to because the layers have specifications and contracts and more importantly,
  2. I can go down the stack in principle because the transformations my code undergoes when traveling through the layers are deterministic.

Taken together, these two properties allow me to reason about the behavior of my code with high accuracy. If a layer below (or the transformation to it) fails to honor the contract, this is a bug that needs to be examined and fixed (and this is possible due to 2.). There are certain exceptions to the determinism, for example in multi-threaded programs, and, as anyone who has ever had the dubious pleasure to debug a race condition can tell you, these exceptions increase complexity by at least an order of magnitude, but thanks to 1. and 2. they are usually understandable at the layer you are working at.

Developing software with “AI” does not have these two properties. While the programming language of the generated code does have a specification, the transformation from your natural language description of the problem to solve to code in this language does not. All you are promised is that you will get something plausible looking. If it fails to generate what you expected, that's not a bug that can be fixed in any meaningful way. It cannot deviate from the specification because there is none. And good luck trying to understand why the results are what they are! Generation being nondeterministic by design effectively blocks any attempt at understanding what is going on.

Case in point: a few days ago, years after prompt injection was identified as a security risk, and after billions of dollars had been invested into development of LLMs and harnesses around them, someone managed to talk Meta's “personal AI agent”, Muse, into revealing its entire filesystem contents. And Meta didn't prevent it. Why not? Because the results of prompting an LLM are impossible to predict.

If I have a problem with my Java application that I suspect may be caused by the Java compiler, I may not be able to solve it myself today, but I can look at the code and come to understand it, or I can ask a JVM developer to take a look and we can solve it together (or pay someone to do it) because we as an industry understand these things. And once the bug is fixed, it usually stays fixed. And the same goes for any other layer. If I have a problem with an LLM, I have none of these options, because we as an industry don't understand those things – not even the people who built them. And up till now, attempts to fix even the most glaring problems have been spectacularly unsuccessful (see the filesystem exfiltration).

This is what we are losing.