This is a card in Dave's Virtual Box of Cards.

Go read Peter Naur's "Programming as Theory Building" and then come back and tell me that LLMs can replace human programmers

Page created: 2025-04-27
Updated: 2025-07-10

For a copy of Naur’s paper and also my notes about it, see:

Dave’s Paper Notes: Programming as Theory Building

In this essay, I will perform the logical fallacy of argument from authority (wikipedia.org) to attack the notion that large language model (LLM)-based generative "AI" systems are capable of doing the work of human programmers.

LLMs have no "theory"

To understand Naur’s paper, you need to understand what he means by "theory", which comes from philosopher Gilbert Ryle’s The Concept of Mind (wikipedia.org).

The Wikipedia page I linked above is not a great place to learn this term and Naur’s use. Instead here’s an analogy by Ryle in the form of a rather long paragraph from The Concept of the Mind (Chapter IX The Intellect), starting on page 263:

"If a farmer has made a path, he is able to saunter easily up and down it. That is what the path was made for. But the work of making the path was not a process of sauntering easily, but one of marking the ground, digging, fetching loads of gravel, rolling and draining. He dug and rolled where there was yet no path, so that he might in the end have a path on which he could saunter without any more digging or rolling. Similarly a person who has a theory can, among other things, expound to himself, or the world, the whole theory or any part of it; he can, so to speak, saunter in prose from any part to any other part of it. But the work of building the theory was a job of making paths where as yet there were none. The point of the analogy is this. Epistemologists very frequently describe the labours of building theories in terms appropriate only to the business of going over or teaching a theory that one already has; as if, for example, the chains of propositions which constitute Euclid’s 'Elements' mirrored a parallel succession of theorising moves made by Euclid in his original labours of making his geometrical discoveries; as if, that is, what Euclid was equipped to do when he had his theory, he was already equipped to do when constructing it. But this is absurd. On the other hand, epistemologists sometimes tell the opposite story, describing what Euclid did in delivering his theories when he had them, as if it was some recrudescence of the original theorising work. This, too, is absurd. These epistemologists describe using a path, as if it were a piece of path-making; the others describe the path-making, as if it were a piece of path-using."

One must not confuse the artifact with the mind that produced that artifact.

I think something that clouds the whole business is that LLMs are capable of appearing to have a theory about a program. From Naur’s paper:

"…​theory is understood as the knowledge a person must have in order not only to do certain things intelligently but also to explain them, to answer queries about them, to argue about them, and so forth."

Indeed LLMs might seem to have this ability, but it’s, charitably, illusion. The appearance of a thing is not the thing. An LLM cannot obtain a Ryleian "theory" because, well, that’s impossible. LLMs "train" on bodies of work. By Ryle’s definition, the text output of theory-making can never be the theory itself.

Theories are developed by doing the work and LLMs do not do the work. They ingest the output of work.

Theory building doesn’t work backwards like that. In fact, that’s Naur’s whole point with this paper:

First, you cannot obtain the "theory" of a large program without actually working with that program. Reading the source or even documentation cannot, by some magical transference, give you the same mind that produced the program in the first place.

Second, you cannot effectively work on a large program without a working "theory" of that program. Naur explains:

"The conclusion seems inescapable that at least with certain kinds of large programs, the continued adaption, modification, and correction of errors in them, is essentially dependent on a certain kind of knowledge possessed by a group of programmers who are closely and continuously connected with them."

Neither machine code nor source code contain the wisdom of knowing how the program works. Nor does it contain the wisdom to know why the program was written the way it was instead of some other way that would have accomplished the same task.

Programming is not text production

Screenshot of quote below

"This observation leads to the important conclusion that the problems of program modification arise from acting on the assumption that programming consists of program text production, instead of recognizing programming as an activity of theory building. On the basis of the Theory Building View the decay of a program text as a result of modifications made by programmers without a proper grasp of the underlying theory becomes understandable."

This is my favorite part of Naur’s paper because it is so practical and visceral. It builds on the "theory" part, but also stands on it’s own.

Writing software is the production of code in the same way that writing poetry is the production of words. In both cases, the code and the words are the artifacts of the real work: building and maintaining a theory of the program.

As Naur writes:

"…​it is important to have an appropriate understanding of what programming is. If our understanding is inappropriate we will misunderstand the difficulties that arise in the activity and our attempts to overcome them will give rise to conflicts and frustrations."

At the time when Naur was writing,

"…​much current discussion of programming seems to assume that programming is similar to industrial production, the programmer being regarded as a component of that production, a component that has to be controlled by rules of procedure and which can be replaced easily."

But that was not true in 1985 and it’s not true now.

To believe that LLMs can write software belies a belief that the main (or only) activity of the programmer is simply to produce source code, which is wrong.

So

To replace human programmers, LLMs would need to be able to build theories by Ryle’s definition OR Naur is wrong about the nature of programming.

I’m betting neither is true.