1 Software Engineering with LLMs

One reframing reorganizes everything downstream. If the model is the program, the dataset becomes the source code, the loss curve becomes the compiler error, and the engineer’s job moves from writing logic to shaping the conditions under which logic emerges. The same shift makes context, not code, the real architecture of a modern AI system: you don’t instruct the behavior, you build the substrate it forms in. And underneath all of Karpathy’s teaching runs one conviction – you cannot trust a stack you can’t build yourself. So he built every layer from scratch, in public, one character of backpropagation at a time.

(…)

The throughline across every [Karpathy’s] move is not the employer. It is the insistence on understanding the machine all the way down, and then showing everyone else how.

– Blake Crosley, https://blakecrosley.com/blog/engineering-philosophy-andrej-karpathy

Is this book worth reading, now that we have LLMs that “will soon be writing most of the code”?

Let’s hear Rachel Thomas, working at the forefront of Deep Learning as part of a company called Fast.AI:

We all make mistakes and I am not trying to pick on the people listed above [whose predictions about AI turned out to be wrong]. However, it is important to ask if you want to stop investing in your own skills because of a speculative prediction made by an AI researcher or tech CEO. Consider the case where you don’t grow your software engineering or problem-solving skills, yet the forecasts of AI coding agents being able to handle ever expanding complexity don’t come to pass. Where does this leave you?

– Rachel Thomas, https://www.fast.ai/posts/2026-01-28-dark-flow/