Comprehension Debt and the "Machine Stops" Risk in the Age of AI

[image]

TL;DR: While AI agents easily absorb routine clerical tasks and agency CV filtering, domain-critical areas like infrastructure and high-stakes problem-solving still require deep human expertise. Over-reliance on AI leads to a deskilling trap where prompt engineering replaces real comprehension—leaving organisations vulnerable if the underlying tech fails.The Displacement Realist: Acknowledge the loss of routine clerical and entry-level jobs (recruiting, agency filters, basic office tasks).

The Displacement Realist:

AI is changing the way we work and live in ways that we only dreamed of. Around 2015 I literally had a dream, in which I was on an IT call out, with a robotic artificial intelligence. In the dream,I communicated with it to agree and action fixes to broken infrastructure. It is entirely possible to see how this could be reality in a few years time. Coworkers may well be synthetic software and there could be a mix of human and robots, working together.

We may have an optimistic view of this future, where respect for the current, human workers will prevail. The transition to a hybrid state will be ordered and beneficial. Or, perhaps not. Entire roles in society are are about to be completely replaced by an automated, agentic workforce.

In my experience, IT jobs have always gone through agencies of one kind or another. Typically, the recruiters are themselves not as technically qualified as job candidates. This disparity increases, the more advanced an engineer become in their roles over time. The recruiters persist as younger in age, their roles are high turnover and still, less experienced in IT. Recruitment agencies have resorted to software based, key word searching and indexing techniques to sift C.V's and Resumes. The nature of recruitment has changed over time to use social platforms but this has generally been their working practice. With AI, IT recruitment is changed forever. What has been a typically young workforce of recruiters with software tooling and in some cases little experience as IT engineers can be replaced, entirely. An AI can do that.

The Infrastructure Reality Check:

Employers typically want workers to 'hit the ground running'. In other words, they want experienced staff that they don't pay to train. You, as an employee, need to somehow self train. Experience is a way to become 'senior' in roles.. We are hearing of junior programming roles being cancelled by companies, as senior programmers can be as productive without them, when using AI. Despite best efforts to do so, up to now at least, experienced programmers have not be entirely replaced with AI.

Similarly, in infrastructure, some have tried and failed to manage their infrastructure entirely with AI. It turns out that when a company database and all its backups are deleted and live applications get re-written in production by AI robots let lose with root access, this is not a good time for the companies that do this.

Infrastructure is deterministic. It is not like software that can remain hidden from view in test. It is entirely on view for customers and stakeholders to see. In this highly transparent perspective, infrastructure has to work. So far, experienced humans are needed to make that happen. They may as engineers, use AI to assist them but when entirely replaced by AI, bad things can happen.

Comprehension Debt

Despite the apparent capabilities of Large Language Models, as vibe driven coding has shown, poor prompting leads to low quality outcomes. Garbage in, garbage out.

We are becoming 'prompt engineers' so as to cleverly craft prompts, specs, agent files, rules and so on to get better outcomes with programming tasks.

From the extreme of vibe coding to structured, carefully designed spec driven prompt engineering, we are still at risk of losing something. Spending 30 minutes to craft a prompt that tells the LLM to save us 30 minutes of manually writing the code could mean losing the ability to write that code. Worse still, if we cannot write the code in the first place, we may never learn and we become entirely reliant on an LLM to do it for us.

We will find ourselves on a sliding scale, a spectrum if I may say, of comprehension debt. In which, we no longer know how anything works. The only time we as humans, interacted with the design process was as a prompt writer.

Exoskeleton vs. Reverse Centaur

Recent studies have suggested that developers think that they are more production with AI than without but in control groups, when compared, it is found that they are mistaken. Non AI generated code takes the same time as AI assisted code. I find this hard to believe as in my experience, AI gives me bionic abilities to create code in multiplemulitple languages, some of which that are outside my 'core' knowledge.

What I think is happening here is that folks that use AI can get pulled down into the weeds so to speak, looking for the perfect way to engineer a solution. I've found that LLMs are predisposed to presenting every which way to solve a problem, over complicating and over engineering almost every time. They cannot 'see the clear path' to the simple answer. They can echo your decision back to you once you apply an 'Occam's razor' approach but rarely can they identify simple, efficiency.

So I believe that if we can remain engaged with our code, we can keep our eyes simple enough to stay productive and AI can become like an exoskeleton, giving us abilities that we most likely have already, but amplified.

But this is when we have the 'luxury' of choosing how to use AI. If, conversely, we are told how to use AI by our owners / employers, who do not value our work and see salary payments as a form of tax to their bottom line, then we have a different problem.

I have on my reading list The Reverse Centaur's Guide to Life After AI by Cory Doctorow who I've been watching in interviews on line. His hypothesis is that the reverse centaur is that humans are conscripted to work for AIs not with them. The result is unsustainable. Burnout is inevitable, the workplace inhumane. Cory suggests that this is becoming common practice.

So if you can put yourself into a place where you can decide how you choose to use Ai in a way that is productive, self fulfilling and supportive, so be it. I would avoid the reverse centaur situation if at all possible.

The Machine Stops

I'm reading The Machine Stops · by E. M. Forster - a short piece of science fiction I find intriguing in its foresight. Whilst I agree with a lot of Cory's ideas, I thing the bigger issue for us all, which ever side of centaur we find ourselves, is the idea that AI will never stop.

In business continuity, a disaster can happen in may ways. Often, the one we least expect can have the most devastating outcome. I've heard folks say, that ( a missile hitting my data centre .. ) will never happen, so why should I to plan for it. But what actually happens is, a burst water main means nobody can get into the head office one day and the business grids to a halt anyway.

Loss or denial of service in some way is more likely to happen and in ways we cannot all of the time predict or plan for.

A simple scenario and one which is entirely predictable though, is AI companies increasing prices over night. This is happening already.

Some companies are finding that monthly subscription plans at $30 per seat are now 'usage based' and they have costs projected at 100's times this figure at their current usage. Before this happened, some companies were using AI token use as Key Performance Indicators for their staff, where if you were not using enough tokens, you were not performing as an individual. Unsurprisingly these objectives are being quietly rolled back now.

But I would suggest it wise to plan for more astringent scenario where the machine ( AI ) stops entirely for us. We are left having to 'go back to pen and paper' so to speak. In other words, do everything ourselves rather than rely on there being a robot slave to do things for us.

It is this epoch like event that could have severe consequences if we become all reliant on AI, having forgotten how to do work.