I am working on a project I can’t say much about, except that it involves the systematic testing of Claude Opus as a coding agent. I got onto the project based on my background in software engineering. Although I identify as a tester, I was once a production coder, and I’ve kept up my coding skills pretty well, over the years.
Ostensibly this project requires me to act as a software engineer and spot problems in the behavior of the coding agent that is helping me on a pretend project. But to my surprise, most of my time is not spent acting as an engineer.
Instead, my time and energy is spent analyzing documents that describe behaviors and create a basis for LLMs to assign scores to those behaviors. I’m also engaging in legalistic and semantical debates with agents performing various AI “skills” that are helping me prepare the detailed and meticulous paperwork that goes with every bug report. These skills must agree with with my findings in order for my bug report to be submittable. So, there’s quite a lot of back and forth.
This is giving me flashbacks to expert witness work for court cases. I love expert witness work. In these projects, they dump a ton of paper on you, and you have to read all of it with an active and suspicious mind. You are looking for patterns and alert to risks. Then you perform experiments and/or write reports. Apart from domain expertise, you need a lot a patience for close reading and a flair for logic and rhetoric. Lawyers do exactly this kind of work, except, unlike an expert witness, they are experts in the law.
(If I had to go back in time and advise myself, I think I would have exhorted the younger me to go to law school.)
We simultaneously control and are controlled by AI.
That is really the key. We are becoming human agents not just in a loop, but in a web. I might even say it’s a web of loops. Our actions have consequences. We can redirect an agent, but then that may trigger other processes that we must operate or otherwise endure. Other agents may react, as well.
Although the project I am on is about testing, and all of the paperwork and analysis I’m doing is about bug reporting, I think my experience is part of a general trend that all technical people are experiencing as software engineers embrace the AIDLC (or more accurately, as the AIDLC engulfs the engineers): they will spend more time reading, analyzing, and arguing over documents that are not code. They will have to pick their battles as they navigate the consequences of pushing back on agent decisions.
They will evolve into generalist thinkers whose principal contributions are:
- to be active thinkers
- to be present as the product is built
- to be confident and assertive as they apply AI
- to mind the process, experiment with it, and fix it
- to be analytical
- to be critical
- to be responsible
- to be trustworthy
- to be a human that other humans can rely on amidst this heaving sea of computing
Leave a Reply