Don't worry, I'm not leaving software. This time. Instead, I want to discuss a specific scenario in the space of possible futures for AI-augmented software development, which I believe has been overlooked and which I believe there is some evidence pointing towards.
For the sake of clarity and completeness, I want to repeat this disclaimer: In this piece I am explicitly talking about the tech and the low-level details of what do you actually do to use these things. In particular, I am not discussing the ethical, political or resource aspects here. Neither am I going to say much about the worrying (and still very much extant) possibility that these tools have in-built and irreducible cognitive decline effects.
With that out of the way, let's begin by discussing the Retort tool produced by Adrian Cockroft (formerly AWS, Netflix, etc) to focus on the technical and delivery aspects of AI-augmented software engineering that actually matter to teams - questions like: "Is the newest model is worth 4× the cost for Rust? How reliably each model gets a Go MCP server completely right? or How long does any of this take?"
The whole piece is worth reading in depth, and keeping an eye on as Adrian adds to it going forward. However, these conclusions deserve particular attention. I want to draw your attention to the fact that being able to weigh these factors is currently still dependent on the engineer having sufficient skill and experience to conduct a proper evaluation.
A second aspect to be aware of is that the matrix of experiments Adrian defines is framed in terms of a methodology derived from manufacturing engineering. This seems to me to be a significant step towards treating "code as industrial product" - with e.g. defect rates that can measured from the outputs of an automated process of code extrusion.
The second piece of the puzzle comes from considering the relative trajectories of proprietary and OSS models - as explained by Steve O'Grady from Redmonk. The trend and implications are clear - OSS models are catching proprietary (aka "frontier") models faster and faster with no sign that a proprietary model provides any kind of moat even in the medium-term.
Next up, two interesting perspectives from the Honeycomb team. Firstly Liz Fong-Jones discussing the detail of how their teams have implemented a development practice that has been reoriented to use genAI tools. In it, she provides a frank, and refreshing look at what works, what requires a realignment of tools and processes, and some important caveats on where this approach has limitations. Again, the whole piece is worth reading in detail, but the headline takeaway is that re-engineering their core teams and processes and ensuring that those processes are easily consumably by genAI tooling buys a ~2X uplift in delivered features.
Secondly, from a month or so ago, here's Honeycomb CTO Charity Majors writing about the contrasting perspectives between "enthusiasts" and "skeptics". This is less directly relevant to today's topic but I think it's a good lens to think about your engineering teams and their various viewpoints. Personally, I would be worried if I didn't have both opinions represented among senior technologists in the group.
To add to the Honeycomb and other public examples, I can add some data points from private conversations with engineering leaders at major tech companies. The headline figure of 2-2.5x improvement is consistent, with some variation on the amount of team restructuring and base re-engineering required. A figure of 10% (of total pre-AI development budget) needed in additional spend on tokens and AI tools seems relatively common as well.
Of note is what I'm not hearing: Any sizeable engineering department claiming a much higher number for improvement - there just aren't any reports of the orders-of-magnitude improvements that were being bandied about 12-18 months ago. Is it the case that now the dust has settled and the tools are closer to being widely deployed in Production, that there's more devil in the detail - and that the overall upside is just not as big as first hoped?
To help connect the dots, it's worth taking a look at the leaked OpenAI financials obtained by Ed Zitron and verified by the FT. This is a remarkable set of numbers as it shows vastly widening losses and a need to continue to raise VC to continue the current R&D trajectory, even as revenues grow. It's not straightforward to figure out "How much would OpenAI have to raise prices to break even?" but the most obvious route provides a figure of an 8X price increase to OpenAI customers.
This is where the rubber hits the road: 2-2.5X SDLC outputs for 10% extra cost is a bargain that any engineering leader will be eager to take. But what about 2X outputs for 80% extra cost? That is much less clear-cut, especially given the variability (and vendor risk) of those ongoing costs, and the noted need for team and platform reorganisation to be AI-consumable.
There's one more under-discussed aspect: Much of what we've discussed here is focused on deployments by and for elite teams. What if it's only the elite-good teams that can benefit in the short-medium term? What happens to the AI coding companies if their currently addressable market is not all developers but only e.g. the top 25% of teams (per Liz's comments)?
Pulling all this together, let's talk about the "So Long and Thanks For All the Fish" scenario:
- Proprietary "frontier" models cannot stay ahead of OSS models long enough to recover costs, never mind profit
- The sunk cost capex is never recovered.
- Proprietary models as a category essentially become non-viable ("So Long")
- The current (or some near-futures) state of proprietary models essentially becomes a starting point for the evolution of OSS models ("Thanks For All The Fish")
- Further development of OSS models is undertaken as loss-leaders, or by non-profit action by nations states.
- Large enterprises deploy OSS models on their own private or hybrid cloud to fully control opex and reduce vendor risk.
Is this a plausible scenario? Perhaps - and it's not just me that thinks so. Here's a senior OpenAI bod openly advocating for the US government to put its thumb on the scales to help prevent this scenario. So much for the Free Market!
In summary, a lot of what has been written about these technologies has been written from the perspective of experts, not the broad mass of rank-and-file developers, or average-performing teams. My sense is that the technology is still very much maturing, and as of right now it is still not universally accessible and may not even be completely mainstream yet. Much of this depends upon how much extra the AI tooling and tokens actually costs, as a percentage of total SDLC spend. So far, many organisations do not seem to have a good handle on those costs or their exposure to sudden increases in AI vendor pricing.
