In the last two to three decades, we’ve seen the same behavior in enterprises: namely, fragmentation, followed by sprint innovation, followed by keeping the legacy for potentially the next cycle.

For most of the last two decades, enterprises have lived comfortably with fragmented infrastructure. You often have legacy systems living next to islands of microservices, inefficiently allocated resources, and, most importantly, heavily outdated security. Key examples include: running old versions of software without long-term support (LTS) that invite frontier AI attacks. Core banking migration that locked customers out for weeks, later attributed to inadequate testing and an underestimated integration surface. The list goes on.

Now, every CIO has an AI mandate and as a result it becomes a question of framing it as an AI program. An initial attempt might have it run alongside existing workloads. The GPU servers arrive and because the existing platform, storage, and network wasn’t designed for AI workloads, they end up getting racked as a separate environment with their own switches, storage, and management plane. As a CIO, you’re investing in a new infrastructure island. Capex and opex gets redirected here. Meanwhile, you already have a large data center estate that’s delivering on day-to-day applications.

The existing environment usually also have years of compliance, monitoring, and automation. Some of it rooted in legacy practices, some of it rooted in mistakes and priority tickets that one would still want to avoid. Because the new infrastructure is now dedicated to these new AI workloads, it idles between the different phases of the demo and build while still consuming a lot of these resources. Data has to be copied to the island before a model can use it and copied back before anyone can act on the result, and every copy is a pipeline to maintain and secure.

That is the crux of your tech debt: the platform underneath the business cannot host the new workload so the workload got its own platform and the organization now pays to run, secure, and staff both. IBM’s Institute for Business Value surveyed 1,300 senior AI decision-makers in 2025 and found that 81% say technical debt is already constraining AI success, and organizations that ignore it see AI returns fall by 18% to 29% while schedules stretch by 15% to 22%.

The warning signs

Tech debt rarely announces itself. It usually shows up as four measurable symptoms and to each one can attach an AI failure:

• Time to value keeps growing. Whenever you’re deploying new services, AI or otherwise, how quick are you seeing production outcomes? If the number increases as complexity increases, and at times exponentially, it’s usually an organizational issue or tech debt. You have to navigate multiple technologies, processes, and approvals to get complex things done, which means that by the time you need to have AI in production it will be easily months if not a whole year at times and it becomes a bespoke setup.
• Time to feature vs feature stability. Google’s DevOps Research and Assessment (DORA) research across nearly 5,000 tech professionals found that AI amplifies whatever is already there: good platforms get better, weak ones get less stable. AI adoption in that data correlated with higher throughput and, at the same time, lower delivery stability, more change failures, and more rework. If teams are shipping more commits and fewer working features, the pipeline is the bottleneck, and adding more AI to the front of it makes the queue longer.
• The security gap mapping. The advent of agentic AI has triggered numerous “AI service accounts” with credentials inherited from other similar users. But an AI agent is hardly a user that can be trusted (or fired). In most estates there is no clarity of ownership of AI agents, an inventory, who’s responsible for the behaviors, or what they should have access to. And it’s one of the biggest signals of tech debt because it’s how we build infrastructure and access before. The problem is when an AI agent carries the same over-broad permissions, you’re inches and seconds away from a breach.
• Infrastructure management is difficult. For a long time now, IT teams have continuously added more technological solutions to their infrastructure. One IT admin has a span of tens-of-pieces of tech. In the absence of automation, this becomes a lot of manual work, on systems that are constantly changing, with shorter deadlines. And then whether it’s the desire to deploy faster or to resolve that priority-one incident quicker, the apparent solution is to simply dump a bunch of logs into a public AI tool with no record of what was sent and where. This, in and of itself, is a sizeable tech debt flag because it shows the seams of a system that can no longer easily be managed, secured, and counted on.

The playbook: Where to start and what to fix

It’s not that CIOs aren’t already aware that they own the tech debt. The question is usually where to start: every IT organization has a significant backlog and these AI initiatives are usually coming ahead of the line in a (sometimes misguided) desire for immediate modernization. Some pieces might remain tech debt for a very long time, but that should be a liability assumed rather than ignored.

As for where to start:

1. How is the tech debt impacting the AI roll out. Given the business outcomes and use cases outlined, what are the necessary data sources, integrations, security, and compliance policies that are needed? Can the current system support them without effort? What would it take to upgrade them? Two to three focused work items for a small tiger team could make a world of difference and can be a repeatable action across the different business outcomes.
2. What/who funds this. The same IBM's research found enterprises that account for the cost of addressing debt in their AI business cases project 29% higher returns than those that do not. There will be an additional cost – own it, plan for it, and make it work beyond just the one AI initiative. Whatever modernization initiative should be broader than the one AI use case.
3. Unleash your data. Or at the very least, start with it. Most successful AI use cases make use of a lot of data (unsurprisingly) – but solving the data path problem and ensuring that the systems of record that hold that data aren’t siloed or hard to reach or not part of the pipeline should absolutely be a priority. In an HBR Analytic Services survey commissioned by data platform vendor Cloudera, only 7% of enterprises said their data is completely ready for AI and 56% named siloed data as their top obstacle.
4. Security and identity are key. Treat AI agents like rogue teenagers that need to have heavy guardrails and lots of monitoring. This means that fixing identity management and access becomes one of the key hurdles to resolve before scaling any agentic AI work.
5. Pilot, succeed, repeat. Working with AI is fundamentally new – there are no decades of best practices to rely on. But understanding what the business needs, piloting on those specific use cases with clear outcomes, succeeding at a small array, and especially understanding the signal-to-noise ratio (what worked/what didn’t/where we were not productive) becomes a key part of success. This will also keep the infrastructure churn at a minimum because it will allow for testing and validating and setting up better foundations (see point one again).

Conclusion

What it all boils down to is – take your most important AI use case, the one that’s the most relevant to your board today and investigate with your teams what would it take to make it successful. What are the current potential pressure points and it will absolutely come with a litany of “reasons why we can’t have it fast and good,” and then start building a plan from there. Is it security that’s the biggest blocker? How do we ensure this outcome stays compliant and secure? Is it the data pipeline – how can we bring some of this data together still while not impacting existing production systems?