Nvidia earlier this month unveiled CUDA Tile, a programming model designed to make it easier to write and manage programs for GPUs across large datasets, part of what the chip giant claimed was its “biggest evolution” since the platform’s 2006 debut.

While many Nvidia-watchers naturally gravitate toward updates to its shiny hardware, this update to the often unfairly overlooked software backend turned several heads, including Jim Keller, one of the great living semiconductor architects who helped design silicon solutions for Apple and Tesla.

In a post on X, Keller expressed curiosity about the launch of CUDA Tile, questioning whether Nvidia just ended “the CUDA ‘moat,’” adding: “If they move to tiles like most other hardware, the AI kernels will be easier to port.”

The proverbial moat the now Tenstorrent CEO spoke of comes as Nvidia’s CUDA has long been the dominant software platform for AI and high-performance computing (HPC). Combined with its highly sought-after hardware, Keller’s comments could be construed as a means of making it difficult for rivals to offer comparable solutions

But with the launch of CUDA Tile, has the situation changed? In a frank conversation with SDxCentral, Stephen Jones, one of the chief architects of CUDA, offered a view more around the idea that CUDA Tile is an update about democratization: making it easier for developers, whether newbie or not, to create with heterogeneous computing.

“I never honestly think about any moat or anything about that with CUDA, my goal is just to get you in parallel,” Jones said. “The reality is, I think that your program comprises lots of different pieces, and you're going to need all the tools, but if I can just reduce some of that cognitive load, then I'm winning."

Sticking with the castle analogies, then, think of CUDA Tile as akin to a partial opening of a drawbridge, with Nvidia making it easier to get into its domain – though the walls encasing it will naturally remain.

But Tile marks a myriad of recent ecosystem shifts for the notoriously proprietary vendor, with offerings like Grove and even NVLink Fusion showing Nvidia now wants others to play with its toys via the pretext of offerings so comprehensive that ease-of-use becomes a competitive advantage, not a vulnerability.

How can CUDA Tile boost AI development?

The concept of Nvidia’s latest CUDA update is a simple one: instead of managing thousands of low-level threads, it allows developers to describe operations over tiles of arrays and tensors rather than manually orchestrating hundreds of thousands of threads. These tiles are essentially big blocks of data that can then be mapped automatically to tensor cores for better performance. It brings together data collected by the compiler and then figures out the best way to run the code on a GPU.

Before CUDA Tile, developers had to do this manually across thread blocks, warps, and low‑level tensor instructions – a process that took a considerable amount of time and skills that likely discouraged many developers wanting to take advantage of the AI boom and begin building.

“It’s about the 80/20 rule,” Jones said. “You spend 80% of your time on the last 20% of your stuff. So the sooner you can get to your first 80% the more time you can spend on the optimization. Not everything needs that last 20% of optimization, and so the goal with tile is to get you to that 80% as quickly as you can.

“The goal has been simplification, time to performance, but also consistency that you can have one program that works everywhere, that when you have to move to the new hardware, your program starts just working, and then it's just a matter of retuning,” Jones added.

CUDA Tile initially works in Python, arguably the most widely used programming language for AI, data science, and back-end development. But it is coming to C++ sometime in 2026.

Jones explained to SDxCentral that the decision to lead with Python was because CUDA Tile was “a really natural match.”

“Python programmers aren't thread programmers, but they are array programmers with things like NumPy,” the architect explained. “This is intrinsically more intuitive to Python programmers. Array management programming is common in every programming language, but by keeping the abstraction at the array level, we've taken a lot of the cognitive burden off the user, allowing them to focus on parallel algorithms rather than the details of mapping it to the hardware.”

The democratization gambit

Beyond making things simpler, CUDA Tile is also about improving performance and portability.

Removing a considerable cognitive and boilerplate burden for programmers while still providing a faster, more efficient path-to-peak performance. Add to that, Jones spoke of how developing CUDA focuses on what he describes as the “principle of least surprise,” that if you “just try something,” it should behave how you intuitively expect. This worldview, aligned with the purpose of what CUDA Tile is actually trying to achieve, ultimately lowers the learning curve for new developers, getting more people to usable performance faster.

“Performance comes in many forms, even just the simple act of mapping data to threads, you might need many elements per thread. So the more the compiler can do for you, the better," Jones explained. "And by expressing your program at a higher level of abstraction, saying, you know, 'here are the vectors I'm trying to add', rather than 'here are the scalers and the pieces that I’ve broken into already,' you're giving the compiler a lot more knowledge to carry that through the program and make better optimizations.”

On portability, there’s also a future-proofing angle. As GPUs evolve, especially as part of Nvidia’s annual cadence, Tile is actually designed so that the same high‑level code can follow along, with the compiler doing the heavy lifting of retargeting to new architectures.

While it’s currently only supported on Nvidia Blackwell products, specifically those running compute capability 10.x and 12.x, Jones told SDxCentral that CUDA Tile will be coming to more architectures in the near future.

“The next release is coming out with Ampere, the one after that's coming out with Hopper,” Jones revealed. “When we release anything in CUDA, we make sure we release it back to the earliest GPU that can support it. Unfortunately, Turing cannot be supported; the tensor core is just too old to do the tricks that we're doing to make Tile work.

“Of course, every time a new GPU comes out, this is a core piece of CUDA," Jones added. "This isn't just a library. This is a core piece of CUDA. And so it runs everywhere that CUDA runs, and onto the earliest GPU that we can possibly make it.”

From Ampere onward, then, including the upcoming Rubin platform, the idea is that the same Tile code should compile and run across architectures, with developers mostly retuning parameters like tile dimensions rather than restructuring entire kernels.

Helping CUDA Tile to ensure its portability across generations is how it’s built. It’s split into front ends and a shared MLIR‑based backend. Once the backend knows how to target a given GPU family, all languages that plug into it benefit.

The real moat reveal and CUDA’s next frontier

For all Keller’s comments about Nvidia throwing away its apparent “moat” following the launch of CUDA Tile, Jones was honest in his view: “I think Nvidia would probably kill me for this, but I'm actually a true believer.”

“I love you no matter how you're getting parallel, as long as you're doing parallel programming … it’s very much about getting the power into your hands to do this," Jones said. "But an array-based model is not every program. This is more about extending CUDA. It's not built on top, but alongside, because here are parts of your program that are still going to want thread-level control.”

Put more succinctly, the CUDA architect said: “The platform, where all the pieces work together, makes the whole greater than the sum of the parts.”

Development behind the scenes at CUDA never stops, of course, and there’ll be plenty more updates following version 13.1.

And up next for Jones and the team is the exciting world of multi-GPU and multinode – where multiple systems within a single server or servers are employed to tackle massive tasks like training large AI models via a combination of parallel processing and fast interconnects.

Jones outlined the scale of the next frontier of focus for his CUDA cohort: "There is no single GPU machine anymore. Everything is scale. GPUs have already got a quarter of a million threads. There’s already a big investment in multi-GPU, and I think it’s going to be key to try to democratize that, trying to be more intuitive and more productive because everybody who's building anything over the next few years is going to end up building it multinode.”

For now, CUDA Tile is the answer to one problem facing today’s developers. The next wave will see much of the same questions of scale, but will have teams from Nvidia looking to make that level of parallelism feel as natural as writing NumPy in Python. At the point where we get around to the proliferation of multinode systems, the real defensible advantage won’t just be about hardware or software “moats,” but about just how quickly developers leverage both more efficiently to power effective AI systems.