Key Takeaways:
- Compute clusters bring different parallelization paradigms, such as Tensor parallel, data parallel, context parallel, and pipeline parallel, each with a different network topology requirement.
- Inference disaggregation means that one cluster has a specific set of hardware/software to do the pre-fill stage, another to do the decode stage, and another fully compute cluster to do the execution by the agents, and these are all stitched together, often by Ethernet, but other protocols also exist.
- In moving from training to inference, a heterogeneous cluster environment helps bring the token cost down, and software becomes critical to utilize the heterogeneity and optimize across the cluster, especially where there is not a single company doing the end-to-end flow.
Experts At The Table: Semiconductor Engineering sat down to discuss changes in AI data center architectures with Satadal Bhattacharjee, global head of cloud and AI infrastructure silicon at Arm; Ashish Darbari, CEO of Axiomise; Moshiko Emmer, distinguished engineer at Cadence; Sharad Chole, chief scientist at Expedera; Cameron Brunner, head of HPC development at Siemens EDA; and Sumit Vishwakarma, director of strategic marketing at Synopsys. What follows are excerpts of that discussion. This is the last of a 3-part series. Part 1 of the discussion is here. Part 2 is here.
L-R: Arm’s Bhattacharjee; Axiomise’s Darbari; Cadence’s Emmer; Expedera’s Chole; Siemens EDA’s Brunner; Synopsys’ Vishwakarma.
SE: Compute clusters comprised of multiple processors acting as a single computer present many design challenges around networking, thermal management, power delivery, and more. Which companies are leading the innovations to make this solution viable?
Brunner: We see hardware vendors and large system integrators developing solutions for how to put the network together. The software can then help place the workloads to maximize utilization of these interconnects. Hyperscalers have been doing a very good job exposing these types of concepts, so you can build environments there that work this way. Most of them have a concept of a placement group where they’ll put the VMs (virtual machines) together in a way that they’ll have a high-speed interconnect. It ends up being a big problem. When you go off of one machine, this is a big problem if you want to try to get that linear kind of scaling as you add more machines to your app.
Chole: A lot of it has to do with the workload. I come from an NPU and AI background, and given how the models have increased in size, the memory became one of the critical bottlenecks that needed to be utilized with the computations. That’s where the cost of HBM, and how the HBM is utilized, become very important factors for GPUs. While you have the parameters that are a static part of the models, there is a dynamic part of the model, which is the context. For example, how much your memory is remembering, how much you’re processing, is connected with how many requests you want to process. On the edge, it’s a simple story. You get a couple of requests at once. But at the data center level, you’re trying to maximize the entire utilization at the API level, and for that it is about how many requests you’re processing per second. When you put this whole thing together, you basically end up going into different parallelization paradigms, such as tensor parallel, data parallel, context parallel, and pipeline parallel, and with each there is a different requirement about topology. The pipeline parallel goes from point A to point B. Tensor parallel is about all-reduce and then broadcast back again. While defining the cluster, keeping this in mind becomes very important. That’s why NVLink is one of the important parts if you have to scale out to tensor parallel topologies. The same situation applies to storage, and similar things apply to how memory transfer needs to happen.
Going back to how CPUs are utilized, tool calls and how the tools are being utilized on the CPU make the agent take workloads slightly differently. It’s important to understand how those tools are being executed. For example, if it’s a completely agentic workflow that you’re running in a data center, you have to orchestrate that. This means that, somewhere, the agentic workload becomes more like a cloud management problem where the tool applications have to be provided and the MCP (Model Context Protocol) server has to be responsive and deployed. If there is enough API usage for your workload, optimizing the cluster becomes beneficial. So we have to think about all possible parallelization strategies to get the scale up. How you go from one GPU to eight GPUs and still get close to 8X performance is always a challenge.
Bhattacharjee: That’s right. One of the trends we are seeing, which Nvidia started, is disaggregating the inference pipeline. In inference, there are a few stages. One is called prefill, where you enter a prompt and it’s just trying to figure out what you’re asking so that it can take action. The prefill stage is extremely compute-intensive. Recently, Nvidia announced that its Groq 3 LPU (language processing unit) will be used in the prefill cluster, showing more than a GPU is required to do some of these tasks. Then there’s decode, where it actually does the task, or creates the response that will be generated and shown. Then with the agents coming in, there is the tool calling or executing the task — for example, booking the Uber ride or making a hotel reservation.
We are seeing that with inference disaggregated, you have a cluster with a specific set of hardware and software to do the pre-filled stage. Then, you have a cluster to do the decode stage, and then you have another fully compute cluster to do the execution by the agents. And these are all stitched together. They’re all communicating with each other, most likely through Ethernet right now, but each of the clusters has a different mix of software and hardware to do the function that they do best. That’s going to be more of the norm going forward, because until now every AI problem was solved with a GPU, and the industry is recognizing that every nail doesn’t have one hammer. GPUs are efficient at doing math calculations and all of that, but when you go into inference, it has completely different workload characteristics, so there’s a need for more diverse hardware. It’s called a heterogeneous cluster, and the software has to understand the heterogeneous cluster. There’s a huge software challenge associated with this, because most of this hardware is now going to come from different companies, and then the software has to make sure that it is seamlessly utilizing each piece of hardware for its purposes. The neocloud companies are involved here. For example, DigitalOcean announced their deployment of a five-layer architecture for inference, specifically tuned toward using heterogeneous clusters underneath, with mention of AMD and Nvidia. They said there will be new hardware that they will incorporate as well. That’s the trend — to bring efficiency out and reduce token costs.
Darbari: A cluster changes the design problem from building a powerful machine to engineering a coordinated distributed system. In a single server, the main concerns are typically local compute, memory, and I/O balance. In a cluster, performance depends heavily on workload partitioning, communication patterns, and whether the interconnect can carry synchronization and data movement efficiently. It is also important to recognize that the cluster has a communication hierarchy. At the lowest layer sits chip‑to‑chip connectivity, whether between dies in a package, accelerators on a board, or devices within a node. Industry standards such as UCIe are emerging to make those die‑to‑die links more interoperable and predictable across vendors, which is critical for building larger coherent nodes out of multiple chiplets.
SE: If CPUs are doing heavy lifting in the AI data center for agentic AI and AI inference complex reasoning loops, maintaining context, routing branching instructions, and other tasks that are difficult to parallelize, what additional things do GPUs need to do besides matrix multiplication? Do they pick up another task when the CPU takes their workload?
Bhattacharjee: If you think about inference, there’s a reasoning part where you need to figure out, based on the prompt, what action needs to be done. The reasoning step is extremely compute-intensive. That’s where the accelerators come in, and I’ll use the word accelerators instead of GPUs, because it doesn’t have to be a GPU. It can be a custom ASIC that does the reasoning function. When it comes to the model doing the reasoning, there’s a lot of matrix multiplication required, and that’s what the accelerator is good at. But when it comes to orchestration tool calling, which is basically agents taking their actions, those can all run on CPUs. And this is done in a very tightly controlled way so that the orchestration is done on what workload runs on CPU and what runs on the NPU, and that’s where the software plays a very critical role.
Emmer: What changes in agentic and reasoning-heavy inference is not that accelerators become less important. It’s that the pipeline becomes more segmented. Some steps are control-heavy, stateful, and branchy. Others are still massively parallel tensor math. A hybrid architecture is naturally well suited to that because it lets you compose different engines and memory structures into one coherent system, rather than forcing one compute type to do everything inefficiently.
Brunner: The concept of heterogeneity inside of clusters is going to continue to grow. We have use cases that we’re looking at in our environments where we have to also start integrating quantum machines, which will be awesome. [Clusters] are a whole different way of feeding those and keeping them integrated. There’ll be one common connected platform with the purpose of solving a business objective, and that’s what we’re really talking about here — serving the needs of the operators using the hardware they have, within their capability, to get the best outcome for their business and their users. Business will drive the solutions that provide the most efficient use of the hardware to solve problems.
SE: What is the preferred interconnect cluster-to-cluster?
Brunner: The big competitors to Ethernet would be Slingshot (Hewlett Packard Enterprise’s high-speed, Ethernet-based network technology) and InfiniBand. In the Top500 (list of supercomputers), there’s a breakdown of 100 Gbps Ethernet and InfiniBand on those chains. The highest performing one is Slingshot. Ethernet has been the standard interconnect for a very long time now. There’s been investment over the years where a team will come in and make a new dedicated protocol to do something that Ethernet can’t do, but history has shown that just motivates Ethernet to improve. It just keeps stepping up. It has more use, but this proprietary tech is usually a little bit better, so it ends up being a tradeoff. If you want to do Slingshot, that’s an HPE Cray system (for high-performance supercomputing), so that’s who you have to get it from. Ethernet is a much broader standard, so it will come down to the choice of the site as to what ecosystem they want to be part of.
Bhattacharjee: From the physical interconnect standpoint, I agree with Cameron that right now it’s all InfiniBand connected. That’s the physical connection, where it’s all copper to connect one rack to another rack. Then there are Google TPUs, and they have decided to use optics for connection — optical interconnect, which is light, not copper. We will start seeing more optical interconnects in the next few years, and that will bring up the speed that is required, because the demand for speed is just crazy.
Vishwakarma: Great point on optical. It’s electrons versus photons, and photons are faster. There’s no resistance and minimal or no energy dissipation. These AI clusters have a huge problem of heat dissipation, so cooling adds a lot to the overall operational cost. Right now, not only with the connection between the two nodes, but within the chip, everything is electrical. There’s a lot of research going into co-packaged optics, about how much closer we can get to the chip, within the chip, using photons or optical connections. So co-packaged optics is an emerging area. Related to that is heterogeneous compute. Because of agentic flows and more inferencing, the architecture itself is trying to be more heterogeneous so that it can handle various kinds of workloads. But creating heterogeneous compute is a task by itself. You need different kinds of logic. You need a GPU, accelerator, NPU, CPU, and high-bandwidth memory. So the package itself is becoming very complex.
One topic we are looking at is the multi-physics effects. Everything is high-speed, and in a package you have multiple die. If it were multi-die HBM, it might collapse because of heat or warpage. The whole node will be impacted. The AI cluster will get impacted. If there are electromigration issues, which result in a lot of signal integrity issues, it’s going to eventually affect the performance of AI clusters.
Emmer: Overall, interconnects can be separated into three architectural layers. The first is chiplet and package interconnect, where the goal is ultra-efficient local bandwidth. The second is node-level coherent or semi-coherent attachment between host and accelerators. The third is cluster fabric, where you optimize for scale, reach, and topology. The standardization discussion is important because every boundary conversion costs latency, energy, and design complexity. You do not need one universal fabric. You need well-matched fabrics with minimal impedance mismatch. Optical can be transformational, but only if it solves a system bottleneck rather than becoming a science project.
SE: With all the data movement at different levels of the system, what software or AI is overlooking it all? Could a virtual machine help with all the different processor behavior? How is it all managed?
Brunner: A virtual machine can help at a higher level. For example, I’ll talk more about the Slingshot network, and some of the ways that this can be done, because it really shows one of the things that’s unique about these types of networks. It relates to the application-specific topic, where you can be provisioning virtual clusters on top of things. The Slingshot network allows you to do quality of service provisioning per workload, manage your data consumption, and place it optimally so you’re not having contention across multiple workloads. Slingshot does local copper connections for power reasons, but then the broader connections are fiber, and also 100 Gbps Ethernet. It all can be done in fiber, as well. For Slingshot, the long hauls are done in fiber, but it’s a combination of building software-defined clusters, then provisioning the network quality of service to meet the needs of that application. That’s the way you can guarantee you don’t have some pathological behavior of the workload that’s running.
Chole: In terms of software stacks, virtual machines are used to virtualize the cluster deployments at a smaller scale, but we do have a Dockerized environment everywhere. Containers are being used pretty much everywhere now. Even with a GPU driver’s toolchain that you want to be reproducible or more robust, we are going through containers as a primary way of deployment. There would be exceptions, but in the AI world, containers have caught up pretty much everywhere. That’s the standard way of bringing things up, but that’s just a platform. You still have a different specialized platform, like tools on top. There is a runtime portion, an orchestration portion, a scheduling portion, a queuing portion, API endpoints, serving platforms — and all those have to work with each other. Every application would have its own stack, and we specialize in those stacks. Given that agentic coding is picking up, it’s making it easier to be more specialized and build the exact feature you want, rather than just trying to use or retrofit the existing solution.
Bhattacharjee: One of the moats that Nvidia has is not their hardware anymore. It’s their software ecosystem that they have built over the last 20-plus years. Even though they are a hardware company, they have invested significantly in software. They’ve optimized it, and that’s the reason why people buy Nvidia and bring it in. It’s a complete solution — hardware, software, everything — and it’s tuned to work and scale. As we move from training to inference, there’s a strong desire to have a heterogeneous environment to bring token costs down, and software will become critical to utilize this heterogeneous environment and optimize across where you don’t have a single company doing end-to-end. It’s the same thing with the Apple ecosystem. They have their hardware and software. They tune it, but now you’re talking about bringing accelerators from different companies, getting the CPUs from a different company. The server is built by an OEM (original equipment manufacturer) or ODM (original design manufacturer), and then you have networking gear, and all of this is stitched together. The software needs to be, first of all, aware of the heterogeneity so it can take advantage of each of these different hardware specialties to optimize it, and also be able to delegate the workload to the hardware that runs appropriately.
There are some companies that have come up, such as Gimlet Labs and Together AI. Their pitch is that they provide the software layer to run on top of a heterogeneous hardware environment, even on disaggregated inference, and make sure the prefill cluster is run optimally, the decode cluster is run up to value, and the compute cluster is run optimally to do each of the workloads. They are taking care of the software orchestration, and they are working with different companies to do that. This is a very critical part when you go beyond the top hyperscalers like Google, which are invested in creating these custom TPU clusters. Not many companies can do that. Especially if you go to the neocloud guys, which are the next level of cloud providers like DigitalOcean, CoreWeave, Lambda Labs [now Lambda AI], Verda, and others, they’re all going to have this same challenge. There is a desire to bring more hardware diversity. There’s a desire to bring token costs down — of course, without sacrificing performance — and do it without having a single company’s hand holding you. This is a big challenge that we are facing right now. We are at the early stages. It will take some time before this problem is solved, but there’s a lot of concentrated effort going into building this heterogeneous cluster with optimized software coming and solving this big problem that we have, where everybody is waiting in line for Nvidia systems because that’s the only system that is proven to be working at scale right now.
Emmer: Also, software orchestration only works well if the hardware exposes the right control and telemetry abstractions. If the system can’t describe locality, bandwidth classes, thermal state, or engine specialization clearly enough, the orchestrator is effectively guessing. This means hardware/software co-design becomes essential.
SE: The central cluster design problems then are Nvidia’s dominance, and power consumption, which we covered in parts one and two of this discussion. Is that the correct takeaway?
Bhattacharjee: At a high level, power is definitely the number one problem. Without power, nothing will work. The second challenge is token efficiency, and that comes through heterogeneity and all the other innovations.
