This article represents the personal views of the author only and is not related to the author's affiliated institution

TL;DR

At a recent ODCC conference, I found the introduction to an I/O NET working group, led by Tencent, to be quite interesting. The two slides from Dr.Xia were particularly noteworthy. The original photos had poor clarity, so I'll briefly summarize them here:

The Three Stages of AI Networks

• First Stage: ScaleOut: Focused on building training clusters for single tasks with tens or hundreds of thousands of cards.
• Second Stage: ScaleUp: Supernodes for high-performance inference, pursuing low latency and handling MoE communication.
• Third Stage: Agent Fabric: A platform for Agentic LLMs. As shown in the figure below:

Primary Tasks of the Agent Fabric:

• Memory Pooling Project: Using ScaleUP to break through the memory wall.
  • Smoothly expand HBM memory.
  • Uniformly adapt to various compute-to-memory ratios.
• ScaleUP IO Die and IP: Supporting the productization and landing of ScaleUP products, mainly including the access of various devices like GPU / CPU / Memory to ScaleUP.
• Data Center Network Convergence Project: An interconnect network to meet the multi-service needs of Agents, including:
  • Convergence of ScaleOut / FrontEnd / Storage.
  • Convergence of ScaleUP and ScaleOut.
• Endpoint-Network Co-design Project: Primarily to solve the problem of efficient, high-quality network systems with multi-vendor interoperability.
  • Interoperability of multi-vendor NICs.
  • Co-design technologies for NICs and switches.
  • New in-situ processing IO-NICs.
• AI Network System-Level Reliability Project

Of course, these are from the networking community's perspective. Some of the problems and solutions mentioned have been explained in many previous articles. This article will provide a detailed summary and analysis of interconnects in the Agent era. We will first analyze from the independent perspectives of CPU/GPU and storage, and then put them together to look at the interconnects.

The table of contents for this article is as follows:

1. CPU Architecture and Interconnects in the Agent Era
1.1 The Two Forms of CPUs and Some Flaws in Vera
1.2 CPU I/O Problems
1.3 Some Suggestions for Improvement
2. GPU Architecture and Interconnects in the Agent Era
2.1 GPU Microarchitecture
2.2 GPU Interconnects
3. Storage Architecture and Interconnects in the Agent Era
3.1 Storage Semantics
3.2 Storage Interconnects
4. Agent Fabric
4.1 Bus Protocol
4.2 System Architecture
4.2.1 GPU Interconnect System
4.2.2 CPU Interconnect System
4.2.3 AI-NIC and DPU Architecture
5. Conclusion

1. CPU Architecture and Interconnects in the Agent Era

Compared to traditional LLM training and inference tasks, the biggest change is the introduction of the Agent Sandbox. Whether for RL training or Agent workflows, it often requires CPU clusters with hundreds of thousands to millions of cores.

1.1 The Two Forms of CPUs and Some Flaws in Vera

From a workload perspective, the requirements for the CPU itself differ vastly in two different scenarios:

1. CPUs serving as control nodes for GPUs and RL training Agent Sandbox need higher single-core IPS (Instructions Per Second) and greater bandwidth for transferring KVCache, etc.
2. CPUs serving as Agent Sandboxes need higher density and lower power consumption for high-density deployment.

This essentially pushes the CPU to two extremes. A single CPU like Nvidia's Vera is clearly unable to satisfy both. Regarding the demand for high single-core IPS, we mentioned in section 2.3 of 《Inside Nvidia GPU: Discussing Blackwell's Shortcomings and Predicting Rubin's Microarchitecture》:

Although the Hopper generation introduced NVLink C2C, allowing Grace to connect directly to Hopper or Blackwell via NVLink, the Grace CPU itself has many issues. As Blackwell's computing power grows stronger, the execution time of many kernels has dropped to the microsecond level, creating the classic Killer Microsecond problem. For nanosecond-level issues, synchronous waiting is fine. For millisecond-level durations, the cost of context switching is not significant. However, at the microsecond level, it poses a significant challenge for the processor. Despite the introduction of many asynchronous programming optimizations, current CPUs like Grace still face numerous bottlenecks. On one hand, the Kernel Launch speed is not fast enough. While one could argue this can be solved with CUDA graphs or persistent kernels, not all workloads fit this model.
On the other hand, there are some microarchitectural flaws in Grace. Although Grace uses ARM's then-strongest Neoverse V2 core, its design did not adopt the 2MB L2 cache used by V2, but instead cut it down to 1MB. In contrast, AWS Graviton 4, which also uses the V2 core, features a 2MB L2 cache. The significant L1 ICache Miss issues some customers are experiencing on the GB200 are largely related to this.

We have noted many flaws in the Grace CPU design, leading to bottlenecks in the GB series in many situations due to CPU limitations. It's hard to say how much Vera will improve things. Let's make a comparison:

To address the Grace Cache Miss problem, Vera Instruction Decode has been increased to 10-way, the L1D Cache to 6-way, and the L2 Cache capacity has been increased to 2MB. This might be what NV was referring to as the "PyTorch Optimized Instruction Buffer" specifically for Grace's issues. Official materials show a 1.5x IPC improvement and include changes to the branch predictor.

However, there are still some Cache and NOC issues involved, and Nvidia will need to put significant effort into solving them.

On the other hand, for high-density scenarios, Intel's Clearwater Forest already has 288 cores, reaching 576 cores in a dual-socket configuration. AMD's Venice can also achieve 256 cores per socket, and 512 cores in a dual-socket setup. Vera's density still has a large gap to close.

Both AMD and Intel offer choices between high-performance cores (I: P-Core / A: Zen6) and high-density cores (I: E-Core / A: Zen6c). In this respect, Nvidia still has quite a few problems. Attempting to use a single Vera to address three different scenarios—GPU control, Agent sandbox, and DPU control (BF4)—is clearly highly irrational.

1.2 CPU I/O Problems

Let's take AMD Venice as an example. Based on the SP7 (LGA-9324) socket, it has 9,324 pins. It is expected to support 16-channel DDR5, a 53% increase in pins compared to SP5, mainly to support the increased memory channels and higher-speed I/O channels (PCIe 6.0), while also providing higher power delivery capabilities to support Zen 6 cores. A large number of pins are also used for power and ground.

From a single CPU core's perspective, a common rule of thumb is that 1 GHz requires about 1 GB/s of bandwidth. For some data-intensive scenarios, it may even require 10 GB/s to match a 1 GHz processing pace. For a 256-core, 3 GHz processor, for example, using 16-channel MRDIMM provides about 1.6 TB/s of memory bandwidth, which is still significantly impacted by the memory wall. Furthermore, interconnects between sockets and external PCIe interconnects also demand greater bandwidth.

Another factor is the demand for storage bandwidth. For example, in the Agent Sandbox scenario, to ensure rapid instance startup, and with recent discussions about agents requiring more checkpointing and rollback capabilities during long, multi-turn executions, higher-speed storage bandwidth is typically needed. Currently, the Agent Sandbox does not involve data-intensive business processing, but future needs for network/storage bandwidth may further increase.

For specific requirements, we can refer to a paper from ASPLOS 2026, 《AgentCgroup: Understanding and Controlling OS Resources of AI Agents》. We can see several requirements:

1. 56-74% of the end-to-end task time is consumed by OS-level execution such as container/Agent initialization and tool calls. Additionally, the average size of an Agent's container image is 3.5GB. From a rapid startup perspective, although lazy loading capabilities exist, if several hundred CPU cores on a single machine concurrently execute 2000 Agent containers, the bandwidth demand on storage for launching images would be enormous, e.g., requiring 400~800 Gbps of storage bandwidth per machine.
2. The average CPU utilization of an Agent is low (below 14% after single-core normalization), but peak memory demand can reach 2-4GB. This means that in a multi-tenant cloud environment, the key resource limiting the number of concurrent instances is memory, not compute.
3. Memory usage exhibits a stable framework baseline of about 185MB, superimposed with short-lived (1-2 seconds) and intense (peak-to-average ratio up to 15.4x) resource bursts driven by tool calls.

Note: RL training, inference, and Agent Sandbox execution in Agent scenarios may require more co-design. For example, training for coding tasks still requires many high-frequency, high-performance CPU cores, while also needing large-scale deployment to accelerate rollout speed. We will analyze this in detail in future articles.

Thus, from the CPU's I/O perspective, there is a high demand for bursty memory bandwidth, coupled with high throughput requirements for storage and networking. The short-lived peak resource demands mean the entire CPU needs higher I/O. However, the available pins on the CPU are highly constrained, as bandwidth is consumed by inter-socket interconnects like UPI/XGMI, DDR memory channels, and PCIe lanes. On top of that, there are CXL lanes intended for memory pooling, and even higher bandwidth interfaces like NVLink C2C for GPU connectivity.

The pins coming out of the CPU are thus heavily constrained. Therefore, providing higher-speed external interfaces (e.g., 224G Serdes) might be a better option. PCIe Gen6 is only 64 GT/s, and even Gen7 is only 128 GT/s. Who knows when Gen8 will land...

1.3 Some Suggestions for Improvement

The real problem is that fragmented bus protocols create many SKUs and various interconnect bottlenecks for the CPU. We need a unified bus (am I hinting at using Huawei's Unified Bus?). It should be able to connect CPU-CPU, CPU-GPU, and also DPU/NIC. This presents several choices:

1. Will CXL work? No. Many have been led down the garden path by CXL; after so many years, we haven't seen many serious implementations. At the same time, its bandwidth is inching forward slowly, just like PCIe. We also have to consider the GPU side; how many GPU vendors are willing to support CXL?
2. Will NVLink C2C work? This is an interesting question. Firstly, Nvidia's own Grace and Vera still have many issues. The CPU controlling the GPU needs even higher IPS. Although Nvidia has invested in Intel, perhaps Intel will provide X86 chips based on NVLink C2C. But what about NICs? SSDs? Does the industry have to create one ecosystem for NV and another for non-NV?
3. Will UALink work? The same problem arises. It's clearly unreasonable for the entire industry to take sides on different buses or for everyone to build their own.

In fact, for any open protocol organization to succeed, there always needs to be a market leader, a 'big brother,' to spearhead it. This was the case with PCIe back in the day, and it's true for this bus today. This is why things like Gen-Z, OpenCAPI, and CCIX haven't succeeded over the past few years.

One question remains: PCIe has been used for so many years, can't we just modify it and make do? Why create a new bus? The reason is that we need to consider the following series of interconnect scenarios:

• CPU-CPU: First, each company might still have its own choice for CPU-to-CPU interconnects. UPI/XGMI and others have many optimizations based on their own CPU microarchitectures and NOCs, so there's not much demand for a unified, multi-vendor standard bus. However, we've also observed that Nvidia uses the same NVLink-C2C for both CPU-CPU and CPU-GPU. A unified bus does offer more flexibility.
• CPU-Memory Pool: Next, let's consider the interconnect between the CPU and a memory pool, as well as between the GPU and a memory pool. If a common interface is exposed, extending the CXL story to build a large memory pool would be valuable for both the KVCache used in GPU node model inference and for addressing the memory resource bursts of the Agent Sandbox. From a supply chain perspective, it's natural to hope for shared modules. There are some constraints here. We can see that virtually no GPU vendor is considering supporting CXL. Can't they just attach a memory pool to their existing ScaleUP bus?
• CPU-DPU: On the other hand, for the CPU-to-DPU interconnect, the bandwidth evolution of Ethernet is far outpacing PCIe, and the evolution of high-density CPUs (256-core to 512-core) also requires more bandwidth. For instance, a 1.6Tbps NIC currently requires two PCIe Gen6x16 connections, and there can be ordering issues between these two PCIe links.
• GPU-AI_NIC: For the GPU-to-AI-NIC interconnect, if different protocols are used, the GPU side would need to communicate using a fixed ScaleUP:ScaleOut ratio and two sets of semantics.

If we need flexible interconnect ratios, a better approach for various chips is to build a relatively unified bus protocol. The problem with PCIe is its Root Complex (RC)-based tree topology. Although CXL has gradually introduced capabilities for building switched networks like Port-Based-Routing, PCIe's own Controller IP and its slowly evolving Serdes have also become a problem. A simple suggestion, then, is to separate the control and data paths. We could reserve a 1x PCIe lane as the control link to maintain software compatibility, and use a higher-speed bus for the data path.

We have also received some news that the future CX10 will also opt for an interface like NVLink C2C to avoid using PCIe. In fact, I have long been suggesting that the NIC itself needs to be attached to the ScaleUP bus, and I believe this is a more correct evolutionary path.

Let's assume we call this bus the New Unified Bus (NUB). Building a cache-coherent bus for CPU-CPU, CPU-GPU, CPU-Memory Pool, and CPU-DPU connections has a lot of business value. For GPU-GPU and GPU-AI_NIC, cache coherence is not necessary. However, a key difference between this NUB and Huawei's UB is that, to consider the die size overhead on the GPU, we cannot introduce message semantics like RDMA on top of it. The logic is simple: a single GPU die is constrained to about $800mm^2$, while an 800Gbps RDMA NIC die is about $200mm^2$. Therefore, I have always been against introducing message semantics on the GPU ScaleUP bus. My personal view is that beyond the rack level, we should still maintain RDMA for transmission via message semantics, as shown below:

Conclusion: For the Agentic era, we need to provide the CPU with a relatively unified, cache-coherent bus that uses higher-speed Serdes. Of course, this involves some business maneuvering, such as the development of AMD's UALink alliance and Broadcom's stance on this interconnect segment. More importantly, it depends on whether Nvidia is willing to build a relatively open bus organization in this domain. There are also some forces that cannot be ignored: the major North American hyperscalers. AWS, Google, and Azure all have their own ARM CPUs and similar business needs.

2. GPU Architecture and Interconnects in the Agent Era

2.1 GPU Microarchitecture

The GPU architecture will also undergo some changes in the Agent era. An obvious example is the solution based on Groq's LPU announced at this year's GTC. We need to further reduce inference latency and increase tokens per second. A previous article has already detailed this: 《Inside Nvidia GPU: Discussing Blackwell's Shortcomings and Predicting Rubin's Microarchitecture》, which Jensen Huang also forwarded within NV.

A core idea is that some warp schedulers within the SM need to be exposed through a scalar core. This would allow for faster control of mbarrier, using TensorCores, TMA, and CUDA Cores as coprocessors. This would be of great help in implementing MegaKernels (e.g., Mirage...) to reduce overall GPU inference latency.

There is a major misconception here, which even comes up when talking with many NV employees. Conventional thinking holds that the GPU is a high-throughput parallel processor that isn't overly concerned with latency, so the metric has often been simply to keep all compute units fully utilized. However, in the Agentic LLM era, there is a greater need for low latency. To guarantee TPS SLAs, the GPU doesn't need to run at a high batch size to fully utilize compute units or memory bandwidth. Instead, it operates at a relatively high utilization rate, using warp scheduling and instruction issuing to reduce latency.

2.2 GPU Interconnects

We note that Rubin+LPX still use a ScaleOut network for interconnection, which will be unified onto NVLink in the future. Another change brought by the Agent era is the higher demand for GPUs to access memory pools or external storage, smoothly expand HBM memory, and uniformly adapt to various compute-to-memory ratios. Of course, connecting on-board LPDDR through HBM is one path. But considering the huge ScaleUP bus bandwidth and more flexible compute-to-memory ratios, perhaps connecting some memory pools to the ScaleUP bus is a better solution.

Then there's a controversial point: the convergence of ScaleUP and ScaleOut. This involves some disputes between the compute teams and the networking teams. Compute teams generally believe that building a multi-layer switch network for ScaleUP is sufficient, but multi-layer switching faces challenges in load balancing and reliable transport. Even with Nvidia's announcement of solutions based on the Orben NVL576 or the Kyber NVL1152, I personally believe there are still significant reliability challenges.

Furthermore, from a customer's perspective, there is a need for flexible ratios. This means that this second-layer ScaleUP network has a clear oversubscription ratio, mainly because data processing always has its data locality. A larger scope doesn't necessarily require a 1:1 non-blocking network. Consequently, congestion control and reliable transport essentially have to be compensated for with additional computing power. For a detailed analysis, please refer to 《A Discussion on Reliable Transport for RDMA and ScaleUP》.

3. Storage Architecture and Interconnects in the Agent Era

The biggest variable for infrastructure in the Agent era is the demand for higher-performance storage. Traditionally, storage is accessed through the FrontEnd network. DeepSeek DualPath has gradually introduced it into the ScaleOut network, and the ODCC I/O NET working group is also discussing the topic of converging ScaleOut / FrontEnd / Storage. However, this is a very difficult task, at least for RDMA NICs like Nvidia's CX series.

3.1 Storage Semantics

Analyzing from the perspective of storage semantics, there are generally two types. Nvidia has a slide that compares them:

For GPU programming, a block-based approach might affect GPU efficiency because a KV operation could involve too many blocks, occupying SM resources. A file-based approach, with a complex file system running on CUDA Cores, is also clearly not feasible. For GPU Initial Direct Storage (GIDS), the optimal interface would need to be the NVMe KV cmdset.

3.2 Storage Interconnects

From a transport protocol perspective, the first challenge in converging ScaleOut / FrontEnd / Storage lies with the PCIe bus. As mentioned in the DeepSeek DualPath paper, PCIe itself lacks a good QoS mechanism and has low bandwidth, so converging it with ScaleOut can easily interfere with collective communications. In the design of next-generation protocols, if AI-NICs or SSDs can be attached to the ScaleUP bus, this handling would be much simpler, and dynamic compute-to-storage ratios could also be easily achieved.

4. Agent Fabric

Let's review the main tasks of the Agent Fabric as defined by I/O NET:

1. Memory Pooling Project: We need to use ScaleUP to break through the memory wall, achieving smooth HBM memory expansion and unified adaptation to various compute-to-memory ratios.
2. ScaleUP IO Die and IP: Supporting the productization and landing of ScaleUP, mainly including the access of various devices like GPU / CPU / Memory to ScaleUP.
3. Data Center Network Convergence Project: An interconnect network to meet the multi-service needs of Agents, including: convergence of ScaleOut / FrontEnd / Storage, and convergence of ScaleUP and ScaleOut.
4. Endpoint-Network Co-design Project: Primarily to solve the problem of efficient, high-quality network systems with multi-vendor interoperability, including multi-vendor NIC interoperability, NIC-switch co-design technology, and the design of new in-situ processing IO-NICs.
5. AI Network System-Level Reliability Project

Putting all these together, it looks like this:

4.1 Bus Protocol

From a topological connection standpoint, it may not seem particularly special, but the devil is in the details. The most critical and controversial point is that some architects will always dream of a single protocol to unify all interconnects, but one size never fits all. Therefore, from a protocol perspective, a clear distinction is necessary, as shown below:

First, for the evolution of memory standards, it's best to leave that to JEDEC. Although Nvidia will introduce Custom HBM in Feynmann, and this area also includes technology routes like attaching LPDDR through HBM or some Optical Memory interfaces, this article will not delve into that.

A major point of contention is between memory semantics and message semantics, especially in the debate between Ethernet-based ScaleUP (and the corresponding ScaleUP + ScaleOut convergence) and ScaleUP protocols like NVLink/UALink. My personal view is that the Chassis Level and Rack Level require a new unified bus, while inter-rack interconnects should continue to use Ethernet-based message semantics protocols (e.g., RDMA). For convenience, we will call the former the Intra-Host Protocol and the latter the Inter-Host Protocol.

It is very difficult to have a single unified protocol that merges ScaleUP (Intra-Host) and ScaleOut (Inter-Host). This issue was analyzed quite clearly 5 years ago during the development of NetDAM. Directly extending an Inter-Host protocol to support memory semantics, with a small Flit size, incurs a huge cost for reliable transport protocols and congestion control, and also takes up a lot of IO Die area, making it difficult to implement on a GPU. Similarly, directly extending an Intra-Host protocol to support networking across multiple switch hops introduces additional costs for implementing reliable transport.

Latency Analysis

First, let's analyze the latency difference: Intra-Host communication protocols typically have a fixed transmission latency of less than 200ns, whereas Inter-Host Ethernet usually has a latency of several microseconds, with uncertainty introduced by packet scheduling, multipathing, and congestion control.

These two lines of thought are also reflected in the design differences between the SUE and UALink protocols. In SUE, queues on the accelerator side differentiate traffic destined for different GPUs, and traffic is sent to the switch via Round Robin scheduling. The switch also forwards traffic on a per-packet basis. As shown below:

We notice that in the Expert-Parallism case, multiple GPUs may send tokens to a single GPU, creating an incast situation. Since the switch also schedules packets to the destination GPU's Egress queue, data from a particular source GPU will experience significant tail latency in this scenario. For example, in the figure above, data sent from source GPU-4 might experience a long tail latency close to 10us because the Egress queue buffer is full. For traditional GPU architectures, to better utilize memory bandwidth (i.e., achieve higher GEMM efficiency), Expert FFN computation is usually performed with a sufficiently large batch size. The compute latency might be tens of microseconds, and with Two-Batch-Overlap Attention, such a long tail latency doesn't have a major impact.

However, in the Agentic LLM era, we need higher TPS. For instance, when using an LPU for processing, the Expert parameters are in SRAM, so computation can be completed very quickly even with a small batch size, with the computation latency itself being under 2us. In this case, the long tail latency from the ScaleUP network will have a very significant impact.

In contrast, with UALink, as shown below:

The sender does not differentiate data for different destination GPUs. It simply packs 64B TL Flits into a 640B Datalink layer Flit and sends it to the switch. The switch parses the TL Flits within and sends them to the Egress queue. The Egress queue can send to the destination GPU as soon as it accumulates 10 TL Flits. With this method, the tail latency can be controlled to around 2us.

Reliable Transport Analysis

In a relatively short-distance (Rack Level) network that only supports a single layer of switches, the problem of reliable transport is very simple. If it involves a multi-layer switch network and long-distance transmission using optical fiber, factors like the overall system MTBF and the load balancing of the switched network must be considered. Furthermore, just calculating the data latency in the optical fiber alone will exceed 2us. If packet loss occurs, recovery would involve complex processing logic.

MMU Consideration

A few days ago, I was discussing an interesting scenario with colleagues from AMD: how should memory addresses be arranged in an ultra-large-scale network with memory semantics? This leads to another question: how can we dynamically add/remove nodes within a converged ScaleUP+ScaleOut domain to maintain overall system reliability and cluster elasticity? The specific technical details are confidential and won't be elaborated on here.

Conclusion

We need to build a new bus protocol at the Rack Level to support memory semantics. For Inter-Rack communication, we can continue to use Ethernet-based message protocols like RDMA. For the physical layer and connectors, we can unify on OIF CEI-112G/224G to reduce costs and maintain rapid evolution.

4.2 System Architecture

4.2.1 GPU Interconnect System

The current GPU interconnect is shown below:

When GPUs require greater storage access bandwidth, storage traffic typically needs to go through the FrontEnd network, through the CPU, and then to the GPU. For Nvidia, there is NVLink C2C between the CPU and GPU. However, introducing storage into the higher-bandwidth ScaleOut network presents many potential problems: first, PCIe lacks good QoS support, and storage traffic can easily interfere with collective communications. Second, there are the old RoCE problems like congestion control, load balancing, and reliable transport.

There are also new business requirements, such as the need for dynamically adjustable compute-to-storage ratios. This brings up a new question: how can GPUs achieve a flexible ScaleUP to ScaleOut ratio? The current fixed PCIe and NVLink ratios directly constrain this possibility. If a unified ScaleUP protocol is adopted, and the AI-NIC is attached to the ScaleUP bus, this problem can be easily solved.

But there are two options here. The first is to integrate a RoCE IP on the GPU, like the accelerators from Microsoft/Meta, and build on an Ethernet ScaleUP.

However, we note that such a solution would cause a large amount of chip area within the accelerator to be occupied, typically requiring an additional NIC I/O Die for interconnection. Considering power and bandwidth requirements, these RoCE-based IPs also have significant shortcomings in reliable transport and load balancing, ultimately increasing the ScaleUP latency. So why not consider another solution: connecting the AI-NIC to the ScaleUP Switch?

The advantage of this is that GPU-GPU communication can maintain the low latency of the original ScaleUP bus, while AI-NICs can be placed according to the compute-to-storage ratio (e.g., 1:4 / 1:8). At the same time, the Ethernet network can be built with an oversubscription ratio based on the actual storage node bandwidth.

4.2.2 CPU Interconnect System

For the CPU node, it needs a bus like NVLink-C2C to connect to the GPU. However, the existing architecture, which uses PCIe to connect to storage and networks, will also encounter the bottleneck of insufficient PCIe bandwidth, especially when a single CPU reaches 512 cores in the future. Considering the working characteristics of the Agent Sandbox, it will have very large bursty memory usage. During multi-turn Agent execution, we have enough time to swap some memory pages to an external memory pool. For example, when Agent Sandbox A finishes its execution and needs to pass the tool call results back to the inference engine, its memory can be swapped out to external storage. When Agent Sandbox B receives the next piece of execution code from the LLM, it can retrieve its memory from the shared memory pool. Therefore, an additional memory pool, or even more extremely, using external SSD storage, can save memory capacity. Thus, we can build the following interconnect system:

4.2.3 AI-NIC and DPU Architecture

This ODCC conference also discussed the design of new storage processing IO-NICs, as well as the issues of multi-protocol, multi-vendor interoperability and end-to-end network convergence.

Regarding multi-protocol interoperability, the reality is that the old RoCE problems like congestion control, load balancing, and reliable transport have not been well resolved even after several years. This has led every company to customize its own RoCE protocol, such as OpenAI's MRC, Google's Falcon, AWS's EFA-SRD, and Microsoft and Meta also have their own protocols. But a major problem has been discovered: many of these protocols have not been successful even after years of evolution. Therefore, some sessions at this ODCC conference were still repeatedly discussing this issue, with topics like multi-vendor interoperability, switch and NIC co-design, and the need for switches with ultra-large buffers for cross-AZ communication.

Coincidentally, during the conference, a customer of Alibaba Cloud tested a cross-data center collective communication using CIPU eRDMA (across a 9-hop switch network) and easily saturated the bandwidth. This is a problem that was solved three years ago, yet NV (Mellanox) has been unable to cure it. The root cause lies in the microarchitectural design of Nvidia's CX series NICs, and the protocols designed by NV and hyperscalers, which are generally based on the CX series microarchitecture, cannot handle these problems well.

A simple example: Nvidia's BF4 uses multiple components—a switch IP based on Spectrum, a new PSA processor, the existing DSA processor, and a Vera general-purpose CPU to handle storage processing tasks. The total power consumption is several hundred watts. Is this really necessary? The reality is that this is due to the flaws in their RoCE protocol and the legacy RoCE IP that has evolved over many years. They also need to support Packet Spraying on the switch and add constant patches and band-aids like larger-buffer switch chips for cross-AZ scenarios. Years have passed, and the problems are still not solved.

5. Conclusion

I am quite hopeful that Nvidia can lead a coalition of CPU vendors (Intel/AMD/ARM/Google/AWS) to define a new bus. Of course, as a CPU bus, it still needs to support some degree of Cache Coherence, or at least simply support something like CXL's Backward-Invalidation. This protocol could also be used in a Non-Cache Coherence mode for GPU ScaleUP. Of course, Nvidia is also licensing solutions like NVLink Fusion. Business logic seems to make it difficult for them to license it to AMD. However, at the end of this year, AWS Trainium 4 will support both NVLink and UALink. If there is no significant performance difference between the two, the standard for interconnect buses will quickly tilt towards UALink.

I believe Jensen still remembers the story of 3Dfx insisting on its Glide API, only to be replaced by a host of companies like NV/ATI that supported DirectX/OpenGL. The power of an ecosystem is immense. When the bus standard begins to tilt elsewhere, combined with the ecosystem's shift from SIMT CUDA to Tile-Based programming, and even when work similar to Nvidia's AVO can optimize kernel performance through Agents, perhaps Nvidia's moat will no longer exist...