Optics and Cable Management
By Mahesh Subramaniam, Michal Styszynski, Himanshu Tambakuwala
Date: Feb 25, 2026
Sample Chapter is provided courtesy of Addison-Wesley Professional.
Scaling Optics for AI Clusters
In Chapter 3, “Network Design Considerations,” we discussed design concepts involved in building an AI/ML data center. Optics and cables are important components of any data center, so requirements of an AI/ML data center.
With AI/ML clusters, the ports on a server currently support 200 Gbps and 400 Gbps throughput and are moving toward 800 Gbps and 1.6 Tbps throughput. Nvidia is the top vendor for the GPU server supplies for AI/ML clusters. It has multiple generations of GPUs—Volta, Ampere, Hopper, and Blackwell. Nvidia currently has A100 (Ampere) and H100 (Hopper) on the market and is moving toward H200 and further versions of GPU based on Blackwell.
From the GPU, Nvidia uses NVLink to connect to the NVSwitch, which is used within the server to communicate between the GPUs that are internal to the server. Each NVLink is 300 Gbps for the Volta generation, 600 Gbps for the Ampere generation, and 900 Gbps for the Hopper generation, moving toward 1800 Gbps for the Blackwell generation. Other vendors have similar solutions; for example, AMD has Infinity Fabric, and Intel has CXL (Compute Express Link), UCIe (Universal Chiplet Interconnect Express), and PCIe (Peripheral Component Interconnect Express) switches.
Figure 4-1 illustrates the system topology of a server with 8 GPUs, where an internal switch helps with communication across the GPUs in the server.
Figure 4-1
System topology
In addition to the NVSwitch for internal communication, there are different NICs that are used for external connectivity between the GPUs across multiple servers. In either case of NVIDIA or AMD-based GPU servers for the scale-out DC use-case, each GPU is connected to a dedicated NIC card, and then each NIC card from the same server connects to different top-of-rack Ethernet or InfiniBand switches. It means for local intra-server communication, the NV switch will be used, and for any server-to-server communication, the external switch is used. GPU-to-NIC card connections are 400 Gbps or 800 Gbps Ethernet or IB. This means a top-of-rack switch is also typically a high-port-density 400Gbps/800Gbps and 1.6Tbps switch. From the AI DC server perspective, besides the GPU-connected NIC cards, there are also storage NIC cards (NVMe-o-F, for example) and out-of-band connections. They are all interconnected via PCIe generation 5 and newer to offer even higher local server interconnects to memory blocks or the CPU.
Figure 4-2 shows a chart from a Dell’Oro report on market adoption of optics for AI clusters from 2020 to 2027. This chart indicates that optics adoption is going to move toward 1.6 Tbps. With the requirement of high-bandwidth optics, there is also a need for high-radix switches that can support a large number of ports per rack unit (RU). In addition, optics need to be power-efficient to reduce the power and thermal budget of a rack. These needs are driving the enhancement of small-form-factor optics, modulation, connectors, and cables.
)
Figure 4-2.1 shows a 2025 study from Dell’Oro showcasing the adoption of 800 Gbps in 2025 and growing demand for 3.2 Tbps optics by 2029.
)
The throughput requirement from server to leaf is expanding to 200 Gbps/400 Gbps/800 Gbps/1.6 Tbps, which is leading vendors to focus on optics to support high-bandwidth connections.
Challenges in Optical Innovation
The optics industry has rapidly advanced from 10 Gbps to 100 Gbps and now to 800 Gbps and beyond. This growth is outpacing traditional models, driven by the need for more bandwidth and faster innovation. Although optical technology doesn’t directly follow Moore’s Law, it is propelled by data center demands, powerful processors, and expanding data volumes.
These are some of the challenges related to optics technology:
Signal quality: Maintaining signal integrity gets harder as data rates rise. Higher speeds mean more attenuation, dispersion, and noise from crosstalk, all of which can harm signal quality. To tackle these challenges, advanced modulation schemes and error correction are required.
Signal conditioning: At elevated speeds, a signal may experience more severe impairments. To preserve signal integrity, sophisticated methods like digital signal processing and equalization are required.
Power consumption and cooling requirements: Higher-speed optics usually require more power, which can create significant challenges in data centers, where energy efficiency is vital. In addition, increased power usage results in high heat dissipation and requires cooling.
Availability and cost: As AI/ML data centers are demanding higher-speed optics, different organizations are working toward coming up with standards for high-speed optics. Optics technology is lagging, and challenges include availability and cost of high-speed optics units.
Packet Flow
Whenever a network device receives data, it is in the form of either electrical signals or optical signals. A signal goes through multiple stages before it reaches the packet-forwarding engine. Similarly, a packet goes through multiple stages before it is transmitted out. Figure 4-3 illustrates the major components related to the optics in a switch.
)
In the following sections, we’ll discuss packet flow for the newer 400 Gbps and onward optics.
Demultiplexers and Multiplexers
As shown in Figure 4-4, when a signal—either in electrical form when using copper cables or optical form when using optical fiber—reaches the pluggable optics, it goes through a demultiplexer (demux) function that splits the signals.
)
The PFE ASIC has multiple SerDes (serializer/deserializers), which form a high-speed interface used to convert data streams received between serial and parallel forms. As the PFE ASIC processes packets serially, the SerDes performs the job of serializing the data received from multiple SerDes links in parallel. If the PFE ASIC supports ~50 Gbps SerDes, a demux could perform the demultiplexing to 8 × 50 Gbps from 400 Gbps optics. If the PFE ASIC supports ~100 Gbps SerDes, a demux could perform the demultiplexing to 4 × 100 Gbps from 400 Gbps optics or 8 × 100 Gbps from 800 Gbps optics.
After the PFE ASIC processes the data, the SerDes again parallelizes the data into separate streams. Either single or multiple SerDes can be mapped to a single optics unit to achieve the required rate. This process is used for creating N:1 conversion of signals, and it is achieved with the help of multiplexing, as illustrated in Figure 4-5.
)
With AI/ML requirements of 400 Gbps, 800 Gbps, and beyond, the SerDes links are now at 200 Gbps and moving toward higher speeds. Further improvements in the mux and demux are required to be able to achieve higher speeds.
Digital Signal Processors (DSPs)
The signals from the demultiplexer are passed to the digital signal processor (DSP), which carries out several functions. Let’s investigate each of them in detail.
Modulation and Demodulation
DSPs are responsible for dealing with sophisticated modulation methods in high-speed optical communications, as illustrated in Figure 4-6. They encode digital data onto the optical carrier wave by converting it into diverse amplitude and phase states. On the receiving side, DSPs decode the optical signal by interpreting the detected changes in amplitude and phase to retrieve the original data.
)
Modulation is the process of converting data or information to electrical or optical signals. Modulation is required from the ASIC to the electrical signals and then later from electrical signals to optical signals. The modulation at each layer may be different. NRZ, also known as PAM-2, is a traditional modulation technique that does not support higher-bandwidth requirements. PAM-4 and above are being used for newer optics to support the higher-bandwidth requirements of AI/ML data centers.
The following modulations are used in different types of network connectivity:
NRZ: Non-Return-to-Zero (NRZ), which used to be a widely used scheme, has two voltage levels to represent 0 and 1. It is commonly used in the 28 GHz range and for some 56 GHz channels.
PAM-4: Pulse Amplitude Modulation with four levels (PAM-4) is a signal-encoding technique that uses four distinct signal levels (that is, voltage levels) to represent four combinations of two-bit logic (00, 01, 10, and 11). PAM-4 modules double the bandwidth of a connection, with each signal level representing 2 bits of logic information. PAM-4 is used for some 56 GHz channels and all 112 GHz channels.
Higher-order modulation: For greater speeds, modulation schemes such as PAM-8 or QAM (quadrature amplitude modulation) are considered. These methods boost data rates by using additional signal levels or combining various amplitude and phase states.
DWDM (dense wavelength division multiplexing): Coherent DWDM technology uses amplitude, phase, and polarization of light. It supports much higher bit rates on a single wavelength when DSPs are used. DWDM is used for transmitting multiple data channels over a single fiber, expanding data capacity in high-speed connections between data centers. 400G ZR modules have ushered in a new era of DWDM technology marked by open, standards-based, and pluggable DWDM optics, enabling true IP-over-DWDM. 400G ZR modules are used for connectivity between data centers (up to 80 km).
To achieve higher data transmission rates, the industry is trying out different modulation methods that make it possible to transmit more data at a time. We can compare it to adding more seats to carry a larger number of passengers on a flight, where the downside is that privacy is reduced. Similarly, transmitting more data at a time results in more noise to the signals and creates a need for more sophisticated mechanisms to deal with it.
Error Detection and Correction
DSPs are responsible for error detection and handling retransmission or correction of received packets to maintain data accuracy. They use forward error correction (FEC) algorithms to rectify errors that occur during transmission. Methods like low-density parity-check (LDPC) codes and Bose-Chaudhuri-Hocquenghem (BCH) codes are often used to enhance data integrity and reliability. At 400 Gbps and above, FEC is needed for reliability, although it introduces latency.
Clock Data Recovery
DSPs play a critical role in synchronizing the transmitter and receiver to maintain data integrity and minimize errors. A DSP extracts clock signals from the incoming data, which is vital for precise data sampling and decoding. This function becomes especially important at high data rates, where accurate timing is essential.
Equalization
DSPs use equalization to improve the signal-to-noise ratio (SNR). Equalization is a signal processing technique that restores the shape of a signal waveform in optics. Equalization algorithms include feed-forward equalization (FFE) and decision-feedback equalization (DFE).
Transmission Modes
Beyond the DSP and mux/demux, we move more toward optics. Two decades ago, most network gear, including switches and routers, relied on copper cables for data transmission. But these cables are limited in terms of how fast data can move. As data speeds picked up, these cables couldn’t keep up, as they only reached about 3 meters for 100 Gbps Ethernet links. They were good for connections in the same rack, though, such as connecting servers to the top of rack (TOR) switch.
Today, we have high-performance switches, routers, and SmartNICs that can handle speeds of 200 Gbps, 400 Gbps, and even 800 Gbps. To handle these speeds, fiber-optic cables have taken over from copper ones. Instead of using electrical signals, optical fiber uses light to send data.
The advantage in fiber-optic cables is that they can go much farther—up to 80 km or even 120 km—without losing signal strength. They’re also much more reliable than copper because they use total internal reflection to carry light and therefore aren’t affected by electromagnetic interference. In addition, fiber-optic cables handle changes in temperature and pressure well.
The following sections detail the different cable options available for connectivity in data center fabrics.
Multi-Mode Fiber (MMF)
MMF is optical fiber that is designed for the transmission of multiple rays of light at a time (see Figure 4-7). Normally, the core diameter of MMF is 50 μm (micrometers) and 62.5 μm. It propagates 850 nm (nanometers) and 1300 nm wavelengths from low-cost light sources like LEDs or VCSELs and has more attenuation. MMF is used for short-to medium-range connections, as well as for connecting devices within the same rack or in nearby racks.
)
There are five different grades of MMF:
OM1: OM1 supports 1 Gbps speeds up to a distance of 300 m and 10 Gbps speeds up to 33 m.
OM2: OM2 supports 1 Gbps speeds up to a distance of 550 m and 10 Gbps speeds up to 82 m.
OM3: OM3 supports 40 Gbps speeds up to a distance of 240 m and 100 Gbps to 400 Gbps speeds up to a distance of 100 m.
OM4: OM4 supports speeds of 100 Gbps to 400 Gbps up to a distance of 150 m.
OM5: OM5 supports speeds of 100 Gbps to 400 Gbps up to a distance of 150 m. It can accomplish 400 Gbps with just four transmit and receive cables and wavelength division multiplexing (WDM) technology. OM5 uses the wavelength 953 nm along with the wavelength of 850 (OM4).
OM1 has a glass core diameter of 62.5 μm. The other grades of MMF have a glass core diameter of 50 μm.
Single-Mode Fiber (SMF)
SMF is an optical fiber that is designed for the transmission of a single ray of light at a time, eliminating distortion from overlapping light pulses (see Figure 4-8). Normally, SMF has a core diameter of 8 μm to 10 μm, which can propagate higher wavelengths of 1310 nm and 1550 nm from the laser. It is an expensive type of cable that is used for long-distance data transmission between different buildings or data centers.
)
Dense Wavelength Division Multiplexing (DWDM)
DWDM enables dense wavelength multiplexing by multiplexing multiple electrical signals to a single optical lane, as illustrated in Figure 4-9. DWDM technology helps in transmitting data over long distances, as described earlier in this chapter. This is specially required in case of data center interconnects.
)
AI Server Connectivity Options
On the switch side, the connection could be either single-mode or multi-mode fiber and optics, depending on the distance. Since the server ports an OSFP transceiver with 800 Gbps, the switch side must be a 800 Gbps port or two 400 Gbps ports with the use of breakout cables. Figure 4-10 illustrates both of these connectivity options. For rail-optimized design (ROD), another option is able to connect 8 GPUs to 8 different switches.
)
Figure 4-11 illustrates the connectivity option with the A100 server. There are 8 ports on the server that can connect to 8 different switches in ROD.
)
Transceiver Types
The choice of optics is based on the connectivity distance between two nodes. In general, optics meant for smaller distances are cheaper than those meant for longer distances. Each type of optical fiber has a suffix to denote the reach and optical lanes. For example, in 400G-SR8, SR stands for short reach, which is up to 100 m, and 8 denotes the number of optical lanes; 400G-SR8 can have 8 optical lanes of 53 Gbps each, which are multiplexed to support 400 Gbps bandwidth.
Table 4-1 lists the transceiver types and the reach and mode of optical fiber with which it can be used.
Table 4-1 Transceiver Types
Transceiver |
Full Form |
Reach |
Mode |
|---|---|---|---|
VR |
Very short reach |
50 m |
MMF |
SR |
Short reach |
100 m |
MMF |
DR |
Data center reach |
500 m |
SMF |
FR |
Far reach |
2 km |
SMF |
LR |
Long reach |
10 km |
SMF |
ZR |
Extended reach |
>80 km |
DWDM |
CR |
Copper |
Up to 7 m for passive Direct Attach Cable (DAC) type Up to 10 m for active Direct Attach Cable (DAC) type |
|
The length of the cable and type of optics needed are determined based on the design option chosen—for example, top-of-rack, middle-of-row, or end-of-row. Within a rack, copper-based Active Electrical Cable (AEC) has good potential. Across racks, Active Optical Cable (AOC) has good potential. Very short reach (VR) optics support is being added currently and may be suitable in AI/ML clusters.
Figure 4-12 illustrates an example of top-of-rack server-to-leaf connectivity and leaf-to-spine connectivity in the rail-optimized design. This topology requires cables of varying length within the rack.
)
Figure 4-13 illustrates an example of middle-of-row and end-of-row design server-to-leaf connectivity and leaf-to-spine connectivity in the rail-optimized design. This topology requires cables of similar lengths from a rack, but across the rack, the cable length varies.
)
){kind=link}