TCP RACK: How ONTAP 9.19.1 Stopped Counting to Three

TCP has spent decades counting to three before deciding a packet is lost. RACK throws out the rulebook - and your storage traffic is better for it.

The Three-Dup-ACK Problem

It's 1999. Napster is new, Y2K panic is real, and TCP fast retransmit gets standardized around the idea that three duplicate acknowledgments mean a packet has been lost in the ether. It was reasonable for the era. So was a dial-up modem. 🐢

The problem isn't that the rule is wrong; three dup-ACKs do indicate loss. The problem is what they can't detect: tail loss. If the last packet in a window drops and there's nothing left to send, no dup-ACKs arrive. The sender just waits. And waits. Until the retransmission timeout fires - often hundreds of milliseconds, sometimes seconds - and then the congestion window collapses. For latency-sensitive NFS or SMB workloads, this is painful.

TCP RACK (Recent ACKnowledgment) is first implemented in ONTAP 9.19.1. It replaces this counting game with something better: time.

Watches Beat Fingers: The Core Idea

RACK's central insight is deceptively simple. Instead of counting how many duplicate acknowledgments have arrived, it asks: "how long ago did we send this packet, and shouldn't we have heard back by now?" Yes, I know there is a timer (RTO) involved in non-RACK implementations, but stick with me for a few minutes.

RACK starts a timer for every packet the moment it is sent. The threshold each timer uses is based on the smoothed round-trip time of the connection - essentially, the recent average of how long it takes for data to make the round trip to the receiver and back - plus a small safety margin of one millisecond. In standards-compliant mode, that margin is one quarter of the smoothed round-trip time instead. Either way, the threshold is capped at twice the smoothed round-trip time, with a configurable ceiling of four seconds.

The practical consequence: if a per-packet timer expires before an acknowledgment arrives, RACK can enter recovery based on timing alone - no waiting for three duplicate ACKs, no slow-path heuristics. When a selective acknowledgment (SACK) does arrive and skips over an unacknowledged packet, that also satisfies the RFC 6675 condition for entering loss recovery on its own. Either way, RACK moves quickly and the connection moves on.

Tail Loss: The Silent Killer RACK Finally Fixes

Consider the worst case for traditional TCP: you send two packets, the second is lost, and you have nothing else to send. The receiver gets the first packet and waits for the second. You wait for an acknowledgment. No duplicate ACKs arrive because there's no out-of-order delivery to trigger them. You're both just... waiting.

The only option is to let the retransmission timeout fire. In ONTAP, that's 230 milliseconds at the low end, and it can stretch to seconds on high-latency paths. When it finally fires, the congestion window doesn't just shrink, it can drop as low as one or two packets. The entire flow has to claw its way back from nearly nothing.

RACK's answer is the Tail Loss Probe (TLP). Rather than sitting idle, RACK arms a probe timer when no SACK-based recovery is possible. The probe fires at roughly twice the smoothed round-trip time - for a connection with a 10 ms round-trip, that's about 20 ms, compared to a 230+ ms retransmission timeout.

The probe sends either a retransmission of the suspected lost packet or a new data segment if one is available. When the receiver responds, RACK confirms the loss and enters recovery. The congestion window is adjusted - around 70% of its prior value with CUBIC, or 50% with NewReno - but it is not clamped down to a handful of packets the way a full retransmission timeout forces. The catastrophic collapse never materializes.

To prevent infinite probe loops on truly broken paths, RACK limits itself to two tail loss probes before falling back to the standard retransmission timer.

Reordering Without False Positives

Time-based loss detection has an obvious vulnerability: packet reordering. If a packet arrives late but not lost, a time-based detector might declare it missing and retransmit unnecessarily.

RACK handles this with adaptive thresholds. When out-of-order delivery is observed, RACK widens its detection window proportionally to the measured smoothed round-trip time. The window expands to accommodate observed reordering, preventing false retransmits on paths where delivery order naturally varies.

With Duplicate SACK (DSACK, RFC 2883) tracking enabled, each observed duplicate acknowledgment - meaning the receiver is reporting something it has already seen, because the network reordered a retransmission - adds more time to the threshold, up to a ceiling of twice the smoothed round-trip time. This is RACK learning from experience. If the network keeps reordering packets and the receiver keeps reporting duplicates, RACK keeps widening its patience window automatically, without manual tuning.

This reordering memory doesn't last forever. After 60 seconds without new out-of-order events, RACK resets the expanded window and the path gets a clean slate.

Microsecond Timers, Proportional Rate Reduction, and Pacing

This is where the engineering gets genuinely interesting.

Microsecond precision. All RACK, tail loss probe, and pacing callbacks are driven by a high-precision timer system built into the FreeBSD kernel. ONTAP runs a 1000 Hz kernel, which gives 1 ms tick granularity at the base level - but RACK's timer subsystem operates at microsecond precision, with typical jitter around 1 µs ± 0.1 µs. That accuracy matters: when RACK computes a threshold of "smoothed RTT plus one millisecond," it fires at close to exactly that point, forming a much more accurate picture of network state and enabling more precise reactions than millisecond-granular timers allow.

Proportional Rate Reduction instead of abrupt recovery. When traditional fast recovery enters a loss event, the behavior is a lull in transmission followed by a spike back to the old rate, before finally settling at the reduced rate - and that spike at the old transmission rate frequently causes self-inflicted additional packet losses. RACK uses Proportional Rate Reduction (PRR) instead. PRR dithers the transmission of additional packets much more evenly over time, avoiding the spike entirely. It also handles lost ACKs more gracefully - because some ACKs may be missing due to the same packet loss event that caused the recovery, PRR accounts for this uncertainty and paces retransmissions smoothly rather than reacting to each ACK in isolation.

Goodput-pegged pacing. RACK maintains a rolling history of actual throughput delivered to the receiver and paces its sending against that estimate rather than the raw congestion window. During slow start it paces at 250% of that baseline, during normal congestion avoidance at 200%, and during a periodic round-trip measurement phase at 60%. The congestion window still provides an upper bound, but the high-precision timer enforces the pacing rate and prevents bursting. The net result is smoother traffic shaping even during and after loss events.

Periodic minimum RTT measurement. Every 9.6 seconds, RACK deliberately reduces its pacing to 60% of its throughput baseline to get an honest measurement of the minimum round-trip time. This prevents the round-trip estimate from drifting upward over time as queues fill in the network - a drift that would otherwise cause the pacing window to grow incorrectly and inflate latency for all users of that path.

The Fine Print: SACK Dependency, Pipe Accounting, and When RACK Makes Sense

SACK is a soft requirement. If the remote side doesn't support selective acknowledgment, RACK becomes a lot less powerful. You still get pacing, but loss detection falls back to triple-dup-ack and retransmission on timeout.

In-flight data is counted conservatively. RACK estimates how much data is outstanding in the network by taking the difference between what has been sent and what has been fully acknowledged, then adding back any bytes that have been retransmitted but not yet confirmed received. Retransmitted unacknowledged bytes count twice. This is intentional - it prevents injecting too much data during burst recovery.

RACK is enabled per IPspace. In ONTAP, TCP RACK is configured on a per-IPspace basis. This makes it well suited for targeted deployments. Long-distance SnapMirror or FlexCache connections are a natural fit. Data SVMs serving home office or remote users over VPN can similarly be placed in a dedicated IPspace and have RACK enabled selectively, without affecting local datacenter traffic.

RACK has a cost - know when to use it. The high-precision per-packet timekeeping and pacing that make RACK effective are also computationally heavier than traditional TCP stacks. On clean, high-speed datacenter or campus networks where packet loss is rare and round-trip times are short, that CPU investment may not pay off - and for extremely high-throughput workloads like AI/ML training, where infrastructure is tightly engineered and controlled, the overhead of maintaining all those timers and pacing calculations could actually reduce the peak bandwidth a storage controller can deliver on a single TCP session. RACK is best reserved for the environments where it earns its keep: high-latency links, long-distance replication, WAN paths, and connections traversing Wi-Fi, DOCSIS, ADSL, or VPN tunnels.

What This Means for Your Workloads

Latency-sensitive NFS/SMB workloads. Any application that issues synchronous I/O - write-then-fsync, metadata-heavy operations, small-block random reads - pays the full cost of tail loss under traditional TCP. A single dropped packet at the end of a window stalls the operation until the retransmission timeout fires. RACK's tail loss probe typically resolves the same scenario in twice the round-trip time instead.

Challenging network paths. In real-world environments with difficult network infrastructure - home office users on VPN, remote workers over the public internet, high-latency WAN links for SnapMirror or FlexCache - TCP RACK has demonstrated 3× to 7× improvement in throughput. These are paths where traditional ACK counting leaves significant performance on the table; RACK recovers most of it.

High bandwidth-delay product links. On long-distance paths, microsecond timer precision allows RACK to form a much more accurate picture of network state and respond more precisely to loss events. The improvement is not just about eliminating quantization error - it's about enabling tighter, faster feedback between the sender and what the network is actually doing.

Where the network is clean, fast, and well-engineered - typical datacenter or campus deployments - RACK's advantages shrink and its CPU cost grows relatively larger. For those environments, the standard stack remains the right choice.

Three dup-ACKs was a reasonable answer to a 1999 problem. RACK is the answer to the problems you're actually running in production today - on the links that need it.

TCP RACK is available in ONTAP 9.19.1.