I was qualifying Goblin Core’s low-latency network path for AWS EFA when a supposedly harmless piece of TCP code began stalling the entire server.

I am adding low-latency AWS networking support to Goblin Core.

AWS EFA exposes its network through libfabric. Renting expensive cloud machines is a poor time to discover that the client, server, completion queues, receive buffers, or small-message handling are broken, so I first qualified the complete libfabric path on hardware I own.

I connected two 14-year-old, four-socket Intel Xeon servers with a direct 100 Gb/s Mellanox ConnectX-5 link.

The machines are ancient by cloud standards, but they are still useful instruments. More importantly, they are mine. I can pin processes, disable distractions, change the server, rerun the tests, and keep investigating until I understand the result.

I tested several combinations:

  • ordinary kernel TCP;
  • libfabric’s software tcp provider;
  • verbs;ofi_rxm over the Mellanox hardware;
  • Redis RESP2 and Goblin Core’s compact SBE protocol;
  • automatic small-message injection and forced fi_send.

Every test used a pipeline depth of one: send one operation, wait for its complete response, validate it, and then send the next.

This was a latency test, not an attempt to fill a 100 Gb/s pipe.

The First Result

Across eight Redis-shaped operations, the fastest configuration completed a round trip in 6.39 microseconds at the arithmetic-average median.

The ordinary Redis-compatible baseline—RESP2 over kernel TCP—took 90.33 microseconds.

Path Protocol Average p50
Kernel TCP RESP2 90.33 µs
Libfabric tcp RESP2 39.59 µs
verbs;ofi_rxm RESP2 7.38 µs
verbs;ofi_rxm SBE 6.39 µs

The fastest path was 14.1 times lower latency than ordinary kernel TCP with RESP2.

The eight operations were:

PING, SET, GET, HSET, HGET, ZADD, ZSCORE, and Pub/Sub.

The Pub/Sub test did not stop when the server acknowledged PUBLISH. Timing ended only after a subscriber on a separate connection received and validated the message.

Each distribution included 20,000 warmup operations followed by 200,000 measured operations. Every response was checked for correctness.

RESP Was Not the Main Problem

The fastest configuration used SBE, Goblin Core’s compact binary protocol.

But RESP2 over the same verbs;ofi_rxm transport still averaged 7.38 microseconds.

That matters.

The binary format helped, but most of the improvement came from changing the transport and the surrounding software path. Redis-compatible RESP over libfabric verbs was still more than twelve times faster than RESP over Goblin Core’s ordinary socket implementation.

Even libfabric’s software tcp provider was interesting. It averaged 39.59 microseconds with RESP2, compared with 90.33 microseconds through the ordinary kernel TCP path.

That does not prove that libfabric TCP is universally faster than sockets. It shows something narrower: in this server, a large part of the latency lived above or around the network itself.

A 100 Gb/s network card does not automatically give an application low latency.

The parser, buffer ownership, syscall pattern, polling model, completion handling, NUMA placement, and application architecture still decide how quickly one request becomes one validated response.

Then the Tail Went Bad

The median numbers looked good.

The tail did not.

An earlier qualification run produced an average SBE/RDM p99.99 of roughly 857 microseconds.

That is absurdly large beside a single-digit-microsecond median. Something was periodically stopping the server.

My first suspicion was message reordering.

That would have been a plausible explanation. The client maintains a bounded reorder window, and a lost or delayed sequence could leave later messages waiting behind it.

So I instrumented the path.

Across nearly two million publisher replies and 220,001 subscriber replies, the client observed:

  • no missing sequence;
  • no duplicate;
  • no reorder-window overflow;
  • no stranded frame;
  • no pending reordered message.

The server’s request counters were clean too.

The transport was not losing control of the messages.

The stall came from somewhere much more boring.

The Bootstrap TCP Listener

Goblin Core’s polled server still had an ordinary TCP listener used for establishing bootstrap connections.

On every pass through its spin loop, the server called nonblocking accept(), whether or not a connection was waiting.

Most of those calls returned EAGAIN.

That looked harmless.

It was not.

On Linux, even an unsuccessful accept() can create and retire socket-, file-, and inode-related objects before it discovers that there is no connection to accept.

Those objects eventually have to be reclaimed.

Tracing showed repeated RCU softirq batches, some lasting roughly 0.4 to 1.1 milliseconds.

The glamorous 100 Gb/s low-latency path was periodically being stalled by a bootstrap listener repeatedly asking Linux whether anyone was at the door.

The Fix

The fix was simple:

Check whether the listener is ready before calling accept(), and drain it only when a connection is actually waiting.

Before the change, one trace contained 4,119 RCU softirq runs longer than 100 microseconds.

During a two-second active trace after the change, every softirq run remained below 100 microseconds. The longest was 20.6 microseconds.

The immediate control run dropped the average SBE/RDM p99.99 from:

857.47 microseconds

to:

29.47 microseconds

A later full provider-matrix run reproduced the result at 30.83 microseconds.

The median had already been good.

One unnecessary syscall pattern was destroying the tail.

The Boring Code Owns the Machine Too

The listener was not part of the interesting design.

It was not RDMA.

It was not the compact protocol.

It was not a Redis data structure.

It was not the Mellanox card.

It was a small bootstrap path doing unnecessary work inside the wrong loop.

That is common in low-latency systems. The headline mechanism can be correct while something apparently harmless nearby dominates the failure distribution.

The only reliable response is to measure the complete system, preserve anomalous results, and trace the mechanism instead of explaining the anomaly away.

What This Test Proved

The work qualified Goblin Core’s libfabric RDM implementation across two physical machines with:

  • real request and response validation;
  • both RESP2 and SBE;
  • small-message injection;
  • explicit fi_send and transmit-completion handling;
  • posted receives and receive completion queues;
  • Pub/Sub over separate publisher and subscriber connections;
  • bounded message reordering;
  • stable low-microsecond medians;
  • p99.99 behavior in the tens of microseconds after fixing the listener.

It also showed that the transport and the surrounding software architecture can matter far more than the nominal speed printed on the network card.

What It Did Not Prove

This was not an AWS EFA hardware benchmark.

The native efa provider was not available on these local machines. I used AWS’s libfabric build with its tcp and verbs;ofi_rxm providers over a direct ConnectX-5 Ethernet link.

EFA still needs its own run.

The point of this work was to arrive in AWS with a qualified implementation, a reproducible benchmark harness, validated send and receive paths, and a known-good local baseline.

Not to rename a local verbs result as an EFA result.

The Part I Enjoy

This is the kind of systems problem I like most.

Start with a surprisingly bad number.

Refuse to average it away.

Instrument the obvious suspect.

Discover that the obvious suspect is innocent.

Trace the whole machine.

Find a tiny piece of boring code manufacturing millisecond stalls inside a microsecond system.

Fix it.

Run everything again.

The result was not merely a faster benchmark. It was an explanation.

And now the code is ready for the next machine.