I built a new space-efficient sorted set for Goblin Core. Then I ran nearly a billion and a half Wikipedia edit increments through it.

It used less than half the memory of the lightest incumbent. It also finished ahead of Redis, Valkey, and Dragonfly, but reducing the memory footprint was the goal.

The new type is a packed zset. It keeps the sorted-set operations—look up a member, change its score, find its rank, return a range—but stores typed IDs and scores in a compact binary layout.

At 80,798,328 members, the INT32/FLOAT32 version used 2.10 GiB of process RSS. Dragonfly, the lightest incumbent, used 4.53 GiB. Packed's footprint was 53.6% smaller.

Here is the result, and then the machinery behind it.

Memory First

The English Wikipedia history dataset supplied native numeric page IDs. Each edit became one command:

ZINCRBY key 1 <page_id>

That builds a leaderboard of page-edit counts. It stores IDs and counters, not article text or revision bodies. The frozen input contained 1,483,700,913 increments.

The results are sorted by final process RSS, lowest first. Replay time is the secondary measure.

Engine Final process RSS Replay time
Goblin packed INT32/FLOAT32 2.10 GiB 7h 08m 02s
Goblin standard 3.46 GiB 7h 34m 35s
Dragonfly, one proactor 4.53 GiB 7h 22m 34s
Valkey 9.1.0 5.21 GiB 8h 19m 20s
Redis 8.8.0 5.47 GiB 8h 25m 41s
Redis 7.2.4 7.55 GiB 8h 52m 13s

Packed used 53.6–72.2% less RSS than the incumbents and 39.2% less than our own standard zset at the same final member count.

It also took 3.3–19.6% less replay time than the incumbents and 5.8% less than standard Goblin. That is a useful secondary result: the smaller footprint did not come with a replay-performance penalty in this run.

All engines passed full-state mapping verification with the same 80,798,328 members, the same total edit count, and no command errors.

This was one concurrent run on naamah, an AMD Threadripper PRO 5995WX machine. Each engine had one ordinary redis-cli client over a Unix-domain socket, waiting for replies rather than using --pipe. There was no explicit CPU pinning. These are final—not peak—RSS measurements and end-to-end replay times; the small timing margins need repeated trials before treating them as stable rankings.

Here is how the layout saves that memory while keeping updates efficient.

An ID Does Not Need to Be a String

A general-purpose sorted set accepts arbitrary string members. That is useful when the member really is a username or a piece of text.

But many applications are ranking page IDs, account IDs, device IDs, or UUIDs. The application already knows the member's type.

I wanted the storage engine to use that information.

Packed zsets support signed INT32, signed INT64, and UUID members, each paired with FLOAT32 or FLOAT64 scores. That gives six layouts. Integer members occupy four or eight binary bytes; UUIDs occupy sixteen. Scores occupy four or eight bytes.

There is no string-to-ID dictionary hiding elsewhere. Wikipedia was useful precisely because it already supplied numeric IDs. Turning arbitrary names into integers would require a mapping whose memory still belongs in the application's budget.

For this benchmark, a score/member tuple has eight bytes of payload. The whole zset costs more than that: it needs two indexes, and both have bookkeeping and spare capacity.

Making those indexes compact is the rest of the job.

Two Views, One Compact Layout

A sorted set answers two different questions:

  • Given a member, what is its score?
  • Given a score or rank, which members belong here?

I use a Swiss hash table for the member-to-score view. Keys and scores stay in their selected binary widths.

The ordered view is an arena-indexed B+ tree. Leaves hold contiguous score/member tuples. Branch and leaf references are 32-bit arena indices, rather than per-member heap pointers. Subtree counts support rank queries, and linked leaves support sequential range output.

At the end of the full replay, that tree had 288,451 leaves, 12,419 branches, and five levels.

This keeps the individual records small and puts neighboring records together. But it still leaves the expensive question: what happens when a member's score changes?

A counter workload asks that question a lot. About 94.6% of the commands in this replay updated an existing member.

Merge a Leaf, Not the World

Each leaf has a sorted base and a small pending-update section.

An update goes into that local section. Another update to the same member in the same leaf replaces its pending record. Old base entries are marked invalid, so the next merge does not have to rediscover which values were superseded.

When the pending section reaches its threshold, the leaf compacts: discard obsolete entries, sort the live pending records, and merge them backward into the leaf's reserved storage. The merge moves surviving runs in blocks and preserves the unchanged prefix.

It does not allocate a temporary vector the size of the whole sorted set.

For INT32/FLOAT32, a leaf's sorted capacity is 512 entries. The default merge exponent is 0.5, giving a local threshold of ceil(sqrt(512)) = 23 pending records. That is twenty-three records in one leaf, not a global merge every square root of eighty million members.

The exponent is configurable from 0 to 1. Reads reconcile the records in the leaves they visit without triggering maintenance merges. Sparse neighboring leaves can redistribute or merge, so moving scores does not leave a trail of empty blocks.

The update path also reuses the score slot already found in the Swiss table. It changes the tree first, then publishes the new score after that succeeds. Capacity is reserved for genuinely new members, rather than for ordinary rescoring.

Those choices keep the storage small and the maintenance local.

Count the Whole Object

After the full replay, packed's internally accounted allocation was 27.03 bytes per live member, versus 44.74 bytes for standard Goblin: 39.6% less object memory.

That is not the same measurement as the 39.2% process-RSS reduction. The object counter accounts for the data structure's allocations; RSS measures resident pages for the whole server. Both matter, and I am not comparing one engine's object counter with another engine's process RSS.

Standard was already using its adaptive integer score storage for this workload. No final merge was forced to improve packed's reported footprint.

The measured saving includes the actual indexes and their allocated capacity, not just the eight-byte payload.

What I Specialized

This is a custom type, with a narrower contract than a string-member zset.

Equal-score integer members sort numerically. Redis puts 10 before 2 in a string tie; packed integer sets put 2 before 10. UUID ties use the sixteen-byte value's lexicographic order. Members are returned in canonical form, not their original textual spelling.

Score precision is also explicit. FLOAT32 rounds to binary32, while FLOAT64 retains binary64 precision. The largest counter in this replay was 2,162,914, below FLOAT32's consecutive-integer limit of 16,777,216, so the increments stayed exact. We verified each representation's required order separately and compared the complete member-to-score mappings.

Only INT32/FLOAT32 was benchmarked here. Arbitrary strings, larger counters, other score distributions, and read-heavy workloads need the appropriate representation and their own measurements. Standard zsets remain available when their more general contract is what the application needs.

Using It

A workload made entirely of integer IDs can select packed zsets at startup:

goblin-core --zset-implementation packed-int32-float32 \
  --packed-zset-merge-exponent 0.5

Ordinary ZADD and ZINCRBY commands then create packed sets. Existing keys keep their representation; the flag does not convert them.

For an explicitly typed key, use the command prefix:

GOBLIN.PACKED_INT32_FLOAT32.ZINCRBY page-edits 1 1001
GOBLIN.PACKED_INT32_FLOAT32.ZRANGE page-edits 0 -1 WITHSCORES

The client still speaks ordinary RESP. page-edits is a normal database key name; its members are stored as binary integers.

The packed-zset documentation covers the six layouts and full command surface. The benchmark report and raw evidence contain the exact configuration, measurements, and verification.

I built this to spend less memory on data whose types the application already knows. At this scale, the packed layout cuts process RSS by more than half against even the lightest incumbent.

Finishing sooner is a useful bonus. The main result is 80.8 million verified page counters in 2.10 GiB of process memory.