Goblin Core is a Redis-compatible database. Redis’s core command execution is single-threaded, giving it low latency, cheap synchronization, and simple transaction semantics. Goblin Core follows that tradition.

Goblin Core’s performance is high, beating the incumbents in many areas, but where it really shines is memory consumption. On realistic datasets, it requires substantially less memory. In one benchmark using 14.3 billion Lichess games, Goblin Core used only 64% as much memory as the next-best competitor while beating all but one competitor on speed.

We pay careful attention to memory because our promise is simple: do more with less hardware.

That attention extends to little details—including how we save your data.

SAVE and BGSAVE

Redis exposes two commands for saving data to disk: SAVE and BGSAVE.

Remember that Redis command execution is single-threaded, and Goblin Core is no different. That single-threaded design gives us cheap, low-latency synchronization.

When you run SAVE, the database stops processing requests, writes a snapshot of everything in memory to disk, and then continues. For a small database, that is acceptable. For a large database containing millions or billions of objects, the pause may last seconds or minutes.

That is less acceptable.

BGSAVE solves this by forking the process. Linux creates a child process that appears to have a copy of the parent’s memory. The child writes the snapshot while the main process soldiers on, continuing to serve requests.

This works well because Linux supports copy-on-write, or COW.

When a process forks, the operating system does not immediately copy all of its memory. Instead, it performs some bookkeeping, gives both processes their own virtual address spaces, and points them at the same physical pages. Those pages are marked read-only.

If either process tries to modify one, the operating system steps in, copies the page, and gives each process its own version.

No copy is made until someone writes.

When the Cow Starts Eating Memory

Writes consume memory during a background save, but that is usually fine. For a long-running Redis-compatible database, the writes arriving during a save window of seconds or minutes are normally small compared with the size of the entire dataset.

As SET, ZADD, and HSET modify previously shared pages, the first write to each page causes Linux to make a private copy. As more pages are dirtied, memory consumption rises.

Usually, this works beautifully.

Then a large hash table needs to grow.

Goblin Core uses Swiss hash tables. Like most hash tables, they eventually run out of room and must allocate a larger table. During that growth event, the entries are redistributed across the new table.

In our case, the table contains pointers to keys and values stored in separately managed memory arenas. We are not moving all the key and value data itself. Even so, a large structure can contain megabytes of metadata and pointers that must be rewritten.

When a table is rebuilt, a great many pages are written at once. During BGSAVE, every newly dirtied page can require a COW copy. The operating system suddenly starts scurrying around looking for memory to hold all those newly dirtied pages.

Imagine that most of your database is one enormous Redis hash and Goblin Core is already using 80% or 90% of the machine’s memory. That is not unreasonable: saving money is the entire point of using a memory-efficient Redis-compatible database.

A table reshuffle can consume the remaining memory. Linux invokes its out-of-memory killer. It may kill the server, the snapshot process, or both.

Out of memory.

The COW goes OOM.

Growing Without Doubling

Goblin Core’s hash-table growth is controlled by occupancy thresholds.

Normally, when 97% of the slots in a Swiss table are occupied, we grow the table by approximately 19%.

Why 19%?

Many libraries double a vector or table whenever it becomes full. The resize is an O(n) operation, but because it happens infrequently, the amortized cost of insertion remains O(1).

We do not like doubling our tables. A freshly doubled table is half empty, and empty slots still consume memory.

Instead, Goblin Core grows by a factor of 24, or approximately 1.19. Four growth events bring the table to twice its original size.

We waste much less memory inside the table, but we grow and copy more frequently. The amortized cost remains O(1), just with a larger constant.

Trade-offs.

You are probably using Goblin Core because your cloud-memory bill is too high. Modern CPUs are also very good at the straight-line memory copies involved in rebuilding a table. Random memory access has become relatively more expensive compared with sequential copying.

That is one reason Goblin Core makes different trade-offs from Redis’s historical design.

What Changes During BGSAVE

A 1.19× replacement table requires about 40% less new allocation than a 2× replacement table, but it still touches a lot of pages. During BGSAVE, that creates a substantial COW burden and risks making the cow go OOM.

Our solution is simple: delay the growth.

While a BGSAVE is running, Goblin Core raises the default growth threshold from 97% to 99%. That gives the table two extra percentage points of occupancy before it must be rebuilt.

There is a performance cost. As the table becomes more tightly packed, finding an empty slot takes a little longer. An unsuccessful HGET may also require more probing.

But the COW burden remains reasonable.

What happens if you are writing like mad and reach 99% anyway?

If an insertion would force the table to grow while BGSAVE is active, Goblin Core returns an error rather than beginning a potentially catastrophic resize.

One of Goblin Core’s guiding principles is that matters of reliability—durable logging, persistence policy, save timing, and recovery—belong in layers that specialize in them.

Goblin Core will never ask you to choose an AOF fsync interval. We delegate durable logging to Kafka or Redpanda. Goblin Core can write changes to that log as they occur and replay them during recovery.

During a background save, preserving the running database is more important than accepting one more write that might kill both processes.

Arena Compaction

And yes, Goblin Core also restricts memory-arena compaction during BGSAVE.

Compaction is less dangerous than rebuilding a giant hash table, but it can still dirty a large number of pages. Deferring it may temporarily reduce memory efficiency or force us to allocate another block, but that is preferable to causing a COW explosion during a save.

So yes: Goblin Core works hard to keep COW from becoming OOM—and to keep the cow from going moo.