Adam DePrince.

Database internals / compact indexes / C++23

Goblin Core

Structures Redis-compatible operations with typed packed sorted sets, a Swiss hash table, and an arena-indexed B+ tree.

Measured 80.8 million INT32/FLOAT32 page counters in 2.10 GiB final process RSS in the Wikipedia replay, with full-state mapping verification.

Inspect Read the layout, workload, and results. RESP/SBE transports and Pub/Sub also support the live event bus.

Storage hierarchy / memory and I/O / C++23

Goblin Store

Design A NUMA-aware, Memcached-compatible large-object cache and HTTP server.

Data path Keeps latency-sensitive object heads in RAM, streams warm data from SSD, and reads cold tails ahead from HDD.

Evidence Published comparisons with Memcached and extstore include workload, latency, memory, and storage-cost context.

Live feed ingestion / C++23

Goblin Slurp

Ingest A native bridge consumes an authorized commercial WebSocket market-data feed and splits source frames into individual events.

Normalize Publishes JSON or SBE events into Goblin Core for downstream subscribers.

Flow control A bounded queue and explicit backpressure make the handoff between ingestion and publication part of the design.

In use The live ingestion component in my research and intraday trading reference system. Goblin Core handles the internal fan-out.

Packet captures / latency and loss

Latency Lab

Inputs PCAP, PCAPNG, and Nasdaq-style binary captures.

Analysis Models rate, framing, queueing, and compression stages; reports per-layer latency distributions, percentiles through p99.99, and exact drop attribution.

Scope Compression compute time is not modeled. The published model makes that assumption explicit.

Messaging / multicast / Python integration

Aeron over InfiniBand

Transport An InfiniBand media extension for Aeron with UD and RC, multicast, zero-copy sends, and published hardware benchmarks.

Integration pyaeron adds native CPython bindings for IPC and UDP, an optional embedded media driver, and direct Python buffer handling.

Downstream consumers / streaming state

RTTA

State 319 public causal algorithms for indicators, online models, and streaming research signals behind C++23 and Python APIs.

Semantics Updates advance one observation at a time in replay and live operation.

Evidence Published benchmarks include algorithm coverage, CPU context, and separate state-only and Python-result measurements.

Rust / terminal software

Goblin Purrfect

Editor A WordPerfect-inspired, Unicode-native word processor written in Rust.

Implementation Equation parsing and interactive previews run inside the editor, using real font outlines and Kitty, Sixel, or Unicode Braille rendering.

Formats Images, Markdown interchange, and LaTeX/PDF output. Read the project and file-compatibility limits.

SIMD sequence algorithms

stride-align

Problem Alignment and distance algorithms are expensive inner loops with very different instruction sets across CPUs.

Work Smith-Waterman, Needleman-Wunsch, edit distance, phonetics, and time-series distance for C++ and Python across several architecture families.

Evidence The benchmark suite publishes backend, scoring, width, workload, and CPU context rather than one universal speedup.

LoongArch optimization

DragonArray

Problem Useful hardware remains slow when upstream packages stop at generic code or narrower vector extensions.

Work Installable LoongArch wheels, NumPy LASX work, SIMD transcendental functions, and a BLAKE3 LoongArch port.

Evidence Patches and benchmarks are published together so architecture support is something users can run, not a slide.

Small-state estimation

fast-kalman

Problem Tiny state-space models should not pay dynamic-matrix or per-tick Python overhead.

Work Fixed-size C++23 Kalman variants, direct Python bindings, and batched updates with irregular time and observation variance.

Measured The published batched two-state, one-observation linear case runs at about 43 ns per measurement on its documented test machine.

External sorting

bsort

Problem Fixed-record data sets larger than RAM turn locality and cache behavior into the algorithm.

Work A C++ and Python external sorter designed around local access, predictable records, and LRU-friendly behavior.

History The experiment became the technical basis for a startup and remains available as open source.