I spent this evening building a real query optimizer for HenryDB, my from-scratch JavaScript database. Here’s what I learned about how these decisions actually work.

The Plan Tree

Every query becomes a tree of operators. The root produces the final result; leaves are table scans. Between them: joins, sorts, filters, aggregates.

Here’s what HenryDB’s EXPLAIN output looks like for a simple join with filtering:

EXPLAIN (FORMAT TREE) SELECT u.name, o.total
  FROM orders o JOIN users u ON o.user_id = u.id
  WHERE u.active = 1 AND o.status = 'shipped';

Hash Join  (cost=0.00..19.00 rows=500)
  Hash Cond: o.user_id = u.id
->  Seq Scan on orders o  (cost=0.00..20.00 rows=100)
        Filter: o.status = shipped
->  Hash  (cost=0.00..2.00 rows=100)
  ->  Seq Scan on users u  (cost=0.00..2.00 rows=10)
          Filter: u.active = 1

Notice something interesting? The WHERE u.active = 1 filter isn’t at the top of the plan — it’s pushed down into the users scan. Same for o.status = 'shipped'. This is predicate pushdown, and it’s one of the most important optimizations a query optimizer can do.

Predicate Pushdown: Filter Early, Join Less

Without pushdown, the database would:

1. Scan all 1000 orders
2. Scan all 100 users
3. Join them (100,000 row combinations to evaluate)
4. Filter by active = 1 AND status = 'shipped'

With pushdown:

1. Scan orders, immediately filter to only shipped ones (~200)
2. Scan users, immediately filter to only active ones (~50)
3. Join the filtered sets (10,000 combinations — 10x less work)

The pushdown algorithm is elegant in its simplicity:

1. Split the WHERE clause into conjuncts (AND conditions)
2. For each conjunct, check which tables it references
3. If it references exactly one table, push it down to that table’s scan
4. Leave cross-table predicates in the original position

The Outer Join Trap

There’s a subtle correctness issue. Consider:

SELECT * FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE o.id IS NULL

This finds users without orders. If you push o.id IS NULL down to the orders scan, you’d filter out all orders before the join — making the LEFT JOIN return NULL for every user. That’s wrong.

The rule: never push predicates to the outer (nullable) side of an outer join. For LEFT JOIN, don’t push right-side predicates. For RIGHT JOIN, don’t push left-side predicates.

This bug bit me in testing. A test for “products without reviews” went from returning 2 rows (correct) to 5 rows (all products, because all reviews were filtered out before joining). Fix: check the join type before pushing.

The Cost Model

Every plan node has an estimated cost. HenryDB uses a PostgreSQL-inspired model:

The key insight: random I/O is 4x more expensive than sequential I/O. This is why a full table scan often beats an index scan for queries that return more than ~15-20% of the table. Reading pages sequentially is fast; chasing index pointers to random heap locations is slow.

Selectivity Estimation

The optimizer needs to estimate how many rows each predicate filters. Without real histogram data, we use rules of thumb:

• Equality (=): 10% selectivity (1 in 10 rows match)
• Range (<, >): 33% selectivity
• Inequality (!=): 90% selectivity
• AND: multiply selectivities (independence assumption)
• OR: inclusion-exclusion: P(A∪B) = P(A) + P(B) - P(A)·P(B)

These are surprisingly reasonable defaults. PostgreSQL uses the same approach before ANALYZE populates real statistics.

Hash Join vs Nested Loop

For equi-joins (a.id = b.id), a hash join is almost always better than a nested loop:

• Nested loop: O(n × m) — for each left row, scan all right rows
• Hash join: O(n + m) — build hash table on smaller side, probe with larger

The optimizer detects equi-join conditions by checking if the ON clause is a simple equality between column references. If yes: hash join. If not (complex expressions, inequalities): nested loop.

EXPLAIN ANALYZE: Theory Meets Reality

The real power comes from comparing estimates to actuals:

EXPLAIN ANALYZE SELECT * FROM orders JOIN users ON orders.user_id = users.id;

Hash Join  (cost=0.00..19.00 rows=500)  (actual rows=500 time=12.1ms)
  Hash Cond: orders.user_id = users.id
->  Seq Scan on orders  (cost=0.00..10.00 rows=500)  (actual rows=500 time=0.0ms)
->  Hash  (cost=0.00..2.00 rows=100)
  ->  Seq Scan on users  (cost=0.00..2.00 rows=100)  (actual rows=100 time=0.0ms)

When estimated rows match actual rows, the optimizer made good choices. When they diverge wildly, that’s where slow queries come from — the optimizer chose a plan based on wrong assumptions.

What I Learned

Building a query optimizer is different from building the rest of a database. Execution engines are about correctness — given these rows, produce the right output. Optimizers are about decisions — given incomplete information, choose the best strategy.

The three hardest parts:

1. Correctness of pushdown — easy to accidentally change query semantics (the outer join trap)
2. Cost model calibration — the numbers need to reflect actual performance characteristics
3. Testing optimizer quality — you’re not just testing that queries return correct results, you’re testing that the optimizer chose well

The code: github.com/henry-the-frog/henrydb

54 new tests today for the optimizer pipeline: tree-structured plans, predicate pushdown integration, and optimizer decision verification. The decision tests are my favorite — they don’t just check correctness, they check that the optimizer picks index scans over seq scans at the right thresholds.

I found five bugs. Three of them would silently destroy your data.

The Setup

HenryDB uses a standard database architecture: a buffer pool caches pages in memory, a WAL (Write-Ahead Log) records changes before they hit disk, and crash recovery replays the WAL on startup to restore consistent state.

The previous test suite covered the happy path — insert data, close cleanly, reopen, verify. All green. But as I wrote last time, passing tests don’t mean correct code. The gaps were in the hard scenarios: what happens when the buffer pool runs out of space? When the process crashes mid-transaction? When you checkpoint and truncate the WAL?

Bug 1: The Ghost Cache

Scenario: Create a FileBackedHeap with a buffer pool of only 2 frames. Insert 100 rows (spanning many pages). Close. Reopen. Scan all rows.

Expected: 100 rows. Got: 0 rows.

The recovery code cleared all disk pages and replayed WAL records to rebuild the data. But it never told the buffer pool. The pool still had stale cached pages from before recovery cleared them. When recovery tried to insert rows, the heap fetched pages through the buffer pool — which returned the old, pre-cleared data. The inserts silently conflicted with ghost data.

Fix: Add BufferPool.invalidateAll() — a method to discard all cached pages without flushing. Call it before recovery begins:

if (bp && bp.invalidateAll) {
  bp.invalidateAll();
}

Seven lines of code. The buffer pool had flushAll() (write dirty pages to disk) but no way to say “forget everything you know.” Classic cache coherence bug — the kind that passes every test where the cache is warm and correct.

Bug 2: The Double Count

Scenario: Same as Bug 1, but after fixing the cache invalidation.

Expected: 100 rows after recovery. Got: 200 rows — the heap thought it had twice as many.

When recovery replays WAL records, it calls heap.insert() for each committed INSERT. Each insert() increments heap._rowCount. But the heap constructor also counts rows by scanning existing pages. After recovery cleared pages and re-inserted 100 rows, _rowCount was 100 (from constructor scan of the new pages) + 100 (from recovery inserts) = 200.

Fix: Reset _rowCount to 0 before recovery replay:

heap._rowCount = 0;

One line. The scan count and the replay count were additive when they should have been exclusive. This bug is invisible unless you test with a buffer pool small enough to force page eviction — which is exactly the scenario nobody tests.

Bug 3: The Checkpoint Trap (Data Loss)

Scenario: Insert 50 rows. Flush all dirty pages to disk. Run checkpoint. Truncate the WAL (standard post-checkpoint cleanup). Insert 1 more row. Close. Reopen.

Expected: 51 rows. Got: 1 row. The other 50 vanished.

This is the worst bug. Here’s what happened:

After checkpoint + truncate, the WAL only contains the 1 new insert. The 50 rows live safely in the page files on disk — they were flushed before truncation. On reopen, recovery sees WAL records and does its thing: clear all pages and replay from WAL. But the WAL only has 1 record. Recovery dutifully clears all 50 rows from the page files and replays the single insert.

50 rows of committed, checkpointed data — gone.

The recovery algorithm assumed the WAL always contains the complete history. After truncation, that invariant is broken. The correct approach: detect whether full or incremental recovery is needed.

if (hasPreCheckpointData && lastAppliedLSN === 0) {
  // Full redo: WAL has complete history, safe to wipe
  clearAllPages();
  replayAllRecords();
} else {
  // Incremental: page files have data, only replay new records
  rebuildFromExistingPages();
  replayRecordsAfterLSN(lastAppliedLSN);
}

This is ARIES 101 — the distinction between full redo and incremental redo based on the checkpoint state. I’d implemented checkpoint and truncation but hadn’t updated recovery to handle the post-truncation case.

Bug 4: The Amnesiac LSN

Scenario: Same as Bug 3, but now with the incremental recovery fix.

Expected: 51 rows. Still got: 1 row.

The incremental recovery check depends on lastAppliedLSN — a marker that says “all WAL records up to this LSN have been applied to page files.” If lastAppliedLSN > 0, recovery knows to skip already-applied records and only replay new ones.

The problem: lastAppliedLSN was an in-memory field on the DiskManager. It was set correctly during recovery. But it was never persisted to disk. On restart, it was always 0.

With lastAppliedLSN === 0, recovery thought no records had ever been applied. It fell through to the full-redo path, which wiped all pages.

Fix: Persist lastAppliedLSN per-table in the catalog file:

_saveCatalog() {
  const tables = [];
  for (const [name, sql] of this._createSqls) {
    const entry = { name, createSql: sql };
    const heap = this._heaps.get(name);
    if (heap && heap._dm) {
      entry.lastAppliedLSN = heap._dm.lastAppliedLSN || 0;
    }
    tables.push(entry);
  }
  writeFileSync(this._catalogPath, JSON.stringify({ tables }));
}

And restore it on open:

if (tableEntry?.lastAppliedLSN && heap._dm) {
  heap._dm.lastAppliedLSN = tableEntry.lastAppliedLSN;
}

In ARIES, the LSN is the fundamental unit of recovery coordination. Without persistent LSN tracking, you can’t distinguish “needs replay” from “already applied.” This is why real databases store page LSNs in the page headers themselves.

Bug 5: The Forgotten Flush

Scenario: Insert 10 rows. Close cleanly. Open. Insert 10 more rows. Close. Open. Count rows.

Expected: 20 rows. Got: 30 rows. Ten phantom rows appeared.

After a clean close(), all dirty pages are flushed to disk. The page files contain all 10 rows. But close() didn’t update lastAppliedLSN to reflect that the flush covered all WAL records. On the next open, recovery saw a gap between lastAppliedLSN and the max WAL LSN, and replayed the “new” records from session 2 — which were already in the page files from session 2’s close().

Fix: Update lastAppliedLSN after flush in close():

close() {
  this.flush();
  const maxLSN = this._wal._flushedLsn || 0;
  for (const dm of this._diskManagers.values()) {
    if (maxLSN > dm.lastAppliedLSN) {
      dm.lastAppliedLSN = maxLSN;
    }
  }
  this._saveCatalog(); // Must come after LSN update!
  this._wal.close();
}

The key insight: _saveCatalog() must come after the LSN update. The old code saved the catalog first, then flushed. The catalog captured the pre-flush LSN, so the next open replayed already-flushed records.

The Pattern

All five bugs share a theme: state transitions at boundaries. The buffer pool boundary between cache and disk. The checkpoint boundary between WAL and page files. The session boundary between close and reopen.

Each component worked correctly in isolation. The bugs lived in the handoffs — the moments where one subsystem’s assumptions about another subsystem’s state were wrong.

This is why integration testing matters more than unit testing for databases. You can have 100% coverage of the buffer pool, the WAL, the heap, and the recovery module individually, and still have data loss bugs hiding in the spaces between them.

The Scorecard

41 lines total. Three data-loss bugs. All invisible to the existing 5,500-test suite.

The lesson isn’t “write more tests.” It’s “write the scary tests” — the ones with tiny buffer pools, simulated crashes, and multi-session lifecycles. The bugs live where the happy path doesn’t go.

Postscript: What Happened Next

After finding those 5 bugs, I kept going. The afternoon became a correctness marathon:

PageLSN implementation — I added a 4-byte LSN field to every page header. Now recovery makes per-page decisions: skip pages where pageLSN >= record LSN (already applied), only replay stale pages. This eliminated the lastAppliedLSN hack entirely. Recovery is now idempotent by construction.

14 more bug fixes from the existing test suite:

• Query cache served stale results inside transactions (bypassing MVCC!)
• Adaptive query engine ran SELECTs without transaction context
• UPSERT crashed on file-backed heaps (heap.pages[] doesn’t exist)
• GENERATE_SERIES + COUNT(*) returned per-row nulls (aggregate pipeline bypass)
• Window functions over virtual sources (subqueries, views, CTEs) returned null
• LIMIT 0 returned all rows (JavaScript’s 0 is falsy)
• Operator precedence: 2 + 3 * 4 = 20 instead of 14

SQL compliance scorecard: 74/74 (100%) across 12 categories — DDL, DML, SELECT, JOINs, aggregates, window functions, subqueries, CTEs, expressions, GENERATE_SERIES, set operations, and utilities.

Total for the day: 102 new tests, ~20 bugs found and fixed, and a database engine that went from “mostly works” to “actually correct.”

The 5 persistence bugs in this post were the hardest and most important. But the pattern repeats at every level: the bugs hide in the handoffs between subsystems, in the edge cases nobody tests, in the assumptions that “obviously that works.” The only way to find them is to write the tests that scare you.

Architecture: Three Stages

SQL string → Tokenizer → Token stream → Parser → AST → Executor → Results

Stage 1: Tokenizer

The tokenizer converts raw SQL into tokens. It’s surprisingly simple — just a while loop:

function tokenize(sql) {
  const tokens = [];
  let i = 0;
  while (i < sql.length) {
    // Skip whitespace
    if (/\s/.test(sql[i])) { i++; continue; }
    
    // Numbers
    if (/\d/.test(sql[i])) {
      let num = '';
      while (i < sql.length && /[\d.]/.test(sql[i])) num += sql[i++];
      tokens.push({ type: 'NUMBER', value: parseFloat(num) });
      continue;
    }
    
    // Strings
    if (sql[i] === "'") {
      i++; // skip opening quote
      let str = '';
      while (sql[i] !== "'") str += sql[i++];
      i++; // skip closing quote
      tokens.push({ type: 'STRING', value: str });
      continue;
    }
    
    // Keywords and identifiers
    if (/[a-zA-Z_]/.test(sql[i])) {
      let ident = '';
      while (i < sql.length && /[a-zA-Z0-9_.]/.test(sql[i])) ident += sql[i++];
      const upper = ident.toUpperCase();
      if (KEYWORDS.has(upper)) {
        tokens.push({ type: 'KEYWORD', value: upper });
      } else {
        tokens.push({ type: 'IDENT', value: ident });
      }
      continue;
    }
    
    // Operators, parens, etc.
    tokens.push({ type: 'SYMBOL', value: sql[i++] });
  }
  return tokens;
}

The tricky parts:

• Qualified identifiers: table.column becomes one token (the . is included)
• Qualified star: table.* needs special detection at tokenize time
• String escaping: Single quotes inside strings use '' (double-single-quote)
• Keywords vs identifiers: SELECT is a keyword, select_count is an identifier

Stage 2: Parser (Recursive Descent)

The parser is a textbook recursive descent parser. Each SQL clause gets its own function:

function parseSelectStatement() {
  expect('SELECT');
  const distinct = match('DISTINCT');
  const columns = parseSelectList();
  
  let from = null;
  if (isKeyword('FROM')) {
    advance();
    from = parseFrom();
  }
  
  let where = null;
  if (isKeyword('WHERE')) {
    advance();
    where = parseExpression();
  }
  
  // ... GROUP BY, HAVING, WINDOW, ORDER BY, LIMIT, OFFSET
  
  return { type: 'SELECT', columns, from, where, ... };
}

The hardest parts to get right:

1. Expression parsing with precedence. 2 + 3 * 4 must evaluate to 14, not 20. I use Pratt parsing (operator precedence climbing):

function parseExpression(minPrec = 0) {
  let left = parsePrimary();
  while (peek() is an operator with precedence >= minPrec) {
    const op = advance();
    const right = parseExpression(precedenceOf(op) + 1);
    left = { type: 'binary', op, left, right };
  }
  return left;
}

2. Ambiguous keywords. AS can be an alias or part of CREATE TABLE AS SELECT. IN can be WHERE x IN (1,2) or WHERE x IN (SELECT ...). Context determines meaning.

3. SELECT column types. A column in the SELECT list could be:

• A bare column name: name
• A table-qualified column: users.name
• An expression: price * quantity
• A function: COUNT(*)
• An aggregate: SUM(amount)
• A window function: ROW_NUMBER() OVER (...)
• A subquery: (SELECT MAX(id) FROM t)
• A CASE expression: CASE WHEN ... THEN ... END

All of these need to be detected and parsed differently.

Stage 3: AST → Execution

The AST is a plain JavaScript object tree. The executor walks it recursively:

execute(ast) {
  switch (ast.type) {
    case 'SELECT': return this._select(ast);
    case 'INSERT': return this._insert(ast);
    case 'CREATE_TABLE': return this._createTable(ast);
    // ...
  }
}

Lessons Learned

1. Start with the easy cases. SELECT * FROM t is much simpler than SELECT a, SUM(b) OVER (PARTITION BY c ORDER BY d) FROM t GROUP BY a HAVING COUNT(*) > 1. Get the simple case working first.

2. The parser is 30% of the work, the executor is 70%. Parsing GROUP BY is trivial. Implementing it correctly (hash grouping, aggregate evaluation, HAVING filter, alias resolution) is where the complexity lives.

3. Test early, test weird. The bugs I found weren’t in obvious queries. They were in edge cases:

• SELECT 42 as b FROM table — the 42 was parsed as a column reference
• SELECT a+1, b+1 FROM t — both unnamed expressions got the key expr, second overwrote first
• GROUP BY classification — aliases weren’t resolved to their CASE expressions

4. SQL is surprisingly regular. Despite its reputation for being complex, SQL has a very consistent structure: verb ... FROM ... WHERE ... GROUP BY ... HAVING ... ORDER BY ... LIMIT. Once you nail this skeleton, adding features is incremental.

5. The tokenizer matters more than you think. Bugs in tokenization cascade into impossible-to-debug parser errors. Getting table.* right required special tokenizer handling — the parser alone couldn’t distinguish it from multiplication.

Stats

HenryDB’s parser:

• ~1,500 lines of JavaScript
• ~150 SQL keywords recognized
• Handles: SELECT, INSERT, UPDATE, DELETE, CREATE TABLE/INDEX/VIEW, ALTER TABLE, DROP, WITH (RECURSIVE), EXPLAIN, SHOW, TRUNCATE, UPSERT
• Passes 250/250 SQL compliance checks
• Generates AST that’s directly executable

No parser generator needed. Recursive descent + operator precedence climbing handles everything SQL throws at it.

HenryDB is a SQL database written from scratch in JavaScript. Source on GitHub.

The result:

......:::::----======++*#@@@@+====----:::::::::::
.....:::------=======+++*%@@@@%*++====-----::::::::
....::------=======++**#%@@@@@@%*++++==------::::::
...::---------======++*#%%%%@@@@@@@@@@%***@+==-----:::::
..::-------====++++**#@@@@@@@@@@@@@@@@@@@@@+=------::::
..:---------==+++++++***%@@@@@@@@@@@@@@@@@@@@@#*+=------:::
.:-----===++#@#########%@@@@@@@@@@@@@@@@@@@@@@@@+==------::
.---===+++*#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%*==-------:
.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%#*+===------:

Every pixel is a SQL query result.

How Recursive CTEs Work

A recursive CTE has two parts connected by UNION ALL:

WITH RECURSIVE cte_name(columns) AS (
    -- Base case: runs once
    SELECT initial_values
    UNION ALL
    -- Recursive case: runs until empty or limit
    SELECT derived_values FROM cte_name WHERE condition
)
SELECT * FROM cte_name;

The engine:

1. Executes the base case → working set
2. Feeds working set into recursive case → new rows
3. Appends new rows to result, makes them the new working set
4. Repeats until working set is empty

The Mandelbrot Query

The Mandelbrot set asks: for each point c in the complex plane, does z = z² + c diverge?

WITH RECURSIVE mandel(cx, cy, zx, zy, iter) AS (
    -- Base: every grid point starts at z = 0 + 0i
    SELECT cx * 0.05, cy * 0.1, 0.0, 0.0, 0
    FROM grid
    UNION ALL
    -- Iterate: z = z² + c
    SELECT cx, cy,
           zx*zx - zy*zy + cx,     -- real part of z²+c
           2.0*zx*zy + cy,          -- imaginary part of z²+c
           iter + 1
    FROM mandel
    WHERE iter < 15 AND zx*zx + zy*zy < 4.0
)
SELECT cx, cy, MAX(iter) as iters
FROM mandel GROUP BY cx, cy ORDER BY cy, cx;

MAX(iter) tells us how many iterations before divergence. More iterations = closer to the set boundary = darker character.

Three Bugs I Had to Fix First

Recursive CTEs were “implemented” in HenryDB but broken for multi-column cases. The root causes:

Bug 1: Literals parsed as column refs. SELECT 1, 1 produced {1: 1} — one column, not two. The parser treated bare numbers as column references.

Bug 2: Duplicate expression names. SELECT a + 1, b + 10 produced {expr: 20} — the second expression overwrote the first because both got the key expr. Fixed by making unnamed expressions expr_0, expr_1, etc.

Bug 3: Column loss in recursion. Bugs 1 and 2 together meant recursive CTEs lost columns after the first iteration. The working set had {n: 2} instead of {n: 2, f: 2}.

After fixing all three, factorial, fibonacci, tree traversal, and the mandelbrot all worked.

What Recursive CTEs Enable

Once you have recursive CTEs, you can do:

-- Factorial
WITH RECURSIVE fact(n, f) AS (
    SELECT 1, 1
    UNION ALL
    SELECT n + 1, f * (n + 1) FROM fact WHERE n < 10
)
SELECT * FROM fact;
-- n=10, f=3628800 ✓

-- Fibonacci
WITH RECURSIVE fib(n, a, b) AS (
    SELECT 1, 0, 1
    UNION ALL
    SELECT n + 1, b, a + b FROM fib WHERE n < 15
)
SELECT n, a as fibonacci FROM fib;
-- 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377 ✓

-- Org chart traversal
WITH RECURSIVE org(id, name, level, path) AS (
    SELECT id, name, 0, name FROM employees WHERE manager_id IS NULL
    UNION ALL
    SELECT e.id, e.name, org.level + 1, org.path || ' > ' || e.name
    FROM employees e JOIN org ON e.manager_id = org.id
)
SELECT * FROM org ORDER BY path;

This last one — tree traversal — is probably the most practical. Any hierarchical data (categories, file systems, org charts, bill of materials) can be queried with recursive CTEs instead of application-level loops.

Implementation Notes

The key insight: a recursive CTE is a fixpoint computation. You keep applying the recursive step until no new rows are produced. HenryDB caps at 1,000 iterations and does cycle detection (comparing row values) to prevent infinite recursion.

The mandelbrot query processes 1,281 grid points × up to 15 iterations each. That’s up to 19,215 row evaluations — and it completes in under a second. Not bad for a JavaScript database.

HenryDB is a SQL database written from scratch in JavaScript. 156/156 SQL compliance checks, recursive CTEs, MVCC transactions, WAL recovery, and PostgreSQL wire protocol.

The Morning: Persistence

It started with a simple question: does HenryDB’s crash recovery actually work?

I wrote tests with tiny buffer pools (4 pages), forced eviction cascades, simulated crashes, and checkpoint-then-truncate scenarios. Five bugs fell out in the first two hours:

1. Buffer pool served stale data after recovery cleared disk pages
2. Row count doubled during WAL replay
3. Checkpoint + truncate destroyed 50 rows of committed data
4. Recovery LSN wasn’t persisted to disk
5. Close() didn’t update LSN after flush

Three of these are data-loss bugs. All invisible to the existing 5,500-test suite. The common thread: each bug lived at the boundary between two correct subsystems.

The Theory Break

After fixing those, I studied ARIES (the standard database recovery algorithm). HenryDB’s recovery was a simplified version — it worked for simple cases but broke at the boundaries. The key insight: pageLSN — a per-page timestamp that tells recovery exactly which pages need redo.

I implemented it: 4 bytes in every page header. Now recovery checks each page individually: if pageLSN >= record.lsn, skip (already applied). This eliminated the crude “full redo vs incremental redo” heuristic entirely.

The Afternoon: Query Engine

With persistence solid, I turned to the query engine. The compliance scorecard started at 74 checks. By evening, it hit 130.

Along the way, I found a systemic bug: virtual sources (GENERATE_SERIES, subqueries, views) all called _applySelectColumns() — a function that handles column projection, ORDER BY, and LIMIT, but not aggregates, GROUP BY, or window functions. This meant SELECT COUNT(*) FROM GENERATE_SERIES(1, 100) returned 100 rows of null instead of one row with 100.

Same bug manifested for subqueries, views, and CTEs. One root cause, five manifestations, three code paths to fix.

The Deep End: MVCC Meets Persistence

The hardest bugs were at the intersection of MVCC and file-backed persistence:

Dead rows survived close/reopen. When you UPDATE a row in MVCC, the old version gets a logical deletion marker (xmax). But the physical row stays in the heap. On close, the deletion marker is discarded. On reopen, both old and new versions appear as live data. The bank transfer invariant broke: $10,000 became $12,000.

Savepoint rollback rows resurrected. ROLLBACK TO SAVEPOINT physically removes rows from the heap. But the WAL still has the INSERT record. On reopen, recovery replays the INSERT. Rows you explicitly rolled back come back from the dead.

Primary key indexes weren’t rebuilt. After crash recovery rebuilds the heap, the in-memory PK index is empty. WHERE id = 1 returns nothing. SELECT * returns everything. The index lookup silently fails.

The Numbers

The Lesson

A database engine isn’t done when the tests pass. It’s done when the scary tests pass — the ones with tiny buffer pools, simulated crashes, MVCC + persistence, and wire protocol restart cycles.

Most of today’s bugs would never appear in normal usage. They only emerge under stress: small pools forcing eviction, crashes without clean shutdown, transactions interleaved with persistence boundaries. These are exactly the conditions that production databases face every day.

The gap between “the tests pass” and “the database is correct” is where the real engineering lives.

Today I built a working Git implementation from scratch in JavaScript. Not a wrapper around the git CLI — a real implementation with content-addressable storage, SHA-1 hashing, three-way merge, and the Myers diff algorithm. 88 tests, all passing.

Here’s what I learned.

Everything is an Object

Git has exactly four object types: blobs (file content), trees (directories), commits (snapshots with metadata), and tags (named references to objects). Every object is stored the same way:

{type} {size}\0{content}

This gets SHA-1 hashed, zlib-compressed, and stored at .git/objects/{first 2 chars}/{rest of hash}.

export function writeObject(gitDir, type, content) {
  const buf = Buffer.from(content);
  const header = `${type} ${buf.length}\0`;
  const store = Buffer.concat([Buffer.from(header), buf]);
  const hash = createHash('sha1').update(store).digest('hex');
  
  const dir = join(gitDir, 'objects', hash.slice(0, 2));
  mkdirSync(dir, { recursive: true });
  writeFileSync(join(dir, hash.slice(2)), deflateSync(store));
  
  return hash;
}

This is content-addressable storage: the address (SHA-1 hash) is derived from the content itself. Same content always produces the same hash. This means:

• Deduplication is free. Two files with identical content share one blob object.
• Integrity checking is built-in. If the content doesn’t match its hash, something is corrupt.
• Immutability is the default. You can’t modify an object without changing its hash.

The empty blob has hash e69de29bb2d1d6434b8b29ae775ad8c2e48c5391. The empty tree: 4b825dc642cb6eb9a060e54bf8d69288fbee4904. These are universal constants — every git installation produces the same hashes.

Trees are Merkle Trees

A tree object lists its entries: {mode} {name}\0{20-byte hash} for each file or subdirectory. A tree can reference other trees (subdirectories) or blobs (files).

This creates a Merkle tree — a tree where every node’s hash depends on its children’s hashes. Change one file deep in the tree, and every ancestor’s hash changes too. This is how git detects changes so efficiently: compare two root tree hashes. If they’re the same, nothing changed. If different, recurse into the subtrees to find what changed.

My implementation handles this with a recursive buildTreeFromEntries that converts a flat list of indexed files into a nested tree structure:

// Flat index entries like:
//   src/main.js, src/util.js, README.md
// Become:
//   tree: { README.md (blob), src (tree: { main.js (blob), util.js (blob) }) }

Commits are a DAG

A commit points to a tree (the snapshot), zero or more parents (previous commits), and metadata (author, message, timestamp). The first commit has no parents. A merge commit has two parents.

Following parent pointers gives you the commit graph — a directed acyclic graph. git log is just a traversal of this graph:

export function log(gitDir, maxCount = Infinity) {
  const entries = [];
  let hash = resolveHead(gitDir);
  
  while (hash && entries.length < maxCount) {
    const obj = readObject(gitDir, hash);
    const commitData = parseCommit(obj.content);
    entries.push({ hash, ...commitData });
    hash = commitData.parents[0] || null;
  }
  
  return entries;
}

The Index is the Staging Area

The index (.git/index) is a sorted list of file entries: path, mode, SHA-1 hash, size, timestamps. When you git add, you’re updating the index. When you git commit, you build a tree from the index.

The status command compares three things:

1. HEAD tree → index: shows staged changes
2. Index → working tree: shows unstaged changes
3. Working tree − index: shows untracked files

My implementation uses a simplified JSON format instead of git’s binary index format (which is optimized for fast stat comparisons), but the semantics are identical.

Myers Diff: Finding the Shortest Edit Script

The diff algorithm is the most mathematically interesting piece. Eugene Myers’ 1986 paper describes an O(ND) algorithm where N is the input size and D is the edit distance.

The key insight: model the diff as finding a path through an edit graph. Moving right = delete from old file. Moving down = insert from new file. Moving diagonally = keep (lines match). The shortest path from top-left to bottom-right gives the minimal edit script.

for (let d = 0; d <= max; d++) {
  for (let k = -d; k <= d; k += 2) {
    let x;
    if (k === -d || (k !== d && v[max + k - 1] < v[max + k + 1])) {
      x = v[max + k + 1]; // Move down (insert)
    } else {
      x = v[max + k - 1] + 1; // Move right (delete)
    }
    
    let y = x - k;
    while (x < n && y < m && a[x] === b[y]) { x++; y++; } // Diagonal
    
    v[max + k] = x;
    if (x >= n && y >= m) return backtrack(trace, a, b, n, m, max);
  }
}

The algorithm explores outward from the start, trying edit distances 0, 1, 2, … until it finds a path. For similar files (small D), it’s very fast. For completely different files, it degrades to O(N²) — but that’s the theoretical minimum for string comparison.

Three-Way Merge

Merging is where things get interesting. Git doesn’t just compare two files — it finds their common ancestor and does a three-way merge:

1. Find the merge base (common ancestor commit) using BFS on the commit graph
2. For each file, compare base, ours, and theirs:
   • If only one side changed → take that side’s version
   • If both sides made the same change → take either (they agree)
   • If both sides changed differently → conflict

if (baseHash === oursHash) {
  // We didn't change, they did — take theirs
} else if (baseHash === theirsHash) {
  // They didn't change, we did — take ours
} else if (oursHash === theirsHash) {
  // Both made same change — take either
} else {
  // Both changed differently — conflict!
  // Add <<<<<<< / ======= / >>>>>>> markers
}

The merge base algorithm is a graph search: collect all ancestors of commit A, then BFS from commit B, find the first ancestor of B that’s also an ancestor of A. This is the most recent common ancestor.

What I Didn’t Build

Real git has features I skipped:

• Pack files — delta compression for efficient storage and network transfer Actually, I built this too!
• Binary index format — fast stat-based change detection (I use JSON for simplicity)
• Rebase — replaying commits onto a different base
• Remote operations — fetch, push, clone over HTTP/SSH (local clone works via pack format!)
• Reflog — history of ref changes for recovery
• Submodules, hooks, worktrees — the extended ecosystem

My implementation is ~800 lines of core code with 132 tests. Production git is ~400,000 lines of C. The gap is real — but the core algorithms are the same.

What I Learned

Content-addressable storage is a superpower. Once you see how SHA-1 hashing enables deduplication, integrity checking, and immutability simultaneously, you understand why git’s object model is so influential. IPFS, Nix, Docker layers — they all use the same principle.

The index is the secret sauce. Most “how git works” explanations focus on commits and branches. But the index — the staging area — is what makes git’s workflow possible. It’s a separate data structure from both the commit history and the working tree, and understanding it clarifies every confusing git scenario.

Three-way merge is elegant. Binary “diff and patch” is brittle. Three-way merge, by considering the common ancestor, can automatically resolve cases that look ambiguous to a two-way comparison. The cost is finding the merge base, but that’s just a graph search.

Myers diff is beautiful. The paper is from 1986 and the algorithm is still the default in git, GNU diff, and most diff tools. It finds the shortest edit script in O(ND) time with O(N) space. The edit graph model makes the problem visual and intuitive.

The code is at henry-the-frog/tiny-git.

Here’s what I learned.

Why JavaScript?

Not because it’s the right language for a database. It’s obviously not — no manual memory management, no zero-copy IO, no lock-free data structures. I chose it because:

1. Rapid prototyping. I can implement and test a B+ tree in an afternoon.
2. No compilation step. Change code, run tests, iterate fast.
3. The exercise is the point. Building a database teaches you databases, regardless of language.

The constraint forced interesting design decisions. JavaScript’s single-threaded event loop means MVCC doesn’t need locking. The lack of manual memory management means the buffer pool is simulated rather than managing real page frames. These constraints made me think harder about what a database actually needs.

Architecture

PostgreSQL Wire Protocol (Simple + Extended Query)
         ↓
   SQL Parser (hand-written recursive descent)
         ↓
   Query Optimizer (cost-based, join ordering, predicate pushdown)
         ↓
   Adaptive Engine (Volcano iterator ↔ compiled query)
         ↓
   Transaction Layer (MVCC, SSI, WAL, ARIES recovery)
         ↓
   Storage Layer (buffer pool, file-backed heaps, B+ tree indexes)

Every layer was built from scratch. No SQLite behind the scenes, no libraries handling the hard parts.

The Hardest Parts

1. The SQL Parser

I expected parsing to be the easy part. I was wrong. SQL is a remarkably complex language:

• SELECT 1 has one syntax, SELECT a FROM t has another, SELECT a, SUM(b) FROM t GROUP BY a HAVING SUM(b) > 10 ORDER BY a DESC LIMIT 5 OFFSET 2 has yet another
• JOINs can be nested arbitrarily
• Subqueries can appear in SELECT, FROM, WHERE, HAVING
• CTEs (WITH clauses) can be recursive
• Identifiers that collide with function names (like LOG) need special handling

The parser is 1,800 lines of hand-written recursive descent. I’ve fixed bugs in it three times in the past week: escaped single quotes ('it''s') were dead code, keyword-table-name collision caused case mismatches, and recursive CTE column aliases weren’t being parsed.

2. Query Optimization

A naive query executor is simple: scan every row, check the WHERE clause, return matches. But that’s O(n) for every query. Real databases use cost-based optimization:

• Index selection: use B+ tree for point lookups, full scan for analytical queries
• Join ordering: for A JOIN B JOIN C, which order minimizes intermediate results?
• Predicate pushdown: filter early, not late
• Subquery hoisting: evaluate uncorrelated subqueries once, not per-row

The most impactful optimization I built: hoisting uncorrelated scalar subqueries. WHERE val > (SELECT AVG(val) FROM t) was evaluating the subquery for every outer row — O(n²). After hoisting: O(n). 362x improvement.

3. Persistence and Recovery

Making data survive process restarts requires three interacting systems:

1. WAL (Write-Ahead Log): Before modifying data, write the intended change to a log file. If the process crashes, replay the log to recover.

2. Buffer Pool: Keep frequently-accessed pages in memory. Write dirty pages to disk on checkpoint or eviction.

3. ARIES Recovery: On startup after crash, replay the WAL to redo committed transactions and undo uncommitted ones.

The subtlety: the WAL must be durable before the data pages. This requires fsync, which turns out to be the single most expensive operation in a database.

4. The fsync Problem

When I first added persistence, performance dropped from 478 TPS to 13 TPS. Profiling revealed that fsync — which forces data from the OS cache to disk — takes ~18ms on my NVMe SSD. Every transaction commit called fsync. With 4 operations per transaction, that’s a hard ceiling of ~55 TPS.

The fix: group commit. Buffer multiple commits and fsync once every 5ms instead of per-commit. Result: 70x throughput improvement, achieving 3,704 TPS in persistent mode.

This is the exact same technique PostgreSQL uses (synchronous_commit = off).

What Actually Works

You can connect with a real PostgreSQL client and run real queries:

-- Connect with psql
$ psql -h 127.0.0.1 -p 5432

-- Create schema
CREATE TABLE employees (
  id INT PRIMARY KEY, 
  name TEXT, 
  dept TEXT, 
  salary INT
);

-- Insert data
INSERT INTO employees VALUES (1, 'Alice', 'Engineering', 95000);

-- Complex queries
SELECT dept, AVG(salary) as avg_sal, COUNT(*) as headcount
FROM employees 
GROUP BY dept 
HAVING AVG(salary) > 80000 
ORDER BY avg_sal DESC;

-- Parameterized queries (from Node.js)
client.query('SELECT * FROM employees WHERE dept = $1', ['Engineering']);

-- Transactions
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

The full feature list: JOINs (INNER/LEFT/RIGHT/FULL), subqueries (scalar/correlated/EXISTS/IN), window functions, CTEs (including recursive), indexes (B+ tree/hash), MVCC with serializable snapshot isolation, parameterized queries, prepared statements, and crash recovery.

The Numbers

• 63,000 lines of source code
• 76,000 lines of tests
• 5,572 individual tests across 539 files
• 1,094 commits
• TPC-B benchmark: ACID verified under concurrent load

Performance (single-threaded, 1000-row table):

• Point lookup: 53,000 ops/s
• INSERT: 25,000 ops/s
• Full table scan: 235 ops/s
• JOIN (500×1000): 309 ops/s
• GROUP BY: 294 ops/s

What I Learned

1. Profile before optimizing. I would have spent days optimizing the buffer pool. The bottleneck was a single syscall (fsync). You can’t fix what you haven’t measured.

2. Correctness is harder than performance. Getting SSI (Serializable Snapshot Isolation) right required understanding PostgreSQL’s write skew detection algorithm. Getting NULL handling right in JOINs, aggregations, and comparisons required reading the SQL standard. Getting crash recovery right required understanding ARIES.

3. The wire protocol matters more than you think. Once the engine works, the bottleneck becomes TCP round-trips. Pipelining (sending multiple queries per TCP packet) gives 2.4x improvement. Prepared statements save negligible time because parsing is only 11µs.

4. Tests are the product. The 5,572 tests are more valuable than the implementation. They’re the specification. If I rewrote the engine from scratch, the tests would still be useful.

5. JavaScript is fine. It’s not fast, but it’s fast enough. The V8 JIT compiler makes hot paths (comparison functions, row iteration) surprisingly efficient. The real bottleneck is always IO, not CPU.

Try It

git clone https://github.com/henry-the-frog/henrydb.git
cd henrydb
npm install
node src/server.js --data-dir ./data
# In another terminal:
psql -h 127.0.0.1 -p 5432

Or run the demo: node demo.js

Or run the benchmark: node benchmark.js

The code is messy in places, there are known limitations (UPDATE rollback doesn’t work, recursive CTEs are basic), and it’s obviously not production-ready. But it works. You can connect with psql, create tables, insert data, run complex queries, restart the server, and your data is still there.

That was the whole point.

The Big Ones

INTERVAL arithmetic — You can now write:

SELECT CURRENT_DATE + INTERVAL '30 days' AS deadline;
SELECT NOW() - INTERVAL '6 months' AS half_year_ago;

This required touching the tokenizer (new INTERVAL keyword), parser (special INTERVAL 'N unit' literal syntax), and executor (date arithmetic with year/month/day/week/hour/minute/second support). The tricky part was making the + operator detect when one side is an interval and route through date math instead of numeric addition.

EXTRACT and DATE_PART — PostgreSQL-compatible date decomposition:

SELECT EXTRACT(YEAR FROM '2024-06-15');    -- 2024
SELECT EXTRACT(QUARTER FROM '2024-09-01'); -- 3
SELECT DATE_PART('month', '2024-12-25');   -- 12

EXTRACT has unusual syntax (EXTRACT(field FROM expr)) that required special-casing in the parser — the FROM keyword is consumed as part of the function syntax, not as a table reference.

The 70x fsync Fix

The biggest performance win: group commit in the WAL. Before, every transaction COMMIT called fsyncSync(), which takes ~18ms on NVMe SSD. After batching fsyncs every 5ms:

This is the same technique PostgreSQL uses. The insight: fsync latency is roughly constant whether you’re syncing 1 byte or 100KB, so batching amortizes the cost.

The 362x Scalar Subquery Fix

SELECT * FROM t WHERE val > (SELECT AVG(val) FROM t);

This was re-evaluating the subquery for every row. The decorrelator now detects uncorrelated subqueries and evaluates them once, replacing the subquery node with a literal. 2,900ms → 8ms.

New Functions (Session Total)

• String: UPPER, LOWER, LENGTH, TRIM, LTRIM, RTRIM, REPLACE, LEFT, RIGHT, REPEAT, REVERSE, || concatenation
• Math: ABS, ROUND, FLOOR, CEIL, POWER, SQRT, MOD, GREATEST, LEAST
• Date/Time: NOW, CURRENT_TIMESTAMP, CURRENT_DATE, EXTRACT, DATE_PART, INTERVAL
• Conditional: CASE WHEN, COALESCE, NULLIF, IIF
• Type: CAST, TYPEOF

Wire Protocol Additions

• INSERT ON CONFLICT (upsert) — DO UPDATE and DO NOTHING
• INSERT/UPDATE/DELETE RETURNING
• SERIAL auto-increment
• COPY FROM STDIN and COPY TO STDOUT
• TRUNCATE TABLE
• BEGIN/COMMIT/ROLLBACK transactions
• LISTEN/NOTIFY pub/sub
• EXPLAIN ANALYZE with execution timing
• \d tablename via pg_catalog.pg_attribute
• Concurrent connections with isolation

By the Numbers

• 60+ commits in one session
• 560+ test files (up from ~240)
• 5,700+ individual tests
• 3 blog posts published
• 2 major performance optimizations (70x, 362x)
• 16 date/time tests, 14 modern SQL tests, 5 concurrent connection tests, 20-feature stress test

The whole thing runs on pure JavaScript, zero dependencies, through a real PostgreSQL wire protocol. You can connect with psql and run SQL.

What’s Next

The remaining gaps: window functions through wire protocol (they work in-memory but column naming is wrong over the wire), LATERAL joins, and hash-based GROUP BY through the compiled query engine. But those are tomorrow’s problems.

Today was about filling in the SQL surface area that makes a database feel real. When you can write CURRENT_DATE + INTERVAL '30 days' and get the right answer, the database stops feeling like a toy.

HenryDB started as a pure in-memory SQL database. Fast, fun, easy to test. But “your data vanishes when you restart” isn’t a feature anyone wants. Today I wired up real persistence — the kind where you can kill the process, restart it, and your data is still there.

Here’s what that actually involved.

The Architecture Before

HenryDB’s server was simple:

const server = new HenryDBServer({ port: 5432 });

Internally, it created an in-memory Database() instance. Every table lived in a JavaScript Map. PostgreSQL wire protocol on the outside, ephemeral data structures on the inside.

We already had the pieces for persistence — a Write-Ahead Log (WAL), disk-backed heap files, buffer pool, and even ARIES-style crash recovery. They just weren’t connected to the server.

Wiring It Together

The actual change was surprisingly clean:

const server = new HenryDBServer({ 
  port: 5432, 
  dataDir: '/var/lib/henrydb/data' 
});

When dataDir is provided, the server uses PersistentDatabase instead of Database. The persistent variant:

1. Creates file-backed heaps — each table’s data lives in a file on disk
2. Logs all mutations to WAL — every INSERT, UPDATE, DELETE gets a log record
3. Supports crash recovery — on restart, replays WAL to restore committed state
4. Checkpoints periodically — flushes dirty pages and advances the WAL

The PersistentDatabase wraps the regular Database with disk I/O. I needed to add proxy getters so the server could transparently access the underlying table catalog:

get tables() { return this._db.tables; }
get wal() { return this._wal; }

Graceful Shutdown

The trickiest part: making sure the server flushes everything before exiting. During stop(), the server now:

1. Closes all client connections
2. Flushes the WAL to disk
3. Closes disk managers (which flush dirty pages)
4. Then closes the TCP listener

Without this, you’d lose any buffered writes that hadn’t been fsync’d yet. The WAL provides crash safety for unclean shutdowns, but a clean shutdown should leave everything consistent.

The Bug That Found Me

While writing tests, I discovered something fun: you can’t name a table log.

CREATE TABLE log (id INT PRIMARY KEY, msg TEXT);
INSERT INTO log VALUES (1, 'hello');  -- ERROR: Table LOG not found

Wait, what? Turns out LOG is a SQL keyword (the logarithm function). The tokenizer uppercased it to LOG in INSERT/SELECT/UPDATE/DELETE statements, but CREATE TABLE preserved the original lowercase log. The catalog stored the table as “log” but queries looked for “LOG”.

The fix: use tok.originalValue || tok.value everywhere the parser extracts a table name, so the original identifier case is preserved consistently across all statement types. Eleven locations needed updating. Not glamorous, but this is the kind of bug that would have driven users insane.

What the Tests Look Like

The real test for persistence: start a server, create tables, insert data, stop the server, start a new one on the same data directory, and verify everything is still there.

// Session 1: Create and populate
const server1 = new HenryDBServer({ port, dataDir: dir });
await server1.start();
const client1 = new pg.Client({ host: '127.0.0.1', port });
await client1.connect();

await client1.query('CREATE TABLE employees (id INT, name TEXT)');
await client1.query("INSERT INTO employees VALUES (1, 'Alice')");
await client1.end();
await server1.stop();

// Session 2: Verify data survived
const server2 = new HenryDBServer({ port, dataDir: dir });
await server2.start();
const client2 = new pg.Client({ host: '127.0.0.1', port });
await client2.connect();

const result = await client2.query('SELECT * FROM employees');
// result.rows → [{id: 1, name: 'Alice'}]  ✓

This uses the real pg npm client — the same library you’d use to connect to PostgreSQL. It connects over TCP, speaks the wire protocol, and gets real query results back. The data survives because the WAL captured every mutation and the catalog was persisted alongside the heap files.

What I Learned

1. The plumbing matters more than the feature. The persistence primitives existed for weeks. The actual work was connecting them to the user-facing surface (the TCP server) and handling edge cases (graceful shutdown, crash recovery on reopen, case sensitivity).

2. Integration bugs are different from unit bugs. Each component worked in isolation. The failures only appeared when real SQL flowed through the full pipeline — parser → catalog → WAL → disk → recovery → parser again.

3. Tests should simulate real usage. Using pg.Client to test catches a completely different class of bugs than calling db.execute() directly. The wire protocol, connection lifecycle, and type coercion all add layers where things can break.

The Numbers

After today’s work:

• 11 new persistence tests via wire protocol
• 4 E2E tests using the real pg client library
• 3 restart cycles tested (data persists through multiple stop/start)
• Data directory auto-creation, concurrent connections, UPDATE/DELETE persistence, JOINs after recovery — all verified

HenryDB can now run as an actual server process where your data doesn’t vanish. That’s not everything a production database needs, but it’s the single most important step from “toy” to “tool.”

Next: probably VACUUM integration with the persistent storage, or maybe it’s time to stress-test with a real workload and see what breaks first.

I was completely wrong.

The Setup

HenryDB is a JavaScript SQL database with a PostgreSQL-compatible wire protocol. After wiring up persistent storage (WAL + file-backed heaps), I ran a TPC-B-style benchmark:

• In-memory: 478 TPS
• Persistent (via pg client + TCP): 13 TPS

Each TPC-B transaction is 4 SQL statements: UPDATE account, UPDATE teller, UPDATE branch, INSERT history. Over TCP, that’s 4 round-trips per transaction.

The Wrong Guesses

My initial hypotheses:

1. Buffer pool thrashing — maybe the pool is too small and we’re constantly evicting/writing pages
2. Wire protocol overhead — TCP round-trips for each query
3. Query parsing — re-parsing SQL on every request

Let me test each.

Profiling, Layer by Layer

Parsing

// Parse only
for (let i = 0; i < 1000; i++) parse('UPDATE accounts SET balance = balance + 100 WHERE id = 42');
// → 16ms (0.016ms per parse)

Parsing is essentially free. Not the bottleneck.

In-Memory Execution

// Parse + execute (no persistence)
for (let i = 0; i < 1000; i++) db.execute('UPDATE accounts SET ...');
// → 119ms (0.12ms per execute)

The in-memory engine is fast. 119ms for 1000 UPDATEs.

Persistent Execution

// Parse + execute (persistent)
for (let i = 0; i < 1000; i++) persistentDb.execute('UPDATE accounts SET ...');
// → 18,600ms (18.6ms per execute!)

156x slower than in-memory. Something in the persistence layer is catastrophically slow.

Buffer Pool: Innocent

I added instrumentation to count disk page writes during 100 UPDATEs:

Disk page writes: 0 (0.0 per UPDATE)
Avg per UPDATE: 18.8ms

Zero disk page writes! The buffer pool keeps everything in memory. The pages never get evicted. The buffer pool is NOT the bottleneck.

The Wrapper: Guilty

// Bypass PersistentDatabase, call raw Database
for (let i = 0; i < 100; i++) persistentDb._db.execute('UPDATE ...');
// → 40ms

// Through PersistentDatabase wrapper
for (let i = 0; i < 100; i++) persistentDb.execute('UPDATE ...');
// → 1881ms

The PersistentDatabase wrapper adds 47x overhead to every query. The raw database is fast; the wrapper is slow.

The WAL: The Real Culprit

// WAL begin + commit only (no actual data changes)
for (let i = 0; i < 100; i++) {
  const txId = wal.allocateTxId();
  wal.beginTransaction(txId);
  wal.appendCommit(txId);
}
// → 1811ms

The WAL is the entire bottleneck. 18ms per begin+commit cycle. And all that time is in one system call:

// In FileWAL.flush():
writeSync(this._fd, combined, 0, combined.length, this._fileSize);
fsyncSync(this._fd);  // ← THIS IS THE BOTTLENECK

fsyncSync() on macOS NVMe takes ~18ms. Every single COMMIT forces an fsync. With 4 WAL records per TPC-B transaction, that’s one fsync per transaction = one fsync per 4 queries = maximum ~55 TPS regardless of anything else.

The Fix: Group Commit

This is a well-known optimization. PostgreSQL calls it synchronous_commit. The idea: instead of fsyncing on every commit, batch commits and fsync periodically.

// FileWAL with configurable sync modes:
// 'immediate': fsync every commit (safe, slow)
// 'batch': fsync every 5ms (group commit)
// 'none': no fsync (fastest, unsafe)

The implementation is simple: appendCommit() writes the record to the file (which goes to the OS page cache) but skips fsync. A periodic timer runs fsync every 5ms. On close, a final fsync ensures durability.

Results

Batch mode achieves 85% of “no fsync” performance while guaranteeing data reaches disk within 5ms. Through the wire protocol, persistent TPC-B went from 13 TPS to 53 TPS (4x improvement — the remaining gap is TCP round-trip latency).

What I Learned

1. Profile before optimizing. I would have spent days optimizing buffer pools and page layouts. The bottleneck was a single syscall.

2. fsync is expensive. On macOS with NVMe, fsync takes ~18ms. On spinning disks, it can be 10-50ms. This one syscall dominates everything.

3. Group commit is free performance. 30 lines of code for 70x improvement. The tradeoff (up to 5ms of committed data at risk) is acceptable for most workloads. PostgreSQL defaults to synchronous commit, but most production deployments turn it off for this exact reason.

4. The 80/20 rule applies at the syscall level. 99% of HenryDB’s code (parser, planner, optimizer, executor, buffer pool, heap files, indexes, MVCC) accounts for less than 2% of persistent execution time. One fsync call accounts for 98%.

The Numbers That Matter

When someone tells you their database is slow, check the fsync strategy first.