Verification of the `sqlite-optimized` Repository

Independent rebuild-and-reproduce audit · 8 August 2026 · repository github.com/ksenxx/sqlite-optimized (commit 4b19d6bcf on upstream SQLite 3.54.0 trunk d0afe9c60), cloned to ~/sqllite-optimized

Verdict. The repository builds cleanly from scratch, no code or wiring is missing, the upstream regression suite is green under the documented test protocol, and the published benchmark speedups reproduce closely: 1.50× geometric mean using the exact documented reproduce steps, 1.55× after adding a fresh profile-guided-optimization (PGO) build, versus the published 1.59× (a 2.5% shortfall consistent with machine drift and PGO-training differences). Every anti-cheating gate — result checksums, speedtest1 --verify, kvtest --integrity-check — passed on every repetition of every configuration. An independent read-only review by a second model (gpt-5.6-sol) recomputed all numbers, confirmed the build fairness, and found no engine bug, no cheating, and no verifier-introduced changes.

1. What was verified, and how

Fresh clone. git clone of the public repository into ~/sqllite-optimized; integrity checked (git fsck clean, HEAD as published).
Build from scratch. configure + make sqlite3.c regenerated the 9.5 MB amalgamation; the four benchmark binaries (speedtest1, kvtest, tatp, ssb) were built twice from the same amalgamation and sources: a baseline build (-O2 -g, stock defaults) and the optimized build (-O3 -march=native + the repository’s OPT_DEFS stack) — exactly the reproduce commands published with the repository.
Benchmarks. The repository’s fixed protocol benchks/bench.sh (3 repetitions, hard-coded expected checksums, fail-fast) was run for baseline, optimized, and a freshly retrained PGO build whose training workloads deliberately used different sizes and seeds than the measured runs.
Tests. The upstream veryquick suite (~332,000 checks) was run against four differently-configured testfixture builds to separate code effects from changed-default effects.
Independent review. A second model, gpt-5.6-sol, performed a strictly read-only audit of every claim, log, and result file (≈0.03% of task budget, well under the 20% cap).

2. Benchmark reproducibility

Medians of 3 repetitions; smaller is faster. “Published” figures are from the original optimization report (its v4 data).

Benchmark	Baseline (today)	Optimized, documented steps	Speedup	Optimized + fresh PGO	Speedup	Published speedup
speedtest1 (official, `--verify`)	11.54 s	5.51 s	2.09×	5.86 s	1.97×	2.06×
TATP transaction mix (academic OLTP)	4.30 s	2.37 s	1.82×	2.15 s	2.00×	1.90×
kvtest blob I/O (official, 3 phases)	2.95 s	2.31 s	1.28×	2.31 s	1.28×	1.25×
SSB 13 queries (academic OLAP)	2.85 s	2.76 s	1.03×	2.48 s	1.15×	1.30×
Geometric mean			1.50×		1.55×	1.59×

Published (with original PGO)

1.59×

Reproduced, fresh PGO

1.55×

Reproduced, documented steps

1.50×

Geometric-mean speedup over the four benchmarks: published versus the two independent reproductions.

Why not exactly 1.59×?

Machine drift. The same physical binaries from the original work, re-run today, give baseline speedtest1 11.77 s (published 10.36 s) and optimized 5.13 s (published 5.02 s) — the machine is measurably slower on this path today, yet the ratio still holds at 2.29×. The original optimized SSB binary still runs its queries in 2.30 s today.
PGO training quality. The published reproduce commands omit the PGO step, which matters most for CPU-bound SSB. A first light PGO retrain gave SSB 1.10×, a heavier one 1.15×; the original, more extensively trained binary achieves 1.24× today. All other benchmarks reproduce at or above their published speedups in at least one configuration.
Per the independent reviewer, the honest statement is: “closely but not exactly reproduced (1.55× vs 1.59×), with plausible run-to-run/machine and PGO-profile sensitivity; the exact residual cause was not isolated.” The largest single deviation is SSB (−12% vs published).

Anti-cheating gates (all passed, every repetition)

speedtest1 --verify internal result verification.
kvtest --integrity-check reported ok after all 9 runs.
TATP transaction counts 139981 39920 140288 7955 55905 8107 7844 and checksum 10123417494647775989, and SSB row count 3394 / checksum 13164251427055023644, matched the hard-coded expected values on every repetition of every build — identical answers from baseline and optimized builds.
The reviewer confirmed via SHA-256 that all builds compiled the identical amalgamation, that the committed benchmark scripts and expected checksums were not altered, and via DWARF metadata that each binary was compiled with exactly the claimed flags.

3. Test-suite results

testfixture configuration	veryquick result	Interpretation
Stock defaults (baseline)	1 error / 332,420 — only `zipfile-25.0`	Green. The one failure is a proven environmental artifact: a directory left behind by `symlink.test`; `zipfile.test` alone in a clean directory passes 0/120.
Code changes only (computed-goto + WAL write buffer)	1 error / 332,420 — only `zipfile-25.0`	Green. The engine code changes introduce no regression.
All tuned defaults except WAL-by-default	26 errors / 332,796, run completed	Every failure is in a test that asserts a specific default value the build intentionally changes (WAL autocheckpoint 1000→16384, cache size, temp store, mmap). All 26 files pass in the code-only build.
Full `OPT_DEFS` incl. WAL-by-default	Fails; aborted mid-suite	Expected: the upstream suite assumes rollback-journal defaults. Bisection proved the pattern: a WAL-default-only build reproduces the `attach2` (26/71) and `autovacuum` (23/339) failures exactly, while the stock build passes them 0 errors.

Caveat carried over from the original work, now made explicit. The claim “full test suite green” applies to the source-code changes under default build options — which is exactly how the repository’s own audit documents (ADVERSARIAL.md, HARDENING.md) say testing was done, and which this verification reproduced. The upstream suite does not support builds whose runtime defaults are changed (WAL-by-default, autocheckpoint, cache/mmap/temp-store); those failures were bisected to the changed defaults, not to code. Two related reviewer notes: (a) two bisected files plus the completed defaults-minus-WAL run make the attribution strong but not exhaustive for every failing case; (b) testfixture builds define SQLITE_TEST, which by design disables computed-goto dispatch, so the suite exercises the fallback dispatch path (the benchmark binaries do exercise computed-goto, under checksum verification).

4. Independent read-only review (gpt-5.6-sol)

Recomputed every median, speedup, and geomean from the raw result files: no arithmetic errors (1.4961× documented steps; 1.5499× PGO round 2).
Verified clone integrity, build fairness (same sources, same feature defines, only OPT_DEFS/CFLAGS differ), and that the optimized defaults were genuinely active (fresh databases born in WAL mode; baseline stays on rollback journal).
Independently re-ran the zipfile and WAL-default bisections and got identical results.
Confirmed no tracked file was modified in either repository during verification.
PGO anti-cheating check passed: training workloads differ from measured workloads in size and seed for all four benchmarks; final binaries carry -fprofile-use and no instrumentation.
Findings were limited to reporting-precision qualifications (incorporated above); no invented problems.

5. Reproduce it yourself

git clone https://github.com/ksenxx/sqlite-optimized ~/sqllite-optimized
cd ~/sqllite-optimized && mkdir build-base build-opt
( cd build-base && ../configure && make sqlite3.c )
cp build-base/sqlite3.{c,h} build-opt/
benchks/build_bench.sh build-base                    # baseline: -O2, stock defaults
. benchks/optflags.sh
CFLAGS="-O3 -march=native -g" benchks/build_bench.sh build-opt $OPT_DEFS
benchks/bench.sh build-base baseline 3               # results: benchks/results/baseline.txt
benchks/bench.sh build-opt  final    3               # results: benchks/results/final.txt
( cd build-base && make testfixture && ./testfixture ../test/veryquick.test )

Raw artifacts from this verification remain in ~/sqllite-optimized/benchks/results/ (baseline-repro.txt, final-repro.txt, final-pgo-repro.txt, final-pgo2-repro.txt) and the build directories build-base, build-opt, build-pgo, build-code, build-waldef, build-tunes.

Verification of the sqlite-optimized Repository