15 September 2026

53 new posts across 7 database and data engineering sources — 9 worth your time.

One week of Databases & Data, 9 stories, as published.

118 million queries per second on Neki

Details a massive scalability benchmark of PlanetScale's Neki, scaling to 118 million queries per second across 512 shards with 1.22 PiB of data. The test demonstrates linear scalability under extreme load.

Why it matters — Demonstrates that horizontal sharding can achieve linear performance scaling up to hundreds of millions of QPS and petabyte-scale datasets.

PlanetScale · 15 September 2026 · Read the original →

Streaming & CDC

Shaun Thomas: PG Phriday: The Folder That Ate the Publisher

Explains how logical replication in PostgreSQL can silently consume massive storage resources on the publisher when subscriptions are created without explicit clauses. It details how replication slots manage storage and the risks of disk exhaustion.

Planet PostgreSQL · 15 September 2026 · Read the original →

Practice

Umair Shahid: How to De-Risk a Mission-Critical PostgreSQL Migration

Outlines a structured methodology for executing high-stakes PostgreSQL migrations, prioritizing metrics like Recovery Time Objective (RTO) over speed. It guides operators on choosing between dump/restore and logical replication based on downtime budgets, emphasizing validation with checksums.

Planet PostgreSQL · 15 September 2026 · Read the original →

Streaming & CDC

Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

Introduces WalShadow, a tool that replicates PostgreSQL data directly into ClickHouse by reading physical WAL files. Benchmarks show it achieves 200 ms latency and processes 289,000 rows per second.

Why it matters — Demonstrates that physical WAL parsing can bypass the overhead of logical decoding for sub-second, high-throughput CDC pipelines.

ClickHouse · 15 September 2026 · Read the original →

Postgres

Radim Marek: The unbearable lightness of one more index

Discusses the operational overhead of index bloat and the process of auditing and dropping redundant indexes on a single table. It highlights the shift in database design reviews from avoiding databases to managing schema complexity.

Why it matters — Reminds engineers that every additional index incurs a write penalty and maintenance overhead, making regular index audits crucial.

Planet PostgreSQL · 15 September 2026 · Read the original →

Storage Internals

Improving Lakebase Postgres Compute Cache on Neon, Part 1

Explains how Neon optimized its Postgres compute cache by allocating most of the machine's memory to shared buffers and backing them with huge pages. This prevents hot pages from falling back to local disk cache, reducing CPU usage and latency.

Why it matters — Backing shared buffers with huge pages keeps the working set in DRAM, avoiding the CPU and latency overhead of disk-cache lookups.

Neon · 15 September 2026 · Read the original →

Analytics & OLAP

Measuring real-time performance per dollar under continuous load: CostBench’s first end-to-end results

Presents benchmark results from CostBench, which evaluates cloud data warehouses under continuous load. The results show ClickHouse Cloud delivering 412x to 1,996x better performance per dollar compared to Snowflake.

Why it matters — Provides a methodology for evaluating database costs under continuous, real-time query and ingestion loads rather than idle states.

ClickHouse · 15 September 2026 · Read the original →

Data Engineering

Mandiant Feeds Deprecated, Migration Required

Google Cloud is deprecating the MANDIANT_ACTIVE_BREACH_IOC, MANDIANT_FUSION_IOC, and OPEN_SOURCE_INTEL_IOC feeds. Users are required to migrate to the GTI_IOC feed, as legacy feeds will be removed after March 18, 2027.

Why it matters — Data engineers must migrate their security intelligence feed integrations to GTI_IOC by March 18, 2027, or lose access to critical threat intelligence.

docs.cloud.google.com · 15 September 2026 · Read the original →

Performance

Elasticsearch Cuts LLM Agent Costs 75%

Elasticsearch Labs demonstrated improving LLM agent accuracy from 60% to 92% and reducing input tokens by up to 75%. This was achieved through pre-computed context and iterative feedback loops from agent failures.

Why it matters — Teams can significantly reduce LLM agent operational costs and improve accuracy by implementing iterative feedback loops and optimized retrieval for pre-computed context.

elastic.co · 15 September 2026 · Read the original →

Another week of trying to index our way out of a migration.

9 stories, every Tuesday

Published here every week. Follow by RSS to get it as it lands.

Subscribe by RSS All issues

← Previous issue Next issue →