Newsdesk
Engineering
Data & AI
Industries
Enterprise Systems
Go-to-Market
Longform
DesignIndiaAll stories

Databases & Data

03 November 2026

74 new posts across 9 database and data engineering sources — 8 worth your time.

One week of Databases & Data, 8 stories, as published.

Postgres

Vadim Yatsenko: Zero-Copy I/O for the Shared Storage Daemon: Removing the Last Copy from the WAL Path

In PolarDB, every database commit requires the Write-Ahead Log (WAL) record to reach shared storage, involving data movement from WAL buffers through a shared-storage daemon to the device. This article details how the last data copy in this critical WAL path was eliminated to improve performance.

Why it matters — Optimizing the WAL path by eliminating unnecessary data copies, such as through zero-copy I/O, is a critical technique for enhancing durability and performance in high-throughput database systems.

Planet PostgreSQL · 03 November 2026 · Read the original →

Postgres

Vadim Yatsenko: Commit Sequence Number (CSN) MVCC: Lock-Free Snapshot Acquisition

PostgreSQL's Multi-Version Concurrency Control (MVCC) determines row visibility by checking if a writing transaction was committed before a query's snapshot. This article describes Commit Sequence Number (CSN) MVCC, a technique for achieving lock-free snapshot acquisition, improving concurrency.

Why it matters — Implementing Commit Sequence Number (CSN) MVCC enables lock-free snapshot acquisition, a crucial technique for improving concurrency and performance in highly transactional database systems.

Planet PostgreSQL · 03 November 2026 · Read the original →

Postgres

Igor Suhorukov: pg_vexec one month on: ahead of ClickHouse on Parquet, close behind on MergeTree

This project ported Greenplum's ORCA analytical query planner to PostgreSQL 19, isolating minimal core patches to create a vectorized, high-performance DBMS. After one month, this database fork shows faster performance, outperforming ClickHouse on Parquet and nearing its MergeTree performance.

Why it matters — Vectorized query execution and advanced query planners, even when backported, can significantly boost PostgreSQL's analytical performance, challenging specialized OLAP databases.

Planet PostgreSQL · 03 November 2026 · Read the original →

Postgres

Designing a planned switchover for minimum disruption

This post details the process of designing a planned switchover for a Neki shard, which consists of a Postgres primary and multiple replicas. The goal is to safely promote a replica to primary with minimum disruption, covering scenarios like pod replacement, kernel upgrades, or moving the primary.

Why it matters — Planned switchovers for database primaries require careful design to ensure minimal disruption during maintenance or upgrades, contrasting with unplanned failovers.

PlanetScale · 03 November 2026 · Read the original →

Data Engineering

How Virgin Media O2 cut compute costs 30% by giving dbt a memory

Virgin Media O2 achieved significant cost and performance improvements by adopting dbt v2 and dbt State at scale. This resulted in a 30% reduction in dbt compute costs and a 35% decrease in model run times.

Why it matters — Leveraging dbt's stateful features and newer versions can lead to substantial cost savings (30% compute) and performance gains (35% faster model runs) in large-scale data transformation pipelines.

dbt · 03 November 2026 · Read the original →

Data Engineering

Why an AI agent's revenue query was $3.5 million wrong, and how AWS fixed it with dbt Semantic Layer

An AI agent generated a plausible revenue query that was incorrect by $3.5 million. AWS addressed this by using the dbt Semantic Layer to provide essential business context, preventing such errors in AI-driven analytics.

Why it matters — AI agents require a well-governed semantic layer to provide accurate business context, preventing significant financial errors ($3.5M) in their generated queries and analyses.

dbt · 03 November 2026 · Read the original →

Analytics & OLAP

How OpenRouter observes, understands, and acts on 740 million LLM requests a day with ClickHouse Cloud

OpenRouter scaled its customer analytics, model rankings, internal AI agents, and routing across 500+ models using ClickHouse Cloud, handling 740 million LLM requests daily and growing monthly requests 33x to 19.7 billion. It serves as their single store for these diverse workloads.

Why it matters — A single OLAP store can handle diverse, high-volume analytics and AI agent workloads, scaling to billions of requests with significant growth.

ClickHouse · 03 November 2026 · Read the original →

Postgres

pg_vexec Ahead of ClickHouse on Parquet

Igor Suhorukov's experimental `pg_vexec` extension for PostgreSQL, which adds vectorized execution, outperformed ClickHouse on Parquet data by 13x and performed closely on MergeTree data in recent benchmarks. This demonstrates PostgreSQL's growing analytical capabilities.

Why it matters — PostgreSQL with vectorized execution now offers competitive analytical query performance against specialized OLAP databases for hybrid workloads.

postgresql.org · 03 November 2026 · Read the original →

Zero-copy WAL optimizations on one end, AI agents guessing the schema on the other.

8 stories, every Tuesday

Published here every week. Follow by RSS to get it as it lands.

← Previous issue Next issue →