Newsdesk
Engineering
Data & AI
Industries
Enterprise Systems
Go-to-Market
Longform
DesignIndiaAll stories

Databases & Data

08 August 2026

57 new posts across 9 database and data engineering sources — 18 worth your time.

One week of Databases & Data, 18 stories, as published.

Practice

On Benchmarking

This post explores the methodology of database and system benchmarking, arguing that a raw performance number is meaningless without understanding why a system stops scaling and whether the test matches production workloads. It warns against choosing architectures based solely on isolated high-throughput metrics.

Why it matters — A benchmark is only valid if you can identify its scaling limits and align its parameters with your actual production workload characteristics.

Data Engineering Weekly · 08 August 2026 · Read the original →

Performance

Avi Vallarapu: Tuning PostgreSQL HOT Updates - A HammerDB Benchmark

This post details how to leverage Heap-Only Tuple (HOT) updates in PostgreSQL to bypass the overhead of standard vacuuming. Using a HammerDB TPROC-C benchmark, it demonstrates how tuning the table fillfactor parameter reduces bloat growth and eliminates vacuum bottlenecks.

Why it matters — Tuning the table fillfactor parameter triggers Heap-Only Tuple (HOT) updates, which bypasses index updates and eliminates vacuum overhead during heavy write workloads.

Planet PostgreSQL · 08 August 2026 · Read the original →

Analytics & OLAP

How Mercado Libre rebuilt its observability platform on ClickHouse Cloud with 50x faster trace queries

This case study details how Mercado Libre migrated its observability platform to ClickHouse Cloud to handle an ingestion rate of 400 million spans per minute. The migration achieved up to 89% data compression and reduced trace query latencies from over five minutes to just four seconds.

Why it matters — Migrating high-throughput observability data to a columnar store achieved an 89% compression ratio and reduced trace query latency from five minutes to four seconds at a scale of 400 million spans per minute.

ClickHouse · 08 August 2026 · Read the original →

Reliability

Umair Shahid: PostgreSQL Lockup vs Stale Connection: How to Tell Them Apart

This post provides a guide to diagnosing database connection failures, specifically distinguishing between a genuine PostgreSQL server lockup and a stale connection that silently died in the application's connection pool. It highlights how identical-looking application errors can stem from entirely different network or pool failures.

Why it matters — Distinguishing between database-level lockups and silent connection pool dropouts prevents misdiagnosing network-level failures as database performance issues.

Planet PostgreSQL · 08 August 2026 · Read the original →

Postgres

Alexey Evlampiev: Your Transaction Boundary Belongs in the Program, Not the Filename

This article examines how to structure phased schema migrations in PostgreSQL, which often require a mix of transactional and non-transactional operations. It argues that migration frameworks should embed transaction boundaries directly within the SQL program itself rather than relying on external file-naming metadata.

Why it matters — Embedding transaction boundaries directly within SQL migration scripts, rather than relying on framework metadata, ensures safer execution of complex, multi-phase schema changes.

Planet PostgreSQL · 08 August 2026 · Read the original →

Storage Internals

Massively parallel Postgres backups

This post discusses the architectural challenges of executing consistent, encrypted database backups every 12 hours on highly active production databases. It focuses on achieving zero impact on production queries while handling massive data volumes through parallel processing.

Why it matters — Designing a backup system around massive parallelism allows capturing consistent, encrypted snapshots of active databases with zero impact on production query performance.

PlanetScale · 08 August 2026 · Read the original →

Vector & AI Data

Multi-agent observability: why one trace isn't enough

This post addresses the unique challenges of monitoring multi-agent AI systems, where interactions span multiple execution loops, shared memory states, and external tool calls. It explains why traditional single-trace observability patterns fail to capture the asynchronous, distributed nature of multi-agent workflows.

Why it matters — Multi-agent systems require distributed, multi-loop tracing rather than single-trace patterns to capture interactions across shared memory and asynchronous tool calls.

Redis · 08 August 2026 · Read the original →

Performance

Postgres Managed by ClickHouse Benchmarks

In PostgresBench at 500 GB, Postgres managed by ClickHouse achieved 26,328 transactions per second on a 16 vCPU, 64 GB RAM instance. This result was 2.4 to 5.2 times the throughput of Crunchy Bridge, Aurora, Neon, and RDS using the same benchmark parameters for transactional workloads.

Why it matters — Teams can achieve significantly higher transactional throughput with Postgres managed by ClickHouse, potentially reducing infrastructure costs and improving application responsiveness for write-heavy workloads.

clickhouse.com · 08 August 2026 · Read the original →

Practice

Cloud Data Warehouse Cost and Latency

Organizations are encountering cost and latency issues with serverless data warehouses like BigQuery, particularly due to 'bytes-scanned' pricing for frequently executed queries. Solutions like ClickHouse offer compute-based pricing and achieve sub-second latency for high-concurrency, real-time analytical workloads, contrasting with the batch-oriented nature of some traditional warehouses.

Why it matters — Data teams must evaluate pricing models and architectural fit to manage costs and achieve required latency for real-time dashboards and user-facing analytics.

clickhouse.com · 08 August 2026 · Read the original →

Performance

Concurrency vs. Throughput: why more parallelism can make databases slower

Examines a 16-minute MySQL production outage caused by a sudden traffic spike that spiraled into lock contention. The post tracks how query throughput collapsed from 15,000 queries per second down to 1,500 as concurrency and error rates peaked.

Why it matters — Uncontrolled concurrency beyond optimal queue depth causes severe lock contention, turning extra parallelism into lower overall system throughput.

PlanetScale · 08 August 2026 · Read the original →

Storage Internals

Asynchronous I/O in DuckDB: Work, Thread, Work

Explains how DuckDB historically relied on early filter and projection pushdowns to avoid I/O bottlenecks on local disks. It details the architectural shift toward asynchronous I/O needed when querying remote or high-latency storage where pruning alone is insufficient.

Why it matters — Filter pushdown optimization is insufficient for high-latency remote storage, requiring dedicated asynchronous I/O pipelines to keep compute threads saturated.

DuckDB · 08 August 2026 · Read the original →

Postgres

Alexey Evlampiev: Your ALTER TABLE Is Fast. The Queue Behind It Is Not.

Demonstrates how an ALTER TABLE statement that executes in just six milliseconds can stall all incoming read queries on a table for sixteen seconds. It illustrates how PostgreSQL lock queues cause cascading query delays regardless of actual execution speed.

Why it matters — The operational impact of DDL is determined by lock queue waiting time rather than the execution duration of the schema change itself.

Planet PostgreSQL · 08 August 2026 · Read the original →

Reliability

Fixed cadence to seconds: making ClickHouse Cloud autoscaling more reactive

Details the re-architecture of ClickHouse Cloud's autoscaling orchestration using Kubernetes controller-runtime and a ClickHouse-backed signals table. The new mechanism introduces a reactive fast path that triggers scale-up events in seconds instead of relying on periodic polling.

Why it matters — Replacing periodic polling with a signal-table architecture reduces autoscaling latency from minutes to seconds during sudden traffic spikes.

ClickHouse · 08 August 2026 · Read the original →

Performance

Radim Marek: The DISTINCT in your COUNT

Analyzes how using DISTINCT inside a COUNT aggregate forces PostgreSQL to run queries on a single core. It explains why this single keyword silently disables parallel query execution across worker processes for large scans.

Why it matters — Adding DISTINCT to a COUNT aggregate silently disables PostgreSQL parallel query execution, turning multi-core scans into single-threaded bottlenecks.

Planet PostgreSQL · 08 August 2026 · Read the original →

Performance

Elizabeth Garrett Christensen: Postgres COUNT(DISTINCT) Too Slow? Fast Approximation Guide

Compares exact COUNT(DISTINCT) queries against HyperLogLog probabilistic estimation in PostgreSQL, showing execution times dropping from 671ms to 320ms. It demonstrates how hashing and aggregating into HLL sketches enables instant mergeable cardinality queries.

Why it matters — HyperLogLog sketches provide mergeable pre-aggregations that replace slow distinct counts with near-instant cardinality estimates.

Planet PostgreSQL · 08 August 2026 · Read the original →

Practice

Advanced Data Modeling in Amazon Keyspaces

Amazon Keyspaces now supports User-Defined Types (UDTs) and Protocol Buffers (Protobuf) for modeling complex data. This allows grouping related attributes to improve data model clarity and application performance in a serverless database.

Why it matters — Engineers can build more maintainable and efficient applications by using UDTs or Protobuf for complex data structures in Amazon Keyspaces.

aws.amazon.com · 08 August 2026 · Read the original →

Postgres

Managed Postgres OLAP Performance Comparison

A comparison of managed Postgres services for OLAP workloads highlights how different storage engines, I/O paths, and cache behaviors impact performance. ClickHouse Managed Postgres demonstrated superior throughput compared to Aurora, despite Aurora having twice the RAM in the tested configuration.

Why it matters — Teams evaluating managed Postgres for OLAP should consider underlying storage engine architecture, as it significantly impacts throughput and resource efficiency.

clickhouse.com · 08 August 2026 · Read the original →

Data Engineering

Databricks Lakeflow for Data Engineering

Databricks introduced Lakeflow, an end-to-end solution for data engineering, encompassing ingestion, transformation, and orchestration of data. It supports both batch and streaming pipelines built on Apache Spark Declarative Pipelines.

Why it matters — Data engineers can simplify building and managing efficient batch and streaming data pipelines with Lakeflow's unified declarative framework.

Databricks · 08 August 2026 · Read the original →

Another week of trying to benchmark our way out of bad queries.

18 stories, every Tuesday

Published here every week. Follow by RSS to get it as it lands.

Next issue →