Newsdesk
Engineering
Data & AI
Industries
Enterprise Systems
Go-to-Market
Longform
DesignIndiaAll stories

Databases & Data

Engines, Pipelines & Query Plans. Published every Tuesday, here and by RSS.

135 stories 16 issues 9 categories

Postgres Analytics & OLAP Streaming & CDC Storage Internals Performance Reliability Data Engineering Vector & AI Data Practice

Top stories

RSS
Data Engineering

How we migrate TBs from Heroku without logical replication

Details strategies for migrating multi-terabyte PostgreSQL databases away from Heroku Postgres where standard logical replication is disabled on default plans. It explains alternative extraction and backfill patterns used to move production databases safely.

Why it matters — Provides approach patterns for running large database cutovers when native change data capture or logical replication functionality is unavailable.

PlanetScale · 17 November 2026 · Read the original →

Data Engineering

Mandiant Feeds Deprecated, Migration Required

Google Cloud is deprecating the MANDIANT_ACTIVE_BREACH_IOC, MANDIANT_FUSION_IOC, and OPEN_SOURCE_INTEL_IOC feeds. Users are required to migrate to the GTI_IOC feed, as legacy feeds will be removed after March 18, 2027.

Why it matters — Data engineers must migrate their security intelligence feed integrations to GTI_IOC by March 18, 2027, or lose access to critical threat intelligence.

docs.cloud.google.com · 15 September 2026 · Read the original →

Practice

Christophe Pettus: 19th Nervous Breakdown

A post-mortem on the PostgreSQL 19 release cycle, which saw eight major features reverted or pulled just weeks before the freeze. It examines the breakdown in the review process and the technical debt incurred by landing complex features too late in the development window.

Why it matters — Aggressive feature landing without sufficient review cycles can force late-stage reverts that delay major database releases.

Planet PostgreSQL · 22 September 2026 · Read the original →

Postgres

How we migrated Clera’s 500 GB production database to ClickHouse Managed Postgres overnight

Details the overnight migration of a 500 GB production database with over 500 tables to a managed Postgres environment. The transition successfully reduced CPU utilization from a constant 100% to a stable 10-20% range.

Why it matters — Infrastructure-level migrations can resolve persistent performance bottlenecks that query-level optimizations cannot, specifically reclaiming up to 80% of CPU capacity.

ClickHouse · 13 October 2026 · Read the original →

Reliability

What else runs on your Postgres server, and how do we stop it from taking the database down?

Describes the mechanisms ClickHouse Managed Postgres uses to prevent supporting services (like backup agents or logging daemons) from compromising database availability. It details the use of cgroup limits, runtime budgets, and disk-full session exemptions to isolate resources.

Why it matters — Implementing disk-full session exemptions allows administrators to connect and run cleanup commands even when the disk is at 100% capacity and rejecting standard connections.

ClickHouse · 25 August 2026 · Read the original →

Another week spent either tuning PostgreSQL or planning to upgrade it.

Weekly editions

Each edition is one week's digest exactly as it was published — a summary of the week, then the stories.