Newsdesk
Engineering
Data & AI
Industries
Enterprise Systems
Go-to-Market
Longform
DesignIndiaAll stories

Engineering Reading

21 November 2026

31 new posts across 11 engineering blogs — 7 worth your time.

One week of Engineering Reading, 7 stories, as published.

Infrastructure

How we scaled metrics aggregation by moving durable state into storage

Datadog details the evolution of its metrics aggregation platform to scale with high data volumes. The post explains transitioning from fixed partition ownership to transferable state and implementing storage-side reconciliation.

Why it matters — Moving state reconciliation to the storage layer allows stateful stream processing systems to decouple partition ownership from state durability.

Datadog · 21 November 2026 · Read the original →

Practice & Craft

When Can You Ship on Offline Evals Alone?

Spotify discusses the decision framework for when teams building AI features can safely bypass slow live A/B tests. It examines the trade-offs of deciding whether to ship directly based on offline evaluation metrics.

Why it matters — Establishing high-confidence offline evaluation standards allows teams to skip multi-week online A/B testing cycles when releasing AI updates.

Spotify · 21 November 2026 · Read the original →

Data & ML

De-identifying text at scale: How Airbnb protects personal data in AI training

Airbnb explains its system for identifying and scrubbing personal data from user messages, reviews, and customer support chats. This de-identification process sanitizes sensitive user information before text data is ingested into AI models for training.

Why it matters — Automated text de-identification on raw communications prevents personal data leakage before training datasets reach ML pipelines.

Airbnb · 21 November 2026 · Read the original →

Reliability

Rebuilding Guardian: How Pinterest Upgraded Its Real-Time Abuse Platform to Modern Infrastructure…

Pinterest describes rebuilding Guardian, its platform for real-time abuse detection, on modern infrastructure. The post details how the system processes real-time event data at scale to generate insights and operational actions.

Why it matters — Re-architecting real-time event infrastructure allows abuse systems to evaluate high-volume live data and enforce policy decisions with minimal latency.

Pinterest · 21 November 2026 · Read the original →

Data & ML

How Meta Uses AI Agents to Scale ML Innovations Across Ads Ranking Models

Meta outlines how its autonomous Ranking Engineer Agent (REA) propagates machine learning optimizations across its ads ranking fleet. The agent automates hypothesis generation, experiment design, and architectural deployment across diverse models.

Why it matters — Autonomous agents can systematically port and validate architectural ML improvements across an entire fleet of production models.

Meta · 21 November 2026 · Read the original →

Developer Experience

Building the Agent Host Protocol for sessions that continue across devices

GitHub details the creation of the Agent Host Protocol for GitHub Copilot. The protocol allows long-running AI agent tasks to maintain session state continuously across different application windows and devices.

GitHub · 21 November 2026 · Read the original →

Infrastructure

How Rakbank built centralized observability at scale with Amazon OpenSearch Service

Rakbank implemented a multi-account telemetry pipeline ingesting 150 GB of data per day using Amazon OpenSearch Service. The architecture uses hot, warm, and cold storage tiers combined with S3 retention for long-term compliance.

Why it matters — Structuring search and log data into hot, warm, cold, and S3 tiers reduced cluster storage costs by 55 percent compared to an all-hot design.

AWS Architecture · 21 November 2026 · Read the original →

Half these systems exist to observe or sanitize what the other half generates.

7 stories, every Saturday

Published here every week. Follow by RSS to get it as it lands.

← Previous issue