Real-time operational data with replay and reconciliation controls

Real time has meaning only when the team names the decision, source event, consumer state, tolerated delay, completeness condition, recovery objective, owner, and consequence of stale or missing data.

• Source commit point and event time
• Capture, processing, publish, and consumer availability time
• Freshness target and measurement percentile
• Completeness or watermark condition
• Tolerated delay and outage behavior
• Recovery and replay objective
• Consumer owner and cost of stale, missing, repeated, or incorrect data

Each source change or event records its source object, primary or business key, insert or update or delete semantics, correction behavior, event identifier, source position, event and commit and arrival times, schema version, partition and ordering rule, expected volume, access requirements, retention, source owner, and change-notification path.

A continuous path starts with an initial source state and then applies changes from a known position. The system records the snapshot boundary, source position, extraction range, validation result, and the point where continuous processing takes ownership so the handoff introduces no hidden gaps or repeated changes.

The ingestion layer persists a durable source position or checkpoint. Stable event and operation identifiers make retries safe.

• Deduplication and idempotent writes
• Per-key ordering where the consumer requires it
• Late and out-of-order event handling
• Transaction and batch boundaries
• Delete and correction behavior
• Partial-write recovery and checkpoint ownership
• Unprocessable-event quarantine
• Backpressure, source throttling, and schema compatibility checks

Each consumer contract states the available object or topic, key, schema version, source coverage, latest processed position, event and publish watermark, freshness and completeness state, known gaps, quarantined records, correction and delete behavior, replay or backfill status, and owner. Consumers can distinguish current, delayed, incomplete, and unavailable data.

Replay records the source and event-time range, schema and transformation versions, destination scope, consumer impact, start and finish positions, approving owner, and reconciliation result. Backfill can run in an isolated path before promotion when consumers cannot safely receive historical or repeated events.

Reconciliation can compare counts by source and operation, key presence, duplicates, checksums, watermarks, source positions, field rules, sampled records, business totals, quarantine, and unresolved differences. Monitoring separates source, capture, processing, publish, consumer, schema, replay, reconciliation, throughput, and cost layers. See it in production in the logistics change-data-capture path on Google Cloud and the Delhivery reporting case study.