Event-driven data ingestion on Google Cloud

Summary

Event-driven ingestion starts work when a meaningful state changes. A file arrives, a database row changes, or an application publishes a business event. A coordination layer reads the event and the declared metadata, decides which workload can run, and records the result. This design reduces schedule-driven polling and makes the reason for each run visible. It also creates new responsibilities. The platform must define delivery semantics, idempotency, data contracts, replay behavior, and operating evidence. The diagram shows the complete system boundary. Change events, file arrivals, and service events enter a coordination plane. The coordination plane reads the data contract, run policy, dependencies, and previous evidence. It starts, waits, rejects, replays, or escalates a workload. The workload processes data, applies quality rules, and writes an approved BigQuery data product. Failed inputs enter a quarantine, repair, and replay path. The platform records freshness, reliability, cost, and contract coverage.

Start with the operating requirement

Choose the ingestion pattern after defining the required behavior. Record these facts before selecting services: A daily finance close and a parcel movement feed have different requirements. The architecture should preserve that difference. Write the answers into a workload profile for each pipeline. A profile that names a 15-minute freshness objective, a 24-hour replay window, and a cost limit per million records gives every later design decision a test. A profile that says only "near real time" does not.

What event means that data is ready?
Which system owns the event?
Can the publisher send the same business event more than once?
Does event order matter within an entity or account?
What delay can the consuming workflow accept?
Which duplicates can the data product tolerate?
How far back must the platform replay data?
Who decides whether rejected data can continue?
Which cost unit will the team measure?

Separate arrival, coordination, execution, and evidence

Treat the platform as four connected planes. Arrival: Accept and retain a signal that data changed; Pub/Sub message, object notification, or change event. Coordination: Decide whether a workload can run; Start, wait, reject, replay, or escalate. Execution and data: Transform, validate, and publish data; Dataflow job and approved BigQuery table. Evidence: Record the operating result; Freshness, rejected records, replay count, and cost. This separation keeps scheduling logic out of transformation code. It also gives operators one place to inspect why a workload ran. The planes fail independently, and the design should respect that. Pub/Sub can accept events while a Dataflow job is down. The coordination plane can defer a start while an upstream dependency is late. The evidence plane keeps recording while execution retries. Coupling these planes into one script removes those recovery options.

Define the event contract

An event contract should contain enough information to make a safe coordination decision. Typical fields include: Keep business identity separate from transport identity. Pub/Sub assigns a message ID to a published message. If a publisher sends the same business event twice as two publish operations, Pub/Sub can assign two message IDs. The consumer still needs a business-level idempotency key. Keep the event small. A notification that carries the entity identifier, the schema version, and a pointer to the payload survives schema changes better than an event that embeds the full record. A full-payload event simplifies consumers but raises message size, redaction, and versioning costs. Pick one pattern per event type and record the reason in the contract.

Event identifier
Event type and version
Business entity identifier
Event time
Publisher
Data location or payload reference
Schema version
Correlation identifier
Replay indicator

Delivery semantics in Pub/Sub

Terms such as exactly once describe a defined part of a system. They do not automatically cover the publisher, external side effects, and final data product. Pub/Sub supports exactly-once delivery for pull subscriptions, including StreamingPull. The guarantee applies within a cloud region and uses the Pub/Sub message ID. When the setting is active, Pub/Sub does not redeliver a message before its acknowledgment deadline expires, and acknowledged messages are not redelivered. Subscribers must track processing progress until acknowledgment succeeds. Set the acknowledgment deadline for the slowest legitimate processing step, not the average. The default deadline is 10 seconds, and a subscription can extend it to 600 seconds. A consumer that processes a large file reference in 90 seconds and keeps the default deadline will see redeliveries under normal load, and each redelivery restarts the same work. Add a dead-letter topic to subscriptions that feed automated workloads. A dead-letter policy with a maximum delivery attempt count between 5 and 100 moves a repeatedly failing message out of the retry loop with its payload and attributes intact. Without it, one malformed event can occupy a subscriber indefinitely. Ordering is opt-in. A publisher sends related messages with the same ordering key, and the subscription must enable message ordering. A redelivered message blocks delivery of later messages on the same key, so a poison message on a busy key can stall an entire entity stream. Plan the dead-letter and repair path for that case before enabling ordering. Pub/Sub can still receive multiple publish operations for the same business event. The application must define business-level deduplication when that distinction matters. See Exactly-once delivery in Pub/Sub and Pub/Sub message ordering.

Exactly-once processing in Dataflow

Dataflow streaming jobs use exactly-once processing by default. Dataflow also supports an at-least-once streaming mode. At-least-once processing can reduce cost and latency when duplicate results are acceptable or when a later stage removes them. Choose the mode from the data-product requirement. Aggregations and inventory balances often need exact results. Telemetry pipelines can sometimes accept duplicates when the sink applies deterministic deduplication. Two settings deserve an explicit decision in every streaming job. Streaming Engine moves shuffle and state work off the worker virtual machines, which changes the cost profile and the failure surface of the job. Autoscaling bounds, including the maximum worker count, cap both spend and catch-up speed after a backlog. A replay that follows a three-day outage will hit the worker ceiling; the cap decides whether catch-up takes minutes or hours. See Exactly-once processing in Dataflow and Dataflow streaming modes.

Exactly-once writes to BigQuery

The BigQuery Storage Write API supports exactly-once write semantics when an application-created stream supplies offsets. The client owns offset management. An ALREADY_EXISTS response means that the offset was written. An OUT_OF_RANGE response means that the client must continue from the last successful offset. The default stream has lower coordination overhead and at-least-once semantics. Use application-created streams and offsets when the requirement supports the added state management. A pending-type stream holds written rows invisible until the application commits the stream, which gives the workload a two-phase boundary for publication. Whichever stream type the design selects, deduplication at the sink remains the last control. A merge keyed on the business idempotency key protects the approved table even when an upstream stage redelivers. See BigQuery Storage Write API best practices.

Move control information into metadata

A metadata control plane can hold the information required to coordinate a run: The control plane evaluates declared state. It should produce a durable decision record that answers: This record becomes part of the platform's operating evidence. Store it in a queryable table with the same retention as the data it describes. During an incident, the decision record is the difference between reading what happened and interviewing the people who were on shift.

Data contract and schema version
Owner and escalation route
Trigger policy
Required upstream states
Processing implementation and version
Quality rules
Retry limit
Quarantine location
Freshness objective
Retention and replay window
Which event caused the decision?
Which contract version applied?
Which dependencies were ready?
Which workload version ran?
Which quality rules passed?
Which operator approved an exception?

Define data inputs and quality constraints

The execution plane should validate inputs against the declared contract before any record reaches an approved table. Classify every rejected input into one of three categories, because each has a different repair path: Dependency failures deserve special handling. In an event-driven system, the customer record may arrive seconds after the order that references it. A wait-and-recheck policy with a bounded window resolves most of these cases without quarantine. A fixed rejection of every unmet dependency creates repair work that no one needs. Record the distribution of rejection categories by source. A source that shifts from dependency failures to contract violations has usually changed its publisher, and the schema version in its events will confirm it.

Contract violations, such as a missing required field or an unknown schema version
Semantic failures, such as a negative quantity or a timestamp outside the valid range
Dependency failures, such as a reference to a customer record that does not exist yet

Design failure and replay before release

An event-driven platform must retain enough state to recover. Use a defined recovery sequence: Quarantine the event or data reference with the rejection reason. Preserve the original event identifier, business key, offset, and contract version. Assign an owner and resolution state. Repair the contract, implementation, or source data. Replay from the saved position. Apply the same quality gate. Link the replay result to the original failure. Avoid recovery procedures that require an operator to reconstruct the original input from logs. Three failure patterns deserve a named response in the runbook. A poison message fails on every attempt and exhausts the delivery policy; the dead-letter path must route it to quarantine with its attributes intact. A hot ordering key blocks every later message for that entity while one message fails; the repair must clear or dead-letter the blocking message, not just retry it. A replay storm follows a long outage when the backlog exceeds autoscaling limits; the platform needs a catch-up plan that protects serving queries from the backfill load.

Measure the accepted data product

Pipeline success does not prove that a consumer received useful data. Measure the state at the approved output. For freshness, measure the difference between event time and availability in the approved table. Report a percentile and period, such as the 95th percentile for the last seven days. An average hides the tail, and the tail is where the missed objective lives. For reliability, record accepted runs, rejected runs, replayed runs, and unresolved failures. Group failures by reason and owner so that a noisy source becomes visible as a source problem rather than a platform problem. For quality, measure contract coverage and business-rule results. Include the number of records held back from publication, because a rising hold-back count with stable inputs is an early sign of contract drift. For cost, select a unit that follows business volume, such as cost per million accepted records or cost per completed source load. Include normal processing, quarantine storage, and replay work, so that a cheap pipeline with expensive recovery shows its real price. For recovery, measure the time from rejection to a successful replay. This metric exposes gaps in ownership and runbooks faster than any architecture review.

Cost, latency, and scale trade-offs

Every stage of the platform trades money for delay, and the trade-offs should be visible in the design record. Revisit these settings quarterly against the measured unit cost. Traffic growth changes which trade-off is correct.

Exactly-once delivery and processing cost more than at-least-once paths because the platform tracks acknowledgments, offsets, and deduplication state. Spend that money on data products that need exact results, and let duplicate-tolerant streams run cheaper.
Small, frequent events reduce latency and raise per-message overhead. Batching at the arrival or execution stage lowers cost per record and adds delay. Measure the delay the consumer actually tolerates before choosing a batch interval.
A larger Pub/Sub retention window and longer BigQuery time-travel configuration make replay easier and storage more expensive. Size the window to the longest realistic recovery, and write the number down.
Streaming capacity held for peak arrival rates sits idle at night. Autoscaling bounds and scheduled batch fallback paths both address the same problem at different response speeds.

Use scheduling when the clock is the constraint

Time-based scheduling remains appropriate when: Event-driven coordination is appropriate when: Many platforms use both patterns. The control plane should record the trigger type and apply the same ownership and evidence rules.

The source produces a complete periodic snapshot.
A contractual process closes at a defined time.
Upstream systems cannot publish a reliable readiness event.
The workload can tolerate the schedule interval.
Reprocessing the interval is the intended recovery method.
Business events have operational value before the next schedule.
Data arrival time varies.
Polling creates unnecessary work.
Downstream systems need a durable readiness decision.
The team can operate replay and idempotency controls.

What incidents and audits require

The patterns that remain dependable during ordinary operations share a few traits. The first version of an event-driven platform fails on identity, not throughput. Publishers resend events, two message IDs describe one business fact, and the platform that lacks a business idempotency key discovers the gap during its first replay. Define the key before the first table is approved. Quarantine without ownership becomes a second dead letter. A rejected event that lands in a bucket with no owner and no resolution state stays there. Assign the owner at rejection time and alert on the age of the oldest unresolved item. Finally, the evidence plane pays for itself during audits and incidents, which are the two moments when nobody has time to build it. Record decision, replay, and freshness evidence from the first live run.

Review checklist

Before release, verify that:

Each event type has an owner and versioned contract.
The business idempotency key is defined.
Ordering requirements name the ordering key and scope.
Delivery guarantees are documented for each stage.
External side effects have an idempotency or compensation strategy.
Quarantine retains the original input and reason.
Replay uses a recorded position.
Quality gates control publication.
Operators can inspect the decision record.
Freshness, reliability, quality, cost, and recovery metrics have owners.