Batch vs real-time pipelines: what actually runs in production
Batch vs real-time pipelines: what actually runs in production
The conversation around batch vs real-time pipelines has never been louder, but it is rarely grounded in reality. Real-time streaming gets the flashy conference talks and vendor case studies. Meanwhile, batch quietly powers the majority of production workloads at the companies you interact with every day.
The truth is that most mature production environments run a mix of batch and streaming — and the choice between them is rarely as obvious as vendor marketing suggests. This article is an honest, experience-based breakdown of how both architectures behave in the real world: where each one earns its place, what the actual trade-offs look like in production, and what data engineers should think through before committing to either approach.
Why the batch vs real-time decision matters more than most teams realize
Architecture decisions have a long tail. A pipeline design that looks reasonable in a proof-of-concept can become a significant source of downstream cost, operational complexity, and maintenance burden for years.
The cost of a misaligned architecture shows up slowly. If a team chooses streaming for a daily reporting workload, they don't fail immediately. They just eventually realize they are paying for continuous, always-on compute for a pipeline that only provides business value once a day.
Often, real-time is chosen for the wrong reasons. The appeal is understandable — it sounds rigorous, sophisticated, and competitive. But the actual business need is often just data that is fresher than daily, which hourly batch runs can solve at a fraction of the cost.
The business questions should drive the architecture decision, not the other way around. Ask: how stale can this data be before it actually costs us money or impacts a customer? If the answer is "a few hours," you rarely need a streaming pipeline.
How batch pipelines work in production
Core mechanics
Batch pipelines collect data over a defined window — an hour, a night, a week — and process it all at once. The typical lifecycle looks like this: data accumulates in a source system, a scheduler triggers ingestion at a defined interval, a transformation layer cleans and reshapes the data, and the processed data lands in a target — a data warehouse, a data lake, or a reporting database. The process repeats on schedule. There is no continuous compute. The system is idle between runs.
Common tools
Apache Spark for large-scale distributed transformation, dbt for SQL-based transformation layers in the warehouse, Apache Airflow, Dagster, or Prefect for orchestration and scheduling, and AWS Glue, Google Dataflow, or Azure Data Factory for managed ETL in cloud environments.
Operational characteristics
Batch pipelines are relatively forgiving to operate. Cost is predictable — you know when compute runs and roughly what it costs. Monitoring is simpler — a pipeline either completes or it doesn't, and when it fails, it fails loudly. Debugging is straightforward — you can inspect the full input dataset at the time of failure and reproduce a bug by re-running the job. Data quality validation is easier — you validate a complete, bounded dataset before loading, so issues are caught before they reach consumers.
Batch code example
Here is a standard PySpark pattern for an overnight batch run using MERGE to update existing records in a Delta table:
# Read the overnight batch of new records daily_batch_df = spark.read.format("parquet").load("/mnt/raw/transactions/date=2026-06-08") daily_batch_df.createOrReplaceTempView("new_transactions") # Merge into our Silver layer Delta table spark.sql(""" MERGE INTO silver.core_transactions target USING new_transactions source ON target.transaction_id = source.transaction_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * """)
# Read the overnight batch of new records daily_batch_df = spark.read.format("parquet").load("/mnt/raw/transactions/date=2026-06-08") daily_batch_df.createOrReplaceTempView("new_transactions") # Merge into our Silver layer Delta table spark.sql(""" MERGE INTO silver.core_transactions target USING new_transactions source ON target.transaction_id = source.transaction_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * """)
How real-time pipelines work in production
Core mechanics
Streaming pipelines process data as events arrive, continuously, without waiting to accumulate a batch. Events flow from source systems into an event streaming platform, a stream processing engine reads those events and applies transformations, processed results are written to a serving layer in near real-time, and downstream consumers receive updates with minimal delay.
Where streaming earns its place
There are use cases where streaming is not just useful, it is the only viable architecture. Fraud detection: a suspicious transaction needs to be flagged before it clears, and a streaming pipeline processes the event as it happens and blocks or flags it before damage occurs. Logistics and supply chain tracking: a fleet of delivery vehicles or a cold chain with temperature sensors requires continuous monitoring with immediate alerting, hourly batches won't catch a refrigeration failure in time. IoT data processing: thousands of devices generating readings every second cannot wait for a batch window. Customer-facing triggers: abandoned cart emails, real-time inventory availability, and in-session personalization require sub-second awareness. Financial market data: trading systems, risk engines, and market surveillance platforms run on streaming by necessity.
Common tools
Apache Kafka as the event streaming backbone, Apache Flink for stateful high-throughput stream processing, Spark Structured Streaming for teams already invested in the Spark ecosystem, and AWS Kinesis, Google Pub/Sub, or Azure Event Hubs for managed streaming in cloud environments.
Operational characteristics
Streaming pipelines are more powerful — and more demanding. Infrastructure overhead is higher: Kafka clusters, stream processors, schema registries, dead-letter queues, and consumer lag monitoring all require ongoing attention. Monitoring is more complex: you are watching consumer lag, throughput, partition health, checkpoint frequency, and state store size continuously rather than waiting for job completion. Failure modes are subtler — which is where streaming earns its reputation for operational difficulty.
Streaming code example
Here is continuous streaming using Spark Structured Streaming. The explicit checkpoint location is non-negotiable for reliable restarts:
# Continuously read from a Kafka event topic streaming_df = spark.readStream \ .format("kafka") \ .option("kafka.bootstrap.servers", "broker:9092") \ .option("subscribe", "live_telemetry") \ .load() # Write continuously to a Delta table streaming_df.writeStream \ .format("delta") \ .outputMode("append") \ .option("checkpointLocation", "/Volumes/prod/checkpoints/telemetry_ckpt") \ .start("/tables/silver/live_telemetry")
# Continuously read from a Kafka event topic streaming_df = spark.readStream \ .format("kafka") \ .option("kafka.bootstrap.servers", "broker:9092") \ .option("subscribe", "live_telemetry") \ .load() # Write continuously to a Delta table streaming_df.writeStream \ .format("delta") \ .outputMode("append") \ .option("checkpointLocation", "/Volumes/prod/checkpoints/telemetry_ckpt") \ .start("/tables/silver/live_telemetry")
Batch vs real-time pipelines: an honest production comparison
Dimension | Batch | Streaming |
Latency | Minutes to hours | Milliseconds to seconds |
Infrastructure cost | Lower — pay for compute during runs only | Higher — continuous compute allocation |
Operational complexity | Lower — clear success/failure, easier debugging | Higher — consumer lag, state, checkpointing |
Failure visibility | High — failures are loud and immediate | Low — failures can be silent and delayed |
Data quality controls | Easier — validate complete datasets before load | Harder — validation must happen in-flight |
Debugging | Straightforward — inspect the full input at failure time | Difficult — requires log replay and state reconstruction |
Team skill requirement | Lower bar — widely understood | Higher bar — requires streaming-specific expertise |
Best for | Analytics, reporting, historical processing | Fraud, IoT, logistics, real-time triggers |
A few dimensions worth expanding on. Latency requirements: micro-batch pipelines running every 15 minutes satisfy 90% of real-time business requests at a fraction of the operational burden, ask how fresh the data actually needs to be before defaulting to streaming. Cost differences at scale: batch has lower infrastructure cost per unit of compute, while streaming carries continuous baseline costs plus the hidden costs of schema registries and the engineering time required to untangle complex failures. Monitoring and failure recovery: batch fails loudly, streaming fails quietly, without proper state management, a streaming pipeline might restart and silently duplicate records or drop aggregations. Team skill requirements: running streaming effectively requires deep distributed systems expertise that not every team has.
What companies actually run in production
Most mature organizations run both batch and streaming workloads. Different business processes have different latency requirements.
Historically, teams used the Lambda architecture, running a slow, accurate batch layer alongside a fast, approximate streaming layer, which meant writing logic twice. The Kappa architecture tried to fix this by treating everything as a stream, but event retention at scale and reprocessing historical data proved too complex and expensive for most teams. Kappa works well for organizations with strong streaming expertise and workloads where historical reprocessing is infrequent.
The medallion architecture, popularized by Databricks, organizes data into three progressive quality layers: Bronze for raw data as ingested, Silver for cleaned and enriched data, and Gold for analytics-ready aggregations with business logic applied. The power of this pattern is that it supports both batch and streaming ingestion at the Bronze layer, with the same downstream quality progression applying in both cases. A Bronze table might be populated by a nightly batch Spark job in one pipeline and by a Kafka consumer in another, the Silver and Gold transformations are the same regardless.
Real examples across industries: in manufacturing, streaming IoT sensor data catches machines overheating instantly while nightly batch jobs process daily output reports. In logistics, streaming pipelines reroute delivery trucks based on live traffic while batch pipelines calculate weekly fleet fuel efficiency. In financial services, streaming engines block fraudulent card swipes at the register while overnight batch jobs handle official ledger reconciliation.
How to choose the right approach for your use case
Before choosing an architecture, answer these questions honestly. What is the maximum acceptable latency, and what actually breaks if this data is 30 minutes old? Is the dataset so large that processing it in an always-on stream will destroy your cloud budget? Does your team know how to manage Kafka consumer groups, exactly-once semantics, and late-arriving data when things break at 2 AM?
A practical starting point: begin with a batch. Validate the data model, transformations, and downstream consumption. Once stable, identify the specific layer where latency matters enough to justify streaming, and add it incrementally.
Watch out for these red flags. "The business wants it in real-time" did you ask them what they actually define as real-time? "We want to be future-proof" over-engineering for hypothetical requirements is a reliable way to build technical debt. "Batch feels old" it feels predictable and cheap, which is usually exactly what you want.
Governance, reliability, and enterprise readiness
Unity Catalog provides a single governance layer for Databricks environments that works regardless of whether data arrived via batch or streaming. Column-level lineage lets you trace any downstream metric back through Silver and Bronze to its original source event. Fine-grained access control means row and column-level security policies are defined once and applied everywhere, batch queries, streaming reads, and notebook exploration. Audit logging provides a complete record of who accessed what, when, and how across both pipeline types. For regulated industries, this unified governance story is a significant practical advantage over architectures where batch and streaming data live in separately governed systems.
Ensuring data quality in streaming environments
The absence of a pre-load validation step, which batch pipelines have by default, means streaming pipelines need compensating controls. A schema registry enforces schema contracts and detects upstream changes before they corrupt downstream states. Dead-letter queues capture and preserve malformed events rather than silently dropping them. Watermarking handles late-arriving data without skewing time-window aggregations. Idempotent writes ensure that reprocessed events don't create duplicate records. And reconciliation jobs, often batch, periodically compare streaming output against a ground truth to catch drift.
Before a streaming pipeline carries mission-critical workloads, the following should be in place: checkpoint strategy defined and tested under restart conditions, consumer lag monitoring with defined alerting thresholds, schema registry in use with compatibility enforcement, dead-letter queue with operational runbooks, exactly-once semantics verified or at-least-once with idempotent sinks explicitly validated, and clear ownership of on-call responsibilities for the pipeline. The teams that skip these steps don't fail immediately, they build technical debt that surfaces at the worst possible moment.
What data engineers should know before making the call
Architecture decisions look simple from the outside and complicated from the inside. Before committing, it is worth putting these questions to stakeholders directly: when you say real-time, what are you actually trying to do that you can't do today? What breaks if this data is delayed by 15 minutes or an hour? How often does the upstream system change its schema or data format? Who will own this pipeline when something goes wrong at 2 AM? What is the cost of being wrong, a wrong answer in a dashboard, or a wrong action taken by a downstream system?
Making the case for a batch in an environment that is excited about streaming requires honesty about what the organization is actually buying. The operational cost of streaming is real and ongoing. The engineering time to build reliable stateful pipelines is significant. The risk of silent failures is higher than most stakeholders appreciate. That does not mean streaming is the wrong choice for the right workloads, it clearly is the right one. But the decision should be built on an honest assessment of the actual business requirement and the team's capacity to operate the system reliably, not on the assumption that newer means better.
At Tenjumps, we help teams make architecture decisions based on what their specific workloads, cost constraints, and operational capacity actually require, not on what is trending. If your pipelines are underperforming, costing more than they should, or failing in ways that are hard to diagnose, we would be glad to talk through what is happening and what the right path forward looks like.
Kavya Kumari is a data engineer at Tenjumps specializing in data pipeline architecture and Databricks implementations.
The conversation around batch vs real-time pipelines has never been louder, but it is rarely grounded in reality. Real-time streaming gets the flashy conference talks and vendor case studies. Meanwhile, batch quietly powers the majority of production workloads at the companies you interact with every day.
The truth is that most mature production environments run a mix of batch and streaming — and the choice between them is rarely as obvious as vendor marketing suggests. This article is an honest, experience-based breakdown of how both architectures behave in the real world: where each one earns its place, what the actual trade-offs look like in production, and what data engineers should think through before committing to either approach.
Why the batch vs real-time decision matters more than most teams realize
Architecture decisions have a long tail. A pipeline design that looks reasonable in a proof-of-concept can become a significant source of downstream cost, operational complexity, and maintenance burden for years.
The cost of a misaligned architecture shows up slowly. If a team chooses streaming for a daily reporting workload, they don't fail immediately. They just eventually realize they are paying for continuous, always-on compute for a pipeline that only provides business value once a day.
Often, real-time is chosen for the wrong reasons. The appeal is understandable — it sounds rigorous, sophisticated, and competitive. But the actual business need is often just data that is fresher than daily, which hourly batch runs can solve at a fraction of the cost.
The business questions should drive the architecture decision, not the other way around. Ask: how stale can this data be before it actually costs us money or impacts a customer? If the answer is "a few hours," you rarely need a streaming pipeline.
How batch pipelines work in production
Core mechanics
Batch pipelines collect data over a defined window — an hour, a night, a week — and process it all at once. The typical lifecycle looks like this: data accumulates in a source system, a scheduler triggers ingestion at a defined interval, a transformation layer cleans and reshapes the data, and the processed data lands in a target — a data warehouse, a data lake, or a reporting database. The process repeats on schedule. There is no continuous compute. The system is idle between runs.
Common tools
Apache Spark for large-scale distributed transformation, dbt for SQL-based transformation layers in the warehouse, Apache Airflow, Dagster, or Prefect for orchestration and scheduling, and AWS Glue, Google Dataflow, or Azure Data Factory for managed ETL in cloud environments.
Operational characteristics
Batch pipelines are relatively forgiving to operate. Cost is predictable — you know when compute runs and roughly what it costs. Monitoring is simpler — a pipeline either completes or it doesn't, and when it fails, it fails loudly. Debugging is straightforward — you can inspect the full input dataset at the time of failure and reproduce a bug by re-running the job. Data quality validation is easier — you validate a complete, bounded dataset before loading, so issues are caught before they reach consumers.
Batch code example
Here is a standard PySpark pattern for an overnight batch run using MERGE to update existing records in a Delta table:
# Read the overnight batch of new records daily_batch_df = spark.read.format("parquet").load("/mnt/raw/transactions/date=2026-06-08") daily_batch_df.createOrReplaceTempView("new_transactions") # Merge into our Silver layer Delta table spark.sql(""" MERGE INTO silver.core_transactions target USING new_transactions source ON target.transaction_id = source.transaction_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * """)
How real-time pipelines work in production
Core mechanics
Streaming pipelines process data as events arrive, continuously, without waiting to accumulate a batch. Events flow from source systems into an event streaming platform, a stream processing engine reads those events and applies transformations, processed results are written to a serving layer in near real-time, and downstream consumers receive updates with minimal delay.
Where streaming earns its place
There are use cases where streaming is not just useful, it is the only viable architecture. Fraud detection: a suspicious transaction needs to be flagged before it clears, and a streaming pipeline processes the event as it happens and blocks or flags it before damage occurs. Logistics and supply chain tracking: a fleet of delivery vehicles or a cold chain with temperature sensors requires continuous monitoring with immediate alerting, hourly batches won't catch a refrigeration failure in time. IoT data processing: thousands of devices generating readings every second cannot wait for a batch window. Customer-facing triggers: abandoned cart emails, real-time inventory availability, and in-session personalization require sub-second awareness. Financial market data: trading systems, risk engines, and market surveillance platforms run on streaming by necessity.
Common tools
Apache Kafka as the event streaming backbone, Apache Flink for stateful high-throughput stream processing, Spark Structured Streaming for teams already invested in the Spark ecosystem, and AWS Kinesis, Google Pub/Sub, or Azure Event Hubs for managed streaming in cloud environments.
Operational characteristics
Streaming pipelines are more powerful — and more demanding. Infrastructure overhead is higher: Kafka clusters, stream processors, schema registries, dead-letter queues, and consumer lag monitoring all require ongoing attention. Monitoring is more complex: you are watching consumer lag, throughput, partition health, checkpoint frequency, and state store size continuously rather than waiting for job completion. Failure modes are subtler — which is where streaming earns its reputation for operational difficulty.
Streaming code example
Here is continuous streaming using Spark Structured Streaming. The explicit checkpoint location is non-negotiable for reliable restarts:
# Continuously read from a Kafka event topic streaming_df = spark.readStream \ .format("kafka") \ .option("kafka.bootstrap.servers", "broker:9092") \ .option("subscribe", "live_telemetry") \ .load() # Write continuously to a Delta table streaming_df.writeStream \ .format("delta") \ .outputMode("append") \ .option("checkpointLocation", "/Volumes/prod/checkpoints/telemetry_ckpt") \ .start("/tables/silver/live_telemetry")
Batch vs real-time pipelines: an honest production comparison
Dimension | Batch | Streaming |
Latency | Minutes to hours | Milliseconds to seconds |
Infrastructure cost | Lower — pay for compute during runs only | Higher — continuous compute allocation |
Operational complexity | Lower — clear success/failure, easier debugging | Higher — consumer lag, state, checkpointing |
Failure visibility | High — failures are loud and immediate | Low — failures can be silent and delayed |
Data quality controls | Easier — validate complete datasets before load | Harder — validation must happen in-flight |
Debugging | Straightforward — inspect the full input at failure time | Difficult — requires log replay and state reconstruction |
Team skill requirement | Lower bar — widely understood | Higher bar — requires streaming-specific expertise |
Best for | Analytics, reporting, historical processing | Fraud, IoT, logistics, real-time triggers |
A few dimensions worth expanding on. Latency requirements: micro-batch pipelines running every 15 minutes satisfy 90% of real-time business requests at a fraction of the operational burden, ask how fresh the data actually needs to be before defaulting to streaming. Cost differences at scale: batch has lower infrastructure cost per unit of compute, while streaming carries continuous baseline costs plus the hidden costs of schema registries and the engineering time required to untangle complex failures. Monitoring and failure recovery: batch fails loudly, streaming fails quietly, without proper state management, a streaming pipeline might restart and silently duplicate records or drop aggregations. Team skill requirements: running streaming effectively requires deep distributed systems expertise that not every team has.
What companies actually run in production
Most mature organizations run both batch and streaming workloads. Different business processes have different latency requirements.
Historically, teams used the Lambda architecture, running a slow, accurate batch layer alongside a fast, approximate streaming layer, which meant writing logic twice. The Kappa architecture tried to fix this by treating everything as a stream, but event retention at scale and reprocessing historical data proved too complex and expensive for most teams. Kappa works well for organizations with strong streaming expertise and workloads where historical reprocessing is infrequent.
The medallion architecture, popularized by Databricks, organizes data into three progressive quality layers: Bronze for raw data as ingested, Silver for cleaned and enriched data, and Gold for analytics-ready aggregations with business logic applied. The power of this pattern is that it supports both batch and streaming ingestion at the Bronze layer, with the same downstream quality progression applying in both cases. A Bronze table might be populated by a nightly batch Spark job in one pipeline and by a Kafka consumer in another, the Silver and Gold transformations are the same regardless.
Real examples across industries: in manufacturing, streaming IoT sensor data catches machines overheating instantly while nightly batch jobs process daily output reports. In logistics, streaming pipelines reroute delivery trucks based on live traffic while batch pipelines calculate weekly fleet fuel efficiency. In financial services, streaming engines block fraudulent card swipes at the register while overnight batch jobs handle official ledger reconciliation.
How to choose the right approach for your use case
Before choosing an architecture, answer these questions honestly. What is the maximum acceptable latency, and what actually breaks if this data is 30 minutes old? Is the dataset so large that processing it in an always-on stream will destroy your cloud budget? Does your team know how to manage Kafka consumer groups, exactly-once semantics, and late-arriving data when things break at 2 AM?
A practical starting point: begin with a batch. Validate the data model, transformations, and downstream consumption. Once stable, identify the specific layer where latency matters enough to justify streaming, and add it incrementally.
Watch out for these red flags. "The business wants it in real-time" did you ask them what they actually define as real-time? "We want to be future-proof" over-engineering for hypothetical requirements is a reliable way to build technical debt. "Batch feels old" it feels predictable and cheap, which is usually exactly what you want.
Governance, reliability, and enterprise readiness
Unity Catalog provides a single governance layer for Databricks environments that works regardless of whether data arrived via batch or streaming. Column-level lineage lets you trace any downstream metric back through Silver and Bronze to its original source event. Fine-grained access control means row and column-level security policies are defined once and applied everywhere, batch queries, streaming reads, and notebook exploration. Audit logging provides a complete record of who accessed what, when, and how across both pipeline types. For regulated industries, this unified governance story is a significant practical advantage over architectures where batch and streaming data live in separately governed systems.
Ensuring data quality in streaming environments
The absence of a pre-load validation step, which batch pipelines have by default, means streaming pipelines need compensating controls. A schema registry enforces schema contracts and detects upstream changes before they corrupt downstream states. Dead-letter queues capture and preserve malformed events rather than silently dropping them. Watermarking handles late-arriving data without skewing time-window aggregations. Idempotent writes ensure that reprocessed events don't create duplicate records. And reconciliation jobs, often batch, periodically compare streaming output against a ground truth to catch drift.
Before a streaming pipeline carries mission-critical workloads, the following should be in place: checkpoint strategy defined and tested under restart conditions, consumer lag monitoring with defined alerting thresholds, schema registry in use with compatibility enforcement, dead-letter queue with operational runbooks, exactly-once semantics verified or at-least-once with idempotent sinks explicitly validated, and clear ownership of on-call responsibilities for the pipeline. The teams that skip these steps don't fail immediately, they build technical debt that surfaces at the worst possible moment.
What data engineers should know before making the call
Architecture decisions look simple from the outside and complicated from the inside. Before committing, it is worth putting these questions to stakeholders directly: when you say real-time, what are you actually trying to do that you can't do today? What breaks if this data is delayed by 15 minutes or an hour? How often does the upstream system change its schema or data format? Who will own this pipeline when something goes wrong at 2 AM? What is the cost of being wrong, a wrong answer in a dashboard, or a wrong action taken by a downstream system?
Making the case for a batch in an environment that is excited about streaming requires honesty about what the organization is actually buying. The operational cost of streaming is real and ongoing. The engineering time to build reliable stateful pipelines is significant. The risk of silent failures is higher than most stakeholders appreciate. That does not mean streaming is the wrong choice for the right workloads, it clearly is the right one. But the decision should be built on an honest assessment of the actual business requirement and the team's capacity to operate the system reliably, not on the assumption that newer means better.
At Tenjumps, we help teams make architecture decisions based on what their specific workloads, cost constraints, and operational capacity actually require, not on what is trending. If your pipelines are underperforming, costing more than they should, or failing in ways that are hard to diagnose, we would be glad to talk through what is happening and what the right path forward looks like.
Kavya Kumari is a data engineer at Tenjumps specializing in data pipeline architecture and Databricks implementations.
Share