Replacing Batch Statement Generation
with Event-Driven .NET

A nightly job regenerates every customer's statement, report, or summary — whether or not anything in that customer's data actually changed since the last run. It works fine at a thousand accounts. At a hundred thousand, the batch window stops fitting inside the night, support starts fielding "why does my statement still show yesterday's numbers" tickets, and one bad record can take the whole run down with it.

This article covers the event-driven alternative: regenerating one record when the thing that invalidated it actually happens, instead of regenerating everything on a timer. It walks through where Azure Service Bus and Event Hub each fit, a concrete .NET pattern for the regeneration handler — naive version first, then corrected for idempotency, cancellation, and dead-lettering — and the audit trail you need alongside it. It closes with when this is the wrong move, because batch is still correct for a real share of reporting workloads.

Why Batch Generation Breaks Down at Scale

A batch job that regenerates everything has three problems that don't show up in a demo and all get worse together as volume grows.

• The window keeps growing. Runtime scales with the number of records, not with how many actually changed. A job that comfortably finished overnight at last year's volume eventually doesn't — and "make the batch faster" is a treadmill, not a fix.
• Data is stale between runs, by construction. A customer who changes something at 9 a.m. sees yesterday's number until the next scheduled run, no matter how small or simple the change was. The gap is invisible until someone notices it, and then it's a support ticket, not a bug report.
• Failure handling is coarse. When record 4,000 of 50,000 throws, the naive job either takes the whole run down or swallows the exception and silently produces 49,999 statements and one missing one — and "silently" is the part that costs you later, when nobody can say which runs were actually clean.

None of this is a reason to rewrite every batch job you own — see the last section. It's the specific shape of pain that event-driven regeneration is the right answer to: sparse, discrete changes where staleness has a real cost.

Event Hub or Service Bus: A Decision Framework

"Event-driven" gets treated as one thing on Azure, but Event Hub and Service Bus solve different problems and the choice matters more than the migration guides suggest.

• Service Bus is a broker: queues and topics with per-message completion, dead-lettering, sessions, and scheduled delivery. It's built for discrete business events where you need to know, for each message, whether it succeeded — "this account's position changed, regenerate this account's statement."
• Event Hub is an append-only log, the same shape as Kafka: built for high-volume streaming — thousands of events per second — read by one or more independent consumer groups at their own pace, tracked by checkpoint rather than per-message acknowledgement. It's built for "a continuous stream of position deltas that several different readers need to process."

The practical test: if you need to reason about this one message — did it succeed, should it be retried, should it be dead-lettered for a human to look at — you want Service Bus. If you need to reason about throughput and independent readers over a continuous stream, you want Event Hub. Statement and report regeneration triggered by discrete account-level changes is almost always the first case, which is why the worked example below uses Service Bus.

That's the short version of the decision. For the full framework — cost tradeoffs at scale, the hybrid pattern when a pipeline genuinely needs both, and migration pitfalls — see Azure Service Bus vs Event Hub for financial pipelines.

There's a third option worth naming so it doesn't get conflated with either: if the actual goal is "generate this fast on request" rather than "keep this fresh between requests," eventing isn't the tool at all — making the synchronous read fast enough, with parallel multi-source reads and a cache racing a response-time budget, solves a related but different problem. That's the pattern behind how we rebuilt account statement generation to return most requests in under five seconds. It doesn't replace the event-driven approach below — it answers a different question: latency on demand, not staleness between updates.

A Concrete Migration Pattern

Start with the shape almost every batch job has: loop over everything, regenerate, save.

// Naive: regenerate every statement on a timer, whether anything changed or not.
public sealed class NightlyStatementJob
{
    private readonly IStatementRepository _statements;
    private readonly IAccountRepository _accounts;

    public NightlyStatementJob(IStatementRepository statements, IAccountRepository accounts)
    {
        _statements = statements;
        _accounts = accounts;
    }

    public async Task RunAsync()
    {
        var accounts = await _accounts.GetAllAsync();
        foreach (var account in accounts)
        {
            // No cancellation, no per-item isolation — one bad account can stall the
            // whole run, and every account is regenerated even if nothing changed.
            var statement = await _statements.GenerateAsync(account.Id);
            await _statements.SaveAsync(statement);
        }
    }
}

The corrected version replaces the timer with a trigger: a handler that regenerates exactly one account, in response to exactly the event that invalidated it. Because Service Bus delivery is at-least-once, the handler has to tolerate being called twice for the same change — that's not an edge case, it's normal operation under retry.

// Corrected: regenerate one account's statement for the event that invalidated
// it. Safe to invoke more than once for the same event — delivery is at-least-once.
public sealed class StatementRegenerationHandler
{
    private readonly IStatementRepository _statements;
    private readonly IIdempotencyStore _seen;
    private readonly ILogger<StatementRegenerationHandler> _log;

    public StatementRegenerationHandler(
        IStatementRepository statements, IIdempotencyStore seen,
        ILogger<StatementRegenerationHandler> log)
    {
        _statements = statements;
        _seen = seen;
        _log = log;
    }

    /// <summary>Regenerates one account's statement for a position-changed event.</summary>
    [Function("StatementRegeneration")]
    public async Task HandleAsync(
        [ServiceBusTrigger("position-changed", Connection = "ServiceBus:ConnectionString")]
        ServiceBusReceivedMessage message,
        ServiceBusMessageActions messageActions,
        CancellationToken cancellationToken)
    {
        var evt = message.Body.ToObjectFromJson<PositionChangedEvent>();

        // Regenerating twice for the same revision is wasted work, not a correctness
        // bug — skipping it just keeps cost and downstream notification noise down.
        if (await _seen.ContainsAsync(evt.AccountId, evt.Revision, cancellationToken))
        {
            await messageActions.CompleteMessageAsync(message, cancellationToken);
            return;
        }

        try
        {
            var statement = await _statements.GenerateAsync(evt.AccountId, cancellationToken);
            await _statements.SaveAsync(statement, cancellationToken);
            await _seen.MarkAsync(evt.AccountId, evt.Revision, cancellationToken);
            await messageActions.CompleteMessageAsync(message, cancellationToken);

            _log.LogInformation(
                "Regenerated statement for {AccountId} at revision {Revision}",
                evt.AccountId, evt.Revision);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            _log.LogError(ex, "Statement regeneration failed for {AccountId}", evt.AccountId);
            await messageActions.DeadLetterMessageAsync(
                message, "RegenerationFailed", ex.Message, cancellationToken);
        }
    }
}

If the trigger really were a high-volume stream rather than a discrete event, the shape changes but the discipline doesn't: instead of completing or dead-lettering one message, you process a batch and checkpoint your position in the log.

// Event Hub has no per-message Complete/DeadLetter — a consumer checkpoints its
// position in the log instead. Lose the checkpoint and you replay from the last
// saved offset, not from "the one event that failed".
public async Task ProcessEventsAsync(
    ProcessorPartitionContext ctx, EventBatch batch, CancellationToken ct)
{
    foreach (var e in batch.Events)
        await HandlePositionDeltaAsync(e, ct);

    await ctx.UpdateCheckpointAsync(ct);  // commits progress for the whole batch
}

Graceful Degradation and the Audit Trail

Moving to events doesn't remove the question of what happens when something goes wrong — it changes the shape of the answer. Service Bus messages are durable, so if the handler is down or backlogged, nothing is lost; changes simply queue until processing resumes. What you do lose, honestly, is freshness during the outage — and the right response is to surface that rather than hide it. Show a "last updated" timestamp on the statement instead of implying it's always current, and alert on queue depth and message age, not just on hard failures.

Record every regeneration — account, triggering revision, outcome, timestamp — to an append-only log, not just application logs. The same change-driven design that makes this fast also makes it harder to answer "was this account's statement ever regenerated for that change?" by eyeballing a database, because there's no batch run boundary to point to anymore. The log is what answers that question. If you're building this for a regulated workload, the four properties of an auditable backend system apply directly to the regeneration log itself.

When Batch Is Still the Right Call

Don't take this as "batch is legacy, events are correct." Batch is the right tool when:

• Changes are wholesale, not sparse. An end-of-day price close or a full data reload invalidates everything at once. Per-record events don't help when every record changes together — you still need a bulk path, and a well-tuned batch job is that path.
• Nobody needs sub-day freshness. If a once-daily number is genuinely what the business agreed to, eventing buys you freshness nobody asked for, at a real ongoing cost.
• The volume doesn't justify the machinery. A message broker, an idempotency store, and dead-letter handling are systems you now operate. For a few thousand records that finish in minutes, a cron job and a well-indexed query already solve the problem — adding events here is solving a scale problem you don't have.

The decision rule is simple: move to event-driven regeneration when staleness has a real, named cost and changes are sparse relative to the total population. Keep batch when changes are wholesale or freshness genuinely doesn't matter — and don't let "batch" become a word that embarrasses you into a migration the workload doesn't need.

How do I choose between Azure Service Bus and Event Hub for this?

Start from what triggers the regeneration. A discrete business event for one entity — a position changed, an order shipped — fits Service Bus: per-message completion, dead-lettering, and sessions give you precise control over one unit of work. A continuous, high-volume stream that several independent consumers need to read at their own pace — telemetry, ticks, sensor data — fits Event Hub's append-only log and checkpointing model. Most statement and report regeneration is the discrete case, so Service Bus is the right default.

What if the same change event is delivered twice?

Plan for it — Service Bus and Event Hub are both at-least-once by design, so redelivery is normal operation, not an edge case. Key the idempotency check on the entity id plus a revision or sequence number, not on the message id, so a genuine retry is recognised as already-handled.

Is this worth doing for a small number of records?

Usually not. A message broker, an idempotency store, and dead-letter handling are real systems to operate. If a nightly job already finishes comfortably inside its window and nobody is asking for fresher data, the honest answer is to leave it alone — the event-driven version earns its complexity at the volume and staleness-cost where batch actually starts hurting, not before.

Can I migrate one entity type at a time, or does this have to be all-or-nothing?

One at a time. Pick the entity type where staleness costs the most — usually the one generating the most support tickets — wire its trigger and handler, and leave everything else on the nightly job until that path is proven. There's no requirement to retire the whole batch job in one cutover, and migrating incrementally is what makes it safe to roll back if the event-driven path misbehaves.

Further Reading

Hohpe & Woolf, Enterprise Integration Patterns (2003), is still the clearest treatment of the competing-consumers and idempotent-receiver ideas this pattern depends on. For platform specifics, the Microsoft Azure Architecture Center's Competing Consumers pattern and its asynchronous messaging technology comparison are the canonical references for choosing and operating the broker itself.

Back to Insights