From a Slow Monolith to Account Statements in Under 5 Seconds

The Constraints

• A five-second response target. The client set a hard SLA: the large majority of statement requests had to return within five seconds, with a graceful path for the ones that could not — never a hung screen and never a timeout.
• Data spread across many sources. A statement composes the core Azure SQL database (positions and dividends — and, via a linked server, additional data in other databases) together with reference data from other microservices reached over their APIs. Reading everything sequentially was a primary cause of the slowness.
• Full reports versus single tabs. An on-screen tab reads only its own section and is cheap; a full PDF or Excel export reads every section and is an order of magnitude heavier. The design had to keep the common, light case fast without letting the heavy case block anything.
• Large cached payloads. A composed section — and certainly a full report — is a sizeable object. Redis is optimised for small values; these payloads were large enough that the cache needed a store suited to bigger objects.
• Unbounded historical ranges. The period selector runs from "this month" to a custom range covering a user's entire history — ten years or more. A full export over such a range cannot be guaranteed within five seconds.

Implementation Highlights

Read the Cache by Trying, Not by Checking

The obvious way to read a cache is to check whether the entry exists and then download it. But against Blob Storage, an existence check costs the same round-trip as the download itself — so checking first makes every cache hit pay twice. Instead we just download by key and treat a "not found" exception as a miss. A hit is one round-trip; a miss costs an exception we were going to pay for anyway.

public sealed class StatementCache
{
    private readonly BlobContainerClient _blobs;

    public StatementCache(BlobContainerClient blobs) => _blobs = blobs;

    // Key is the user ID (never the username), regulation, period and tab —
    // everything that changes the output, so identical requests share a blob.
    private static string KeyFor(StatementRequest r) =>
        $"{r.UserId}/{r.Regulation}/{r.Period}/{r.Tab}.bin";

    public async Task<StatementPart> GetOrAddAsync(
        StatementRequest request,
        Func<StatementRequest, Task<StatementPart>> compose,
        CancellationToken ct)
    {
        var blob = _blobs.GetBlobClient(KeyFor(request));
        try
        {
            // No ExistsAsync() first: that check is a whole round-trip, the
            // same cost as the download — so on a hit we'd pay for two.
            var hit = await blob.DownloadContentAsync(ct);
            return hit.Value.Content.ToObjectFromJson<StatementPart>();
        }
        catch (RequestFailedException ex) when (ex.Status == 404)
        {
            // Miss — compose once, then persist for the next reader.
            var fresh = await compose(request);
            await blob.UploadAsync(
                BinaryData.FromObjectAsJson(fresh), overwrite: true, ct);
            return fresh;
        }
    }
}

Compose Every Source in Parallel

A statement is stitched together from the SQL database — whose stored procedures read the main tables and, via a linked server, the linked databases — and from other services. The monolith read these in sequence; we read them concurrently, so a section is only as slow as its slowest source rather than the sum of all of them. A full PDF or Excel report runs its sections in parallel too — and any section already cached from on-screen browsing is reused, so an export after a browse only queries what is genuinely missing.

// One section, composed in parallel from two app-level sources: the SQL
// stored procedure (which itself reads the main tables AND, via a linked
// server, the linked databases) and the other microservices' APIs.
public async Task<StatementPart> ComposeAsync(
    StatementRequest r, CancellationToken ct)
{
    var sql     = _sql.LoadSectionAsync(r, ct);       // main DB + linked server
    var service = _api.GetReferenceDataAsync(r, ct);  // other microservices

    await Task.WhenAll(sql, service);
    return StatementPart.Merge(sql.Result, service.Result);
}

// A full report is every tab — so the tabs run in parallel, and cached
// tabs (e.g. from on-screen browsing) cost nothing to reuse.
public Task<StatementPart[]> ComposeReportAsync(
    StatementRequest r, CancellationToken ct) =>
    Task.WhenAll(AllTabs.Select(tab =>
        _cache.GetOrAddAsync(r with { Tab = tab }, ComposeAsync, ct)));

Generate in the Background, Race the Response Budget

Rather than decide up front whether a request is "big", every request is generated in the background and raced against the response budget — five seconds here, but read from central config (CCM) rather than hardcoded, so it can be retuned per environment without a redeploy. Beat the budget and the response is returned inline; miss it and the work simply keeps going, while the user is told the statement will arrive by email. The caller never blocks on a slow read, and nothing is thrown away — the same generation that missed the budget produces the emailed file.

public sealed class StatementResponder
{
    private readonly IGenerator   _generator;
    private readonly IEmailSender _email;
    private readonly ICcmSettings _ccm;

    public StatementResponder(
        IGenerator generator, IEmailSender email, ICcmSettings ccm)
    {
        _generator = generator;
        _email     = email;
        _ccm       = ccm;
    }

    public async Task<StatementResult> RespondAsync(
        StatementRequest r, CancellationToken ct)
    {
        // The budget is an operational knob, not a constant — read it from CCM
        // (central config) so it can be retuned per environment without a deploy.
        var budget = TimeSpan.FromSeconds(
            _ccm.GetDouble("Statement.ResponseBudgetSeconds"));

        var work = _generator.GenerateAsync(r, ct);

        // Beat the budget → return inline. Miss it → keep generating in the
        // background and email a link, so the caller never waits on a big read.
        if (await Task.WhenAny(work, Task.Delay(budget, ct)) == work)
            return StatementResult.Ready(await work);

        _ = work.ContinueWith(
            t => _email.SendDownloadLinkAsync(r, t.Result),
            TaskScheduler.Default);

        return StatementResult.Preparing(
            "Your statement is being prepared — we'll email a download link.");
    }
}

Related Case Studies