From Blocked Engineers to Parallel Velocity

The Problem

The standard development loop for robotics software looks deceptively simple: change an algorithm, flash the firmware, run the robot, observe the result, repeat. In practice, that loop is bottlenecked by one variable that appears in no specification document: access to physical hardware.

The client — a US company building next-generation autonomous systems — had a testing environment that had grown organically alongside their product. Early in the company's life, three physical test rigs were sufficient. As the engineering team scaled and algorithm development became parallel across multiple specialisations — path planning, sensor fusion, obstacle avoidance, multi-robot coordination — the rig count didn't scale with it.

The queuing system started as a shared calendar. Engineers booked hour-long slots. When contention grew, the slots became shorter and the handoffs more disruptive. An engineer investigating a subtle edge case in their path-planning algorithm would finish their hour and hand the rig to the next team — regardless of whether the investigation was complete. Context-switching mid-enquiry became the norm. A bug found in hour one was a bug that might not be reproduced until tomorrow's slot.

Multi-robot scenario testing existed in a different problem category entirely. Running a coordinated swarm test on physical hardware required setting up multiple rigs simultaneously, networking them, and hoping no single rig failure corrupted the shared run. In practice, serious multi-robot integration testing almost never happened — the logistics overhead made it impractical at the team's size and facility constraints. Algorithms were tested individually and then observed in production formation. Production was, in effect, part of the test environment.

The engineering organisation's throughput was materially lower than the headcount suggested. The bottleneck was not the engineers' ability to write algorithms — it was the physical infrastructure available to validate them. Adding more rigs had been modelled: the economics didn't hold. Each rig required calibration, maintenance, floor space, and a technician. And adding physical rigs wouldn't solve the multi-robot scenario problem at all — quantity was not the constraint.

The Architecture

The system has three distinct layers: a submission surface that engineers and CI pipelines interact with, an orchestration layer that enforces quotas and schedules work onto the cluster, and an execution layer where the simulations themselves run.

Scenarios are defined as YAML files checked into the same repository as the algorithm code. A scenario specifies the environment (map, physics seed, obstacle configuration), the robots participating (model, start position, algorithm variant to test), and the checkpoints that define pass/fail (no collision at 30 seconds; all robots at goal by 60 seconds). The YAML is the contract between the engineer and the harness — and it is also the replay artefact. Given the same YAML and seed, the simulation produces the same result every time.

The Orchestrator is the central authority. It receives a validated scenario from the API, checks the submitting team's remaining quota for the billing period, provisions a Kubernetes namespace for the scenario (isolating it completely from all other concurrent scenarios), and creates one Kubernetes Job per robot. Each Job runs the same simulation container image — the robot model and algorithm variant are injected as environment variables. The seed is injected identically across all robots in the scenario, which is what makes replay deterministic: the same inputs, the same sequence.

Within the scenario namespace, robots communicate over a Kubernetes Service — a simple TCP socket abstraction that the simulation engine uses to exchange position and state updates. This is the "scenario network" in the diagram. The namespace boundary means that a runaway robot in scenario A cannot flood the network for scenario B — they are as isolated as containers on different hosts.

Telemetry capture runs as a sidecar container in each robot's pod. Every position update, sensor reading, and decision event is written to a time-indexed stream in Azure Blob Storage, tagged with the scenario ID and robot ID. When a scenario fails at a checkpoint, an engineer can download the telemetry stream, point the replay tool at it alongside the original scenario YAML, and reproduce the failure frame-by-frame — without touching the cluster.

What We'd Do Differently

We accepted a deliberate fidelity trade-off at launch: the standard physics tier was fast (under two minutes to first results) but omitted some sensor noise characteristics that were present in the physical environment. For the first six months, this was the right call — engineers needed iteration speed, not physical accuracy, for the class of algorithms they were working on.

A year in, the obstacle avoidance team hit a category of edge cases that only manifested under realistic sensor noise. The high fidelity tier existed but hadn't been used much — and when the team switched to it, they discovered that some of their test assertions were written for standard-fidelity behaviour and gave false passes under higher fidelity. The fidelity tier choice was buried in the YAML and in no way forced teams to think about it before submitting.

If we were building this today, the fidelity tier would be a required field with no default — it would force an explicit choice at submission time, making the assumption visible. We'd also build the exact fidelity tier (full physical accuracy, slower) from day one, not as a retrofit, because retrofitting it required changes to how the simulation container was configured — changes that affected all tiers.

The multi-tenant network isolation worked well but cost us more engineering time than anticipated. Kubernetes namespace isolation does not automatically isolate network traffic — you need NetworkPolicy resources applied to every scenario namespace, and the policy templating had to be tested carefully to avoid both under-isolation (scenarios seeing each other's traffic) and over-isolation (robots within the same scenario unable to communicate). We'd invest in a dedicated namespace provisioner service from day one rather than generating the NetworkPolicy in the same path as the Job template.

If You're Solving This Today

In 2026 we'd look seriously at Azure Container Apps Jobs as a simpler alternative to raw AKS for the execution layer — the consumption-plan billing and built-in job lifecycle management remove a meaningful slice of the orchestration complexity we wrote ourselves. For the deterministic replay requirement, the current approach (seed + YAML → identical output) remains the right model, but we'd evaluate whether storing the full telemetry stream in Azure Data Explorer instead of Blob Storage would make post-run analysis meaningfully faster.