Comparing orchestration services for physical AI evals

TL;DR

SignalFlag is the fastest orchestration setup we tested for physical AI evals — because we optimized for it.

SignalFlag churned through 100 tests in ~28 minutes, compared with ~42 minutes on hosted GitHub, 52 minutes on a self-hosted setup and 36 minutes on AWS Batch. We have optimized around the image, data management and reuse patterns common to physical AI workflows.

Why eval orchestration matters

Offline testing in robotics involves computing evals for several familiar purposes:

• Evaluating model checkpoints
• Gating an engineer's pull request
• Evaluating large regression suites that run overnight

Overnight results need to be ready before dawn. PR tests need to have finished RTFN. For teams aiming to evaluate 100 or more model checkpoints a day, speed and cost matter throughout the day too.

These types of tests require heavy compute resources and are best suited to running at scale in the cloud or on-prem clusters.

We examined several “off the shelf” options for cloud orchestration of batches of tests in parallel, including GitHub Actions*, self-hosted GitHub runners*, AWS Batch and (our platform) SignalFlag for “fit” for robotics testing workloads. We used a replay workload at different scales as our benchmark.

Those differences matter to the engineer waiting on a change and we explore them in more depth in this blog. Reducing billable overhead can also reduce cost when compute is billed by the minute. They also raise a practical infrastructure question: how much work should a robotics team take on to home-roll a dependable testing service?

What does running an evaluation like this actually involve?

We ran a series of controlled experiments representative of physical AI workflows (see Appendix A for the experimental method). The same orchestration work applies to checkpoint comparisons, PR checks and regression suites.

SignalFlag returns results quickest (yay!)

The graph above compares SignalFlag with the two GitHub and AWS Batch workflows:

At ten tests, SignalFlag finished 1 minute 50 seconds sooner than hosted GitHub. At 50 tests, SignalFlag saved 12 minutes 42 seconds against hosted GitHub. At 100, it saved 14 minutes 4 seconds, a 34% reduction in elapsed time. Against a self-hosted configuration, the reduction was 24 minutes 4 seconds, or 46%.

For an engineer, the useful measure is the wait for a result they can act on. In the tested setup, SignalFlag shortened that wait while completing the metrics and result processing within the recorded time. SignalFlag also, for example, makes available evals of tests that have completed early.

Getting off to a fast start

A surprising amount needs to happen before the test container actually starts running, especially at the start of a batch of tests: an instance has to come up, the test image needs to be pulled, any test data needs copied across and staged and the test container needs to be started. The chart below compares these times across our cohort:

SignalFlag utilizes specialized AMIs, pre-caches the test data, and several other techniques to start as quickly as possible. SignalFlag does not bill for instance start times, just when work actually starts.

Scaling back down

After the final job completes, the median delay to the recorded shutdown marker was 1:34 for SignalFlag, 1:01 for AWS Batch and 2:04 for self-hosted GitHub. SignalFlag intentionally waits at the end of a test in order to better pick-up other work. We do not, however, bill for the instance shutdown period as we can quite often pick up other work.

Note: This is the baseline

These results are the baseline. Our forward-deployed engineers can apply the expertise behind these optimizations to a customer's own workload, identifying opportunities to reuse image layers, test data and model artifacts, customer parallelization compute workflows, etc. bringing those further speedups.

Cohort Comparisons

Speed is only part of the decision. Each setup below asks a different amount of your team, from running CI checks to building a full testing service.

GitHub-hosted runners are an accessible starting point for offline tests: most people are already running unit and (sometimes) integration tests there. They put execution beside the PR workflow and remove most cluster administration. The team still needs to arrange test selection, recording access, evaluation and a useful result summary. They suit teams whose immediate requirement is a manageable set of CI checks. This is also an expensive option, with high markups. The lack of larger GPU nodes and having important data out of band is disqualifying in many cases. Finally, as is also the case w/ self-hosted GitHub, evaluations still need to be implemented. Without processing metrics, comparing checkpoints and presenting the results, finishing the jobs still leaves engineers to assemble the answer themselves.

Self-hosted GitHub runners keep that workflow while adding control over hardware, networking and storage. They can make sense for private data, or specialized machines that the organization already operates. The additional responsibility is keeping usable runners available: updates, isolation, image cleanup, capacity, logs and recovery. GitHub maintains the Actions service; the customer maintains the runner machines and their software.

AWS Batch is an alternative foundation for a team building its own testing service. AWS supplies compute environments, scheduling queues, dependencies and retry policies. A team can start with a manifest, a container job and outputs in S3. An application needs to be built around those jobs to organize tests, manage data and artifacts, run metrics, connect results to PRs and distinguish a test failure from an infrastructure failure.

SignalFlag provides the test workflow around execution: organizing batches and tests, processing metrics and collecting results. A team can focus on defining useful metrics and connecting the outcome to engineering decisions. It gives up some of the freedom of a completely custom execution service in exchange for adopting those capabilities together. SignalFlag is available self-hosted, offering more flexibility in compute for very little infrastructure management.

Deployment and maintenance effort

These planning estimates are informed by our deployment experience and recent implementation benchmarks. Getting a pilot working is one thing, but turning it into a sturdy service that engineers trust takes additional work and ongoing maintenance.

The following are engineering planning estimates, assuming an experienced engineer (w/ Claude/Codex, of course), an existing account and network, a working container and the necessary “authority”. The shared-service column is total effort, including the pilot, and these ranges exclude the metrics development component. Dollar ballparks use an illustrative fully loaded rate of USD 100/hour, eight-hour days and five-day weeks; they cover engineering effort only, excluding compute, storage and platform charges*.

SignalFlag vastly reduces the amount of testing infrastructure the team needs to assemble. The integration still needs an owner: someone must maintain the API contract, metrics definitions and CI connection. With great power comes great responsibility, but relying on SignalFlag is significantly less than a whole home-rolled implementation (or being a super hero).

Appendix A: experimental method

Running these tests takes more than a fast replay stack (or high FPS in a sim). Robotics stacks are generally large docker images (and, increasingly, large weights files). Field recordings and sim assets are measured in gigabytes. Tests need specific machines, permissions and storage, and their outputs need sophisticated processing before an engineer can tell whether a change is good.

A useful evaluation batch starts with a versioned image (or images, depending on how a stack is built) and a set of tests. The system has to make both available to workers, distribute the tests, collect outputs, run evaluations and publish a result that can be traced back to the code or model version.

Note: A huge part of robotics evals is the actual metrics that get computed and we have lots to say about this, but here we really focus on the orchestration aspect.

For this experiment, we used a representative robotics docker image and a 4.6 GB MCAP per test (both on the lower end of size that we see w/ customers). Each replay was designed to last for five minutes at the level of compute specified, followed by some output handling. Holding replay time constant made the surrounding overhead visible: image preparation, data access, scheduling, container startup and result collection. This CPU experiment focused on orchestration overheads, independently of inference speed. The data management and reuse patterns that SignalFlag optimizes also matter on GPUs; the size of the savings depends on the workload.

Each experiment followed the same sequence:

1. Build one shared image and record when it becomes available.
2. Submit the test suite at a configured concurrency target.
3. Prepare the image and recording data, then execute each test.
4. Collect outputs and finish evaluation and result publication.

The shared image build happens once per workflow, so is irrelevant in the comparison. Both GitHub variants include a 30s evaluation allowance since SignalFlag runs its actual evals processing by default and it’s a pain to turn off*.

We ran experiments with 10, 25, 50, 100, and 250 tests in total (drops in the ocean at self-driving scale, but representative of what we see in many robotics orgs). Each ran with an identical sub-linear concurrency cap (which maxed out at 47 for the largest test batch—if you want you can try guess the equation we used w/ some high school math).

Robotics eval orchestration FAQ

What is eval orchestration in physical AI?

It is the workflow that turns a set of tests into an answer an engineer can use: make the right code, model and data available, distribute tests across machines, collect outputs, compute metrics and publish results tied to the version under test. Finishing containers is only part of the job. The useful outcome is knowing how a checkpoint or code change affects behavior, whether a PR passes its checks, and where regressions appear.

What did the benchmark show?

For the tested 100-test replay workload, SignalFlag finished in about 28 minutes, compared with 36 minutes on AWS Batch, 42 minutes on GitHub-hosted runners and 52 minutes on self-hosted GitHub runners. That is a 34% shorter wait than hosted GitHub and 46% shorter than the self-hosted configuration. These results describe the workload and configurations we tested; the size of the savings depends on the deployment.

Why can orchestration make such a difference?

The replay itself is only part of the elapsed time. Machines must start, large images and recordings must become available, tests must be scheduled, and outputs must be processed. SignalFlag optimizes those steps around the image, data and reuse patterns common to physical AI. In this experiment, replay duration was held constant so those surrounding costs were visible. The speedup came from getting the whole workflow through faster.

When do GitHub Actions or AWS Batch make sense?

GitHub-hosted runners are an accessible starting point for manageable CI checks. Self-hosted runners add control over hardware, networking and private data, along with responsibility for the machines. AWS Batch supplies a foundation for teams building their own testing service. In each case, the team still needs to connect execution to test selection, data and artifact management, metrics and useful results. SignalFlag provides the testing workflow around execution, reducing the infrastructure the team needs to assemble and maintain with the same flexibility as AWS Batch or GHA.

How much engineering effort does a reliable testing service take?

Our planning estimates are 4–6 engineer-weeks for GitHub-hosted runners, 4–8 for autoscaling self-hosted runners and 6–12 for vanilla AWS Batch. Those totals include a pilot but exclude metrics development, and assume an experienced engineer, an existing account and network, a working container and the necessary permissions. Ongoing ownership is additional. With a provisioned SignalFlag backend, we estimate under one engineer-week for the integration; the effort table gives the full breakdown.

Do the measured speedups apply to GPU workloads?

This was a CPU benchmark designed to isolate orchestration overhead from inference speed. GPU workflows also need images, weights, recordings and results handling, so the same optimization opportunities matter. The benchmark does not establish a GPU speedup: the savings depend on the workload, compute configuration and opportunities for reuse.

Does SignalFlag replace our CI system or define our metrics?

SignalFlag connects to the existing CI workflow and handles test orchestration, metrics processing and result collection. The team still defines what good behavior means, maintains its metric definitions and owns the integration. SignalFlag makes it extremely easy to define metrics. The aim is to spend that effort on evaluating the robotics system, with much less infrastructure to assemble and maintain.