Benchmark evidence is workload-shaped

Evidence legend: Durable marks a measurement principle; Version-specific marks a pinned workload or harness; Experimental marks a comparison that needs controlled reruns.

A benchmark number answers a narrow question: how did this named workload behave under this harness? The runtime benchmark module makes that narrowness visible. It measures time, and its base class also records group and slot counts; it does not turn a synthetic tree into an application-wide frame-time promise (ComposeBenchmarkBase.kt#L52-L119).

Read the workload before the result

The meaningful unit is mechanism → workload → metric → control → limit. For example, an insertion optimization should be tested with insertManyGroups and insertAlternatingGroups, then checked against removals and reverse movement. A deep-tree result should be paired with breadth variants, not reported as “Compose is faster.”

Methodology changes the question

measureCompose measures initial setContent; measureRecompose composes once, mutates a model, sends snapshot notifications, advances the test scheduler, and checks that pending work is gone (ComposeBenchmarkBase.kt#L123-L207). Setup and reset are outside the measured section. The harness uses an Android activity, main-thread measurement, a test frame clock, and BenchmarkRule; results therefore depend on device, OS, artifact build, compiler/runtime revisions, warmup/calibration, and benchmark configuration. Group/slot counts are structural observations, not latency.

Experimental Comparing gap-buffer and link-buffer modes needs the same checkout, compiler, device, build type, workload, and controls. Run each mode separately; record timing distributions and group/slot counts. Do not combine a changed flag with changed source or report a single percentage from one phone.

Reproduce named workloads

On a connected Android device at the pinned checkout:

./gradlew :compose:runtime:runtime:compose-runtime-benchmark:cC \
  -Pandroid.testInstrumentationRunnerArguments.class=androidx.compose.runtime.benchmark.ComposeBenchmark#benchmark_05_Recompose_TenRect_Wide
./gradlew :compose:runtime:runtime:compose-runtime-benchmark:cC \
  -Pandroid.testInstrumentationRunnerArguments.class=androidx.compose.runtime.benchmark.ComposeBenchmark#benchmark_06_Recompose_TenRect_Narrow
./gradlew :compose:runtime:runtime:compose-runtime-benchmark:cC \
  -Pandroid.testInstrumentationRunnerArguments.class=androidx.compose.runtime.benchmark.SlotTableIntegrationBenchmark#reverseGroups
./gradlew :compose:runtime:runtime:compose-runtime-benchmark:cC \
  -Pandroid.testInstrumentationRunnerArguments.class=androidx.compose.runtime.benchmark.EffectsBenchmark

Controls: keep the device, ABI, build variant, compiler/runtime artifacts, test clock, method, and benchmark configuration fixed; add DeepTreeBenchmark, SiblingBenchmark, and SnapshotStateObserverBenchmark only when their mechanism is the question. Expected: the harness emits timing plus group/slot metrics for composition/recomposition fixtures. Limits: no command here produces a universal performance percentage, and this content edit did not run the upstream benchmarks.

Misconceptions and check

• “Fewer groups means a faster app.” It means fewer measured groups in that fixture; other work may dominate.
• “Narrow recomposition is a frame benchmark.” It is a controlled recomposition workload with a test clock, not a full device frame.
• “The roadmap or a benchmark name proves an optimization shipped.” Only the tested revision and release evidence can bound that claim.

Before comparing two implementations, write the control and the confounder beside the metric. If you cannot name both, you do not yet have a performance claim.

Evidence ledger

Freshness

Refresh when benchmark methods, metrics, test clocks, device/build configuration, or the slot/recomposition mechanisms change. Re-run paired controls before describing an optimization’s direction or size.