# Reproducing a LeRobot regression with TraceML

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%3E%5C
**Category:** Show and Tell\
**Created:** [September 7, 2026, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066 "2026-09-07T15:19:49Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhinavsriva](https://avatars.discourse-cdn.com/v4/letter/a/51bf81/32.png) [@abhinavsriva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhinavsriva)%5C
**Post date:** [September 7, 2026, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F1 "2026-09-07T15:19:49Z")

</div>

While reproducing a known LeRobot training regression, I found a pattern that the average timing alone doesn’t show: most DataLoader fetches were quick, but recurring fetches took over five seconds.

I maintain [TraceM](https://github.com/traceopt-ai/traceml/)L, an open-source PyTorch training diagnosis tool. We are building toward more automatic performance regression diagnosis: when training gets slower after a change, help identify where the extra time went. Today, you instrument the training loop, record two runs, and use `traceml compare` to compare timings and bottleneck diagnoses.

For this case study, I compared LeRobot before and after [upstream fix #2408](https://github.com/huggingface/lerobot/pull/2408). The LeRobot contributors found and fixed the bug; this experiment measures its effect.

The workload was ACT, 200 steps per run, on one NVIDIA L4. Both revisions used the same dataset, training settings and TraceML instrumentation in the Accelerate-based loop. The second pair reversed execution order:

| Metric | Pair 1: broken → fixed | Pair 2: broken → fixed |
| --- | --- | --- |
| Step time | 1496.9 → 116.8 ms | 1542.7 → 117.2 ms |
| DataLoader fetch, CPU | 1375.7 → 1.5 ms | 1427.5 → 1.3 ms |
| Compute | 115.9 → 110.5 ms | 110.4 → 111.1 ms |

TraceML reported **INPUT-BOUND** → **COMPUTE-BOUND** in both pairs. The recurring input stalls disappeared after the fix; compute stayed around 110–116 ms on this host. Exact timings vary across machines, and startup can affect the first pair. Inspect both orders before attributing a compute change to the fix.

 ![lerobot_input_wait](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/5/55196d730bbca0c174cccec38fcd53e591727159.png)

**To try it:** use Linux x86-64, Git, Python 3.10 with venv support, internet access and one NVIDIA GPU with a driver supporting CUDA 12.8. Ubuntu may need `sudo apt-get install python3.10-venv` first.

```bash  
git clone [GitHub - traceopt-ai/traceml: Open-source performance diagnostics for PyTorch training runs. · GitHub](https://github.com/traceopt-ai/traceml.git)  
cd traceml  
bash examples/advanced/lerobot\_v3\_image\_regression/run\_reproduction.sh  
```

The runner sets up its environment, downloads the dataset, runs one pair and prints the comparison location. Add `–pairs 2` for both orders.

If a dataset, dependency or configuration change has slowed your own PyTorch training, try the [TraceML quickstart](https://github.com/traceopt-ai/traceml#quickstart). Share the before/after comparison and whether it helped you find where to investigate. Looking for feedback.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [September 7, 2026, 3:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F2 "2026-09-07T15:23:43Z")

</div>

Unlisted on September 7, 2026, 3:23pm UTC



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [September 7, 2026, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F3 "2026-09-07T15:37:37Z")

</div>

Listed on September 7, 2026, 3:37pm UTC



---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 8, 2026, 4:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F4 "2026-09-08T04:45:06Z")

</div>

For now, I tried a quick reproduction on Colab:

* * *

For this pinned case, yes: the compact before/after comparison did point me toward the right subsystem.

I ran the merged [LeRobot regression case study](https://github.com/traceopt-ai/traceml/tree/107432d17c45f9c28e9704db5e1d3eba3cb616c3/examples/advanced/lerobot_v3_image_regression) on a Colab T4, keeping the historical LeRobot revisions and `datasets==4.1.1`, and ran two pairs with the second pair reversing the order.

| Metric | Pair 1: broken → fixed | Pair 2: broken → fixed |
| --- | --- | --- |
| TraceML diagnosis | INPUT-BOUND → COMPUTE-BOUND | INPUT-BOUND → COMPUTE-BOUND |
| Step time | 5337.15 → 301.96 ms (-94.34%) | 4845.45 → 307.77 ms (-93.65%) |
| Input wait | 4946.33 → 4.50 ms (-99.91%) | 4479.50 → 6.46 ms (-99.86%) |
| Compute | 372.82 → 289.91 ms (-22.24%) | 351.09 → 293.80 ms (-16.32%) |

So on this host too, the obvious first place to investigate from the `traceml compare` result would have been the input side. About 98% of the total step-time reduction in both pairs is accounted for by the reduction in input wait.

I put the executed two-pair notebook here, including the environment, reproduction code, comparison outputs, and the per-step analysis:

[Executed Colab T4 reproduction notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/TraceML/PUBLIC_TraceML_LeRobot_Regression_Colab_T4_executed_clean.ipynb)

The part I found especially interesting was the **shape** of the input wait, though.

On the broken runs, the median fetch was only about **2–3 ms** , while p95 was about **19–20 seconds**. In other words, the loader was not uniformly taking ~5 seconds per fetch: most fetches were fast, while a minority of very large stalls dominated the run-level average.

That makes the existing `INPUT-BOUND` diagnosis look useful to me for the first-level question — _where did the extra wall time go?_ — but it also made me wonder whether a small amount of temporal-tail information might be useful in the compact regression workflow when the distribution is this skewed.

The case-study analysis notebook already looks at per-step timing and summary statistics such as median/p95, while [`traceml compare`](https://github.com/traceopt-ai/traceml#compare-runs) is deliberately a compact comparison of the run summaries. So I do not mean turning `compare` into a full profiler, or necessarily adding a particular p95/p99 schema. Even something that makes “mostly fast with intermittent large stalls” distinguishable from “uniformly slower input” could potentially help decide what to inspect next.

The other thing I would preserve is the warning about absolute numbers. The high-level result seems much more portable than the timings themselves. The original L4 run, the separate clean-box L4 reproduction in [TraceML PR #450](https://github.com/traceopt-ai/traceml/pull/450), and this T4 run all reach `INPUT-BOUND → COMPUTE-BOUND` with input wait collapsing by roughly 99.9%, but the absolute step times and even the secondary compute behavior differ substantially across hosts.

In particular, my T4 run still showed a ~16–22% compute reduction in both orders. I would **not** attribute that to the LeRobot dataset fix from this experiment alone. The input-wait reduction overwhelmingly explains the overall speedup, and the clean-box L4 experiment in #450 is a good example of why reversed-order runs matter: a large apparent compute change in its cold first pair disappeared once the order was reversed and the system was warm.

So the part I would treat as robust here is:

> the regression was overwhelmingly input-side, and the compact comparison identified that correctly;

rather than:

> every phase should reproduce the same absolute before/after timing on another machine.

> **Colab T4 reproduction details**
>
> The run used:
> 
> - TraceML: `107432d17c45f9c28e9704db5e1d3eba3cb616c3`
> - LeRobot broken: `f6b16f6d97155e3ce34ab2a1ec145e9413588197`
> - LeRobot fixed: `67a6c8dfef90351830ad02afc5bf1bd299f9a521`
> - dataset revision: `13e0d3bff90f02ec417761b58199eb1d33a72efb`
> - `datasets==4.1.1`
> - Python 3.10.21
> - PyTorch 2.7.1 + CUDA 12.8
> - Accelerate 1.11.0
> - ACT
> - 200 steps/run
> - batch size 8
> - `num_workers=4`
> - seed 1000
> - Tesla T4, 15 GB
> - 2 logical CPUs
> - two A/B pairs, with pair 2 reversed
> 
> I made one acquisition-only change for Colab: the LeRobot Git checkout used `--filter=blob:none` to avoid transferring unnecessary blobs. The selected revisions, dataset, workload, TraceML instrumentation, and comparison logic were unchanged.
> 
> The historical dependency pin seems important here. This is a reproduction of a particular old regression, not a benchmark of current LeRobot or current Hugging Face Datasets.
> 
> PR #450 also ended up recording CPU information, the resolved package set, and the workload settings after its clean-box review, which seems particularly useful for an input-bound case where host CPU capacity can materially change the absolute numbers.

> **What the raw input-wait distribution looked like**
>
> Here are the input-wait statistics from the T4 run:
> 
> | Run | Mean | Median | p95 | p99 | Max | \>5 s |
> | --- | --- | --- | --- | --- | --- | --- |
> | Pair 1 broken | 4946.17 ms | 2.73 ms | 19751.80 ms | 21874.32 ms | 26747.25 ms | 64/200 |
> | Pair 1 fixed | 4.50 ms | 0.37 ms | 2.77 ms | 6.80 ms | 751.85 ms | 0/200 |
> | Pair 2 broken | 4479.43 ms | 2.29 ms | 18904.92 ms | 20065.99 ms | 24583.58 ms | 52/200 |
> | Pair 2 fixed | 6.43 ms | 0.37 ms | 2.66 ms | 3.58 ms | 1149.32 ms | 0/200 |
> 
> This is the main reason I think the temporal shape is useful information.
> 
> For Pair 1, for example, the mean says roughly 4.95 seconds of input wait, but the median says 2.73 ms. Those describe very different-looking failure modes even though the same total throughput can still correctly be classified as input-bound.
> 
> There is also visible recurring structure in the slow steps, but I would stop at that observation. With `num_workers=4`, worker/prefetch behavior is one plausible branch, but this trace alone does not distinguish that from decoding, storage/cache behavior, scheduling, or another input-pipeline mechanism. I would not infer the implementation root cause from the cadence alone.
> 
> If that implementation-level question ever became important, a small `num_workers=0/1/4` DataLoader-only comparison would be a cheap discriminator. It does not seem necessary for the higher-level regression-diagnosis question here.

> **Two optional directions for future regression cases**
>
> One low-cost pattern that seems complementary to this style of diagnosis is a **counterfactual input control**.
> 
> For example, NVIDIA’s [DALI Loader Evaluator](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/plugins/pytorch_data_loader_evaluator.html) measures a real PyTorch DataLoader and then replays cached batches to approximate a near-zero-overhead loader. For an unknown regression, a workflow could conceptually be:
> 
> 1. use `traceml compare` to identify where the extra time moved;
> 2. if it says `INPUT-BOUND`, inspect whether the slowdown is uniform or stall-heavy;
> 3. if stronger causal confirmation is useful, compare against cached/synthetic same-shape input;
> 4. only then reach for a heavier profiler if the remaining question requires it.
> 
> I do not mean TraceML should depend on DALI — just that cached replay is an existing example of a relatively cheap counterfactual test for “is the real input path actually limiting training?”
> 
> There is also a newer LeRobot regression that might make an interesting second ground-truth case because it has the same broad symptom class but a different mechanism.
> 
> The current reproduction is tied to the historical `datasets==4.1.1` behavior around [LeRobot PR #2408](https://github.com/huggingface/lerobot/pull/2408). Later, the relative behavior of row-first/column-first access changed with newer `datasets` versions; [#3523](https://github.com/huggingface/lerobot/issues/3523) documents that transition.
> 
> A later delta-query problem, discussed in [#2895](https://github.com/huggingface/lerobot/issues/2895) and concretely reproduced with image-dtype datasets in [#4392](https://github.com/huggingface/lerobot/issues/4392), could cause numeric delta queries to decode image columns that were not actually needed. The merged fix in [PR #4314](https://github.com/huggingface/lerobot/pull/4314) instead routes those queries through cached single-column `select_columns` views.
> 
> That seems potentially useful as another validation case because it is still an input-side training regression, but it is not simply a rerun of the #2408 mechanism.
> 
> It also reinforces a useful scope warning for this case study: I would keep the current result described as a **pinned historical reproduction** , rather than generalizing the old “column-first is faster” result into a rule for current `datasets` versions.

Overall, this case did answer the question I would want the lightweight comparison to answer first: **the extra training time moved into the input path, and that is where I should investigate.**

The T4 run mostly added a second piece for me: once the high-level attribution is known, a small amount of temporal-shape information can materially change what “input-bound” looks like — in this case, from an apparent multi-second loader to mostly millisecond fetches interrupted by very large stalls.

---

<div class="post-metadata">

**Author:** ![abhinavsriva](https://avatars.discourse-cdn.com/v4/letter/a/51bf81/32.png) [@abhinavsriva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhinavsriva)%5C
**Post date:** [September 8, 2026, 7:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F5 "2026-09-08T07:35:19Z")

</div>

Thanks for running and sharing the executed notebook. This is useful independent validation of the input-side diagnosis.

Yes, the absolute timings and compute changes shouldn’t be generalized, and the mean alone hides the difference between the slow input and occasional large stalls. I will keep that distinction in mind for the compact comparison.

---

<div class="post-metadata">

**Author:** ![Sattyam](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sattyam/32/62925_2.png) [@Sattyam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSattyam)%5C
**Post date:** [September 9, 2026, 4:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F6 "2026-09-09T16:07:13Z")

</div>

The point about pinning exact revisions for both TraceML and LeRobot, not just LeRobot as a floating dependency, matches something I have had to be strict about for a different reason: we pin lerobot==0.5.1 exactly rather than a floating range, because the per-step rollout logic in our LeRobot adapter was read off that specific version’s evaluator, and 0.6.0 moved import paths and changed extras and torch floors enough that bumping the pin needs a re-read of the call signatures, not just a version-string edit.

Reproducibility work like this makes the cost of an unpinned dependency legible in a way “it broke in prod” reports usually do not.

---

<div class="post-metadata">

**Author:** ![alikhaters](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alikhaters/32/61986_2.png) [@alikhaters](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falikhaters)%5C
**Post date:** [September 14, 2026, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F7 "2026-09-14T11:06:40Z")

</div>

Reversing execution order is a good control here. One additional view that would help is the distribution of individual fetch times rather than only the average, with the recurring slow fetches aligned to worker resets, cache misses, or storage reads. That would make the diagnosis portable across machines where the absolute timings differ.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 15, 2026, 2:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F8 "2026-09-15T02:33:05Z")

</div>

For now, I tried narrowing this down a little further:

* * *

Following the suggestion above to correlate the slow fetches with worker/cache/storage behavior, I ran two small DataLoader-only controls against the same historical regression.

The short version is:

> **the recurring stall cadence looks genuinely coupled to the DataLoader worker pool, but not to ordinary worker restarts and not primarily to FIFO return ordering.**

The expensive work still appears to be on the historical dataset-access side. The worker configuration seems to determine **how that cost becomes visible over time**.

The first control was simply to keep the historical broken stack fixed and vary `num_workers`.

| Condition | What happened |
| --- | --- |
| broken, `num_workers=0` | essentially every fetch was slow |
| broken, `num_workers=1` | large stalls essentially every step |
| broken, `num_workers=2` | large stalls recurred about every 2 steps |
| broken, `num_workers=4` | large stalls recurred every 4 steps |
| fixed, `num_workers=4` | no \>5 s stalls |

I ran the broken sweep in both directions (`4→2→1→0` and `0→1→2→4`) to make order/cache effects a little less likely to fool me.

For the runs with enough \>5 s stalls:

- `workers=1` → median stall gap ≈ **1**
- `workers=2` → median stall gap = **2**
- `workers=4` → median stall gap = **4**
- in the 2-worker and 4-worker runs, the observed large-stall gaps were exact multiples of the worker count

So the four-step pattern from the earlier T4 reproduction no longer looks like an arbitrary periodicity.

The second control was useful because it falsified one of my own first guesses.

I had suspected that ordered DataLoader return might be producing a simple head-of-line effect: one slow required batch blocks the consumer while later batches become ready behind it, producing one long wait followed by several nearly instantaneous fetches.

PyTorch does normally return multiprocessing DataLoader batches in order, and the pinned PyTorch 2.7.1 used here already has both [`prefetch_factor`](https://github.com/pytorch/pytorch/blob/v2.7.1/torch/utils/data/dataloader.py) and `in_order=True` by default.

So I held everything else fixed at:

```plaintext
num_workers = 4
prefetch_factor = 2
same historical broken LeRobot revision
same datasets==4.1.1 environment
same deterministic sampler stream

```

and compared:

```plaintext
in_order=True
vs.
in_order=False

```

I also recorded the original sampler-batch rank of every returned batch, so I could verify that `in_order=False` was actually allowing later-ready batches to overtake earlier ones.

It did.

Across the two unordered runs I observed **14 overtaking deliveries** , while the ordered runs had none.

But the large-stall cadence remained:

```plaintext
ordered:
    stall steps = 0, 4, 8, 12

unordered:
    stall steps = 0, 4, 8, 12

```

Both conditions still had:

```plaintext
4 stalls > 5 s out of 16 deliveries
median large-stall gap = 4

```

That seems like a useful negative result.

**Removing FIFO enforcement changed the delivery order, but it did not remove the four-worker cadence.**

So I would now weaken my earlier head-of-line interpretation substantially.

There was still some effect on the shape of the tail:

| Broken condition | mean fetch | p95 | \>5 s stalls | stall gap |
| --- | --- | --- | --- | --- |
| ordered, pass 1 | ~4.71 s | ~19.28 s | 4 | 4 |
| unordered, pass 1 | ~4.31 s | ~16.67 s | 4 | 4 |
| unordered, pass 2 | ~4.38 s | ~16.43 s | 4 | 4 |
| ordered, pass 2 | ~4.62 s | ~17.75 s | 4 | 4 |

So ordering can redistribute some of the waiting and change which batch exposes it, but it does **not** appear sufficient to explain the recurring large stalls.

The interpretation I currently find most consistent with the observations is therefore closer to:

```plaintext
expensive historical dataset access
        ↓
substantial CPU-side producer work
        ↓
multi-worker DataLoader / prefetch pool
        ↓
worker-count-coupled availability pattern
        ↓
consumer sees ~1 / ~2 / ~4 step stall cadence
depending on worker count

```

rather than:

```plaintext
FIFO ordering
        ↓
four-step stalls

```

In other words, I think it is useful to separate two questions:

1. **Why is producing these batches expensive?**
2. **Why does that expense appear to the training loop in this particular temporal pattern?**

For this historical case, the first question was already answered upstream: [LeRobot PR #2408](https://github.com/huggingface/lerobot/pull/2408) traced the training slowdown to `_query_hf_dataset` and changed the dataset access path. I am not trying to rediscover or reassign that root cause.

The new part here is mostly about the second question.

A few other observations helped narrow it further:

- I saw **no worker PID-set changes** during any of the measured broken runs, so ordinary worker restart/reset does not look like a good explanation for recurring within-run stalls.
- Slow intervals contained much more aggregate worker CPU time than fast intervals.
- The median worker physical `read_bytes` during the slow intervals was 0, and the median major-page-fault count was also 0.
- The known-fixed revision with four workers again had **no \>5 s stalls**.

So, from the counters I collected, the evidence points more strongly toward **CPU-side dataset processing** than toward recurring physical storage reads.

I would not stretch that farther than the measurements allow: `read_bytes == 0` does not prove that every cache/memory effect is irrelevant, I did not measure CPU hardware-cache misses, and I did not independently isolate the OS scheduler.

One other nice sanity check was that removing ACT compute did not destroy the original heavy-tail shape.

The previous full ACT T4 reproduction looked approximately like:

```plaintext
broken pair 1:
    mean input wait ≈ 4.95 s
    median ≈ a few ms
    p95 ≈ 19.75 s

broken pair 2:
    mean input wait ≈ 4.48 s
    median ≈ a few ms
    p95 ≈ 18.90 s

```

The short DataLoader-only four-worker control produced roughly:

```plaintext
pass 1:
    mean ≈ 4.74 s
    median ≈ 1.8 ms
    p95 ≈ 19.58 s

pass 2:
    mean ≈ 4.41 s
    median ≈ 4.0 ms
    p95 ≈ 17.23 s

```

I would **not** call those numbers a strict numerical reproduction—the short probe deliberately uses artificial consumer pacing rather than real ACT compute—but qualitatively the same unusual shape survives:

> most delivered batches are immediately available, while a small regularly spaced subset dominates the wall time.

For reference, the earlier full T4 reproduction is still here:

[Executed Colab T4 reproduction notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/TraceML/PUBLIC_TraceML_LeRobot_Regression_Colab_T4_executed_clean.ipynb)

There is also an important host-specific caveat: this Colab T4 instance exposed only **2 logical CPUs**.

So I would not infer from this that, for example, `num_workers=2` is generally preferable to 4. Four CPU-heavy workers contending on a two-CPU host can obviously affect absolute stall duration.

What seems much more reusable to me is the **structural change** under the controlled variation:

```plaintext
worker count changes
        ↓
stall cadence changes with it

```

rather than the absolute number of seconds.

This actually reinforces my impression of where a tool like [TraceML](https://github.com/traceopt-ai/traceml) is useful.

The compact diagnosis answered the first important question correctly:

```plaintext
Where did the extra training time go?

INPUT-BOUND

```

Then the _shape_ of the input timing suggested which cheap experiment to try next:

```plaintext
INPUT-BOUND
    ↓
uniformly slow, or heavy-tailed?
    ↓
heavy-tailed
    ↓
random, or temporally structured?
    ↓
structured
    ↓
vary one cheap parameter
    ↓
cadence follows num_workers

```

I do not think TraceML needs to become a full DataLoader/PyTorch profiler to make this useful.

If you are deciding what belongs in a compact comparison, I would personally find a very small amount of distribution/temporal evidence valuable—enough to distinguish something like:

```plaintext
uniformly slow input

```

from:

```plaintext
mostly-fast input with recurring large stalls

```

because those two cases suggest rather different next experiments even if their **mean input wait is identical**.

The current TraceML README already points an `INPUT-BOUND` diagnosis toward workers, transforms, tokenization, collation, or storage. This case makes me think the useful extra layer may be less about automatically naming the final root cause, and more about preserving enough timing shape to select a cheap discriminator.

> **Exact control setup and the full worker-count result**
>
> The historical pins remained the same as in my earlier reproduction:
> 
> - TraceML reference revision: `107432d17c45f9c28e9704db5e1d3eba3cb616c3`
> - LeRobot broken revision: `f6b16f6d97155e3ce34ab2a1ec145e9413588197`
> - LeRobot fixed revision: `67a6c8dfef90351830ad02afc5bf1bd299f9a521`
> - dataset: `imstevenpmwork/aloha_sim_transfer_cube_human_image`
> - dataset revision: `13e0d3bff90f02ec417761b58199eb1d33a72efb`
> - `datasets==4.1.1`
> - batch size: 8
> - action horizon: 100
> - seed: 1000
> - `prefetch_factor=2` when multiprocessing was enabled
> 
> The broken worker-count sweep was run twice:
> 
> ```plaintext
> pass 1: 4 → 2 → 1 → 0
> pass 2: 0 → 1 → 2 → 4
> 
> ```
> 
> The probe recorded blocking `next(dataloader)` time, delivered worker ID/PID, worker PID-set changes, worker CPU time, physical read counters, page faults and system I/O counters.
> 
> Summary:
> 
> | revision | pass | workers | median | p95 | \>5 s | median \>5 s gap |
> | --- | --- | --- | --- | --- | --- | --- |
> | broken | 1 | 0 | 6248 ms | 6845 ms | 32 | 1 |
> | broken | 1 | 1 | 6027 ms | 6923 ms | 30 | 1 |
> | broken | 1 | 2 | 4416 ms | 8292 ms | 15 | 2 |
> | broken | 1 | 4 | 1.8 ms | 19581 ms | 8 | 4 |
> | broken | 2 | 0 | 6219 ms | 6722 ms | 32 | 1 |
> | broken | 2 | 1 | 5884 ms | 6418 ms | 32 | 1 |
> | broken | 2 | 2 | 3785 ms | 8924 ms | 16 | 2 |
> | broken | 2 | 4 | 4.0 ms | 17228 ms | 8 | 4 |
> | fixed | 1 | 4 | 0.39 ms | 1.55 ms | 0 | — |
> 
> A schematic view of the result is approximately:
> 
> ```plaintext
> workers=1:
> S S S S S S S S ...
> 
> workers=2:
> S . S . S . S . ...
> 
> workers=4:
> S . . . S . . . S . . . ...
> 
> S = >5 s fetch
> 
> ```
> 
> That simple scaling relationship is why I now think the worker-pool coupling is more interesting than the original observation that the four-worker run happened to stall every fourth delivery.

> **The \`in\_order\` control and why I changed my interpretation**
>
> For the second control I fixed:
> 
> ```plaintext
> num_workers=4
> prefetch_factor=2
> 
> ```
> 
> and compared `in_order=True` against `in_order=False`.
> 
> The sampler permutation was fixed and every returned batch carried its original sampler-batch rank.
> 
> That matters because it lets us distinguish:
> 
> ```plaintext
> I requested unordered behavior
> 
> ```
> 
> from:
> 
> ```plaintext
> batches actually overtook one another.
> 
> ```
> 
> They did overtake one another.
> 
> The result was:
> 
> | condition | mean | median | p95 | \>5 s | stall gap | overtakes |
> | --- | --- | --- | --- | --- | --- | --- |
> | broken pass 1, ordered | 4708 ms | 0.37 ms | 19282 ms | 4 | 4 | 0 |
> | broken pass 1, unordered | 4312 ms | 423 ms | 16667 ms | 4 | 4 | 9 |
> | broken pass 2, unordered | 4377 ms | 656 ms | 16432 ms | 4 | 4 | 5 |
> | broken pass 2, ordered | 4616 ms | 1.56 ms | 17750 ms | 4 | 4 | 0 |
> | fixed, ordered | 53 ms | 0.37 ms | 211 ms | 0 | — | 0 |
> 
> The important invariant was:
> 
> ```plaintext
> ordered:
> 0, 4, 8, 12
> 
> unordered:
> 0, 4, 8, 12
> 
> ```
> 
> So my earlier thought that FIFO/head-of-line blocking might be the primary explanation for the four-step cadence did not survive the control.
> 
> PyTorch 2.7.1 defines [`in_order=False`](https://github.com/pytorch/pytorch/blob/v2.7.1/torch/utils/data/dataloader.py) as disabling FIFO enforcement for multiprocessing DataLoaders. The same source also documents `prefetch_factor=2` as two prefetched batches per worker by default when `num_workers > 0`.
> 
> Because actual overtaking happened here, but the four-step cadence survived, I think the safer conclusion is:
> 
> > FIFO ordering can alter the presentation of the wait, but the recurring cadence is more fundamental to the worker/prefetch producer side than to ordered return itself.
> 
> I also would not interpret the earlier observation that the slow returned batches appeared to belong to “worker 0” as evidence that worker 0 itself was defective.
> 
> Once FIFO was disabled, the worker identities associated with the slow deliveries changed, while the four-step cadence remained.
> 
> So the more useful invariant was **worker-width cadence** , not a particular worker identity.
> 
> `in_order=False` is only a diagnostic control here. PyTorch explicitly warns that disabling ordered delivery can reduce reproducibility and, with imbalanced data, can skew the data order seen by training. I would not turn this experiment into a recommendation to train that way.

> **Historical LeRobot context and why I would keep this version-pinned**
>
> I think it is important not to turn this historical reproduction into a general claim that one Hugging Face Dataset access direction is always better.
> 
> The original [LeRobot PR #2408](https://github.com/huggingface/lerobot/pull/2408) was merged in November 2025 and fixed the specific historical regression used for this TraceML case study.
> 
> But the story around `_query_hf_dataset` continued evolving.
> 
> For example:
> 
> - [#2549](https://github.com/huggingface/lerobot/issues/2549) later reported a \>20× slowdown from the column-first path on another dataset/setup.
> - [#2895](https://github.com/huggingface/lerobot/issues/2895) reported that multi-index queries used with `delta_timestamps` could be about 40× slower with direct indexing than with `select()`.
> - [#3523](https://github.com/huggingface/lerobot/issues/3523) showed that the relative performance changed again across `datasets` versions; with `datasets>=4.4.0`, a hoisted row-first access pattern was around 3.5–3.7× faster than the measured column-first path in that microbenchmark.
> 
> So I think the reusable lesson is not:
> 
> ```plaintext
> column-first is faster
> 
> ```
> 
> or:
> 
> ```plaintext
> row-first is faster
> 
> ```
> 
> but something more conditional:
> 
> ```plaintext
> access pattern
> × transform semantics
> × datasets version
> × requested columns/indices
> 
> ```
> 
> can materially change the input cost.
> 
> There is also a newer, distinct example in [#4392](https://github.com/huggingface/lerobot/issues/4392): a numeric delta query on an image-dtype dataset can end up formatting/decoding image columns that the query does not need, saturating DataLoader workers and making training CPU-bound.
> 
> That is not the same mechanism as this historical `datasets==4.1.1` reproduction, but it is an interesting future validation case for the same higher-level diagnosis:
> 
> ```plaintext
> same symptom:
> INPUT-BOUND
> 
> different implementation cause
> 
> ```
> 
> At the time I checked, [PR #4314](https://github.com/huggingface/lerobot/pull/4314) is still open. It proposes answering delta queries through cached single-column `select_columns` views, specifically to avoid feeding unrelated image columns through the transform. The reported benchmark there is very large, but since it is still an open PR I would treat it as a proposed current direction rather than a merged/current baseline.
> 
> That version dependence is why I would keep the claims in this thread explicitly attached to the pinned historical LeRobot / `datasets==4.1.1` case.

> **What I would test next — only if someone wants the lower-level mechanism**
>
> For the practical TraceML question, I think I would stop here.
> 
> We already got useful discrimination from two small controls:
> 
> ```plaintext
> worker count
> → cadence follows worker width
> 
> FIFO disabled
> → batches really overtake
> → cadence remains
> 
> ```
> 
> If someone specifically wanted to continue into the internals of the worker/prefetch behavior, the next cheap variable I would try is probably:
> 
> ```plaintext
> prefetch_factor=1
> vs.
> prefetch_factor=2
> 
> ```
> 
> while keeping `num_workers=4`.
> 
> That could help separate effects of worker width from queue/prefetch depth.
> 
> Beyond that, I would probably stop using timing-only inference and move to a more explicit profiler/queue-level observation if the exact internal mechanism really mattered.
> 
> But I do not think that extra depth is necessary for the original performance-diagnosis question.

So my updated reading of the whole case is roughly:

```plaintext
TraceML:
    correctly points to INPUT-BOUND
        ↓
per-step data:
    shows a strongly heavy-tailed / periodic input pattern
        ↓
cheap worker-count control:
    shows the cadence follows worker count
        ↓
PID + I/O counters:
    weaken worker-restart and recurring-physical-I/O explanations
        ↓
in_order control:
    produces real batch overtaking,
    but the four-step cadence survives
        ↓
current interpretation:
    historical CPU-heavy dataset access is the root cost;
    worker-pool behavior shapes the temporal cadence;
    FIFO ordering changes the presentation somewhat,
    but is not the main explanation

```

For me that is a useful outcome for the original TraceML question.

The compact diagnosis did not need to know the final low-level mechanism in advance. It only needed to point to the correct subsystem, while preserving enough evidence that the next cheap experiment was obvious.

That seems like a good boundary for a lightweight diagnostic tool.

---

<div class="post-metadata">

**Author:** ![abhinavsriva](https://avatars.discourse-cdn.com/v4/letter/a/51bf81/32.png) [@abhinavsriva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhinavsriva)%5C
**Post date:** [September 16, 2026, 12:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F9 "2026-09-16T12:40:02Z")

</div>

Thanks, John. The compact comparison currently shows averages, and we plan to add the median and p95 of per-step timings to make occasional large stalls easier to distinguish from consistently slow input. Your experiments help clarify why that distinction matters.

---

<div class="post-metadata">

**Author:** ![thordata](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thordata/32/62596_2.png) [@thordata](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthordata)%5C
**Post date:** [September 18, 2026, 9:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproducing-a-lerobot-regression-with-traceml%2F180066%2F10 "2026-09-18T09:02:16Z")

</div>

The distinction between average fetch time and occasional large stalls is really useful.

For video-heavy datasets, I’m also curious whether pre-decoding clips versus reading raw MP4s on demand changes the worker-count pattern.
