2000512 - Race condition in Bugbug Push Schedules artifact generation

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[Myeongjun Go]

We identified a race condition in the Bugbug Push Schedules(i.e, perfherder-data-bugbug-*.json) artifact generation logic [0].
When multiple PERFHERDER_DATA outputs are produced nearly simultaneously, the current file-naming logic[0] is not atomic.
As a result, two or more writers may attempt to write to the same artifact path at the same time.

Symptoms:

• Expected output: 5 artifacts for Bugbug push schedules (e.g., 1~5) [1]
• In failing runs: only 4 artifacts are created
• One of the generated artifacts is malformed JSON (e.g., trailing }}, ]}]}, or other leftover bytes)

This pattern is consistent with two concurrent writers both opening the same file in "w" mode:

1. Both processes resolve the same artifact name (e.g., index 1)
2. File is truncated multiple times
3. One write partially overwrites the other, leaving trailing garbage

Use a naming scheme that avoids filename collisions. Instead of using increasing numbers (1, 2, 3…), include something unique—like a timestamp and a small hash, so that two processes never try to write to the same file at the same time.

[0]
https://searchfox.org/firefox-main/rev/1c2b83c1fdb8ba523831bcdb63947f5f7a4cfa5e/taskcluster/gecko_taskgraph/util/bugbug.py#58,72
[1]
https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=R2xyPK-lSZWG4dPYGLefHg.0&tier=1%2C2%2C3&revision=d475a2b98667b8d90016d0faf3accaafdc69fd18

Comment 1

[Myeongjun Go]

Attachment: Bug 2000512 - Race condition in Bugbug Push Schedules artifact generation r=sparky

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/34db77d645cd
https://hg.mozilla.org/integration/autoland/rev/108908c8b9cd
Race condition in Bugbug Push Schedules artifact generation r=sparky,taskgraph-reviewers,bhearsum

Comment 3

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/108908c8b9cd