1820334 - Perma toolkit/components/glean/EventExtraGIFFTMaps.cpp differs. See the diff.html or diff.txt artifact

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: chorotan [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=407811971&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FnJne8T0RhKxQb9j4EfmJA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-03-04T11:21:11.695Z]  |########################## |   98%                             ETA:  0:00:00 
[task 2023-03-04T11:21:11.700Z]  |########################## |   98%     a/xpcom/xpcom-config.h  ETA:  0:00:00 
[task 2023-03-04T11:21:11.700Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2023-03-04T11:21:11.700Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2023-03-04T11:21:11.700Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2023-03-04T11:21:47.742Z]  |###########################|  100%                             Time: 0:00:26 
[task 2023-03-04T11:21:47.742Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2023-03-04T11:21:47.743Z] + /builds/worker/bin/report_error diff
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/EventExtraGIFFTMaps.cpp differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/EventGIFFTMap.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/GleanJSMetricsLookup.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/GleanJSPingsLookup.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/GleanMetrics.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/GleanPings.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/HistogramGIFFTMap.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/ScalarGIFFTMap.h differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/api/src/factory.rs differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/api/src/metrics.rs differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.759Z] TEST-UNEXPECTED-FAIL | toolkit/components/glean/api/src/pings.rs differs. See the diff.html or diff.txt artifact
[task 2023-03-04T11:21:47.761Z] + fail
[task 2023-03-04T11:21:47.761Z] + exit 1
[taskcluster 2023-03-04 11:21:48.035Z] === Task Finished ===
[taskcluster 2023-03-04 11:21:48.857Z] Unsuccessful task run with exit code: 1 completed in 90.904 seconds

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1816190

:travis_, since you are the author of the regressor, bug 1816190, could you take a look?

For more information, please visit auto_nag documentation.

Comment 2

[Intermittent Failures Robot]

18 failures in 3752 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 17
• autoland: 1

Platform and build breakdown:

• diff: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1820334&startday=2023-02-27&endday=2023-03-05&tree=all

Comment 3

[Chris H-C :chutten]

First time I've heard about "Diffoscope". Found a doc which doesn't explain much. By searching the code I also found this hidden doc which is a little more helpful in understanding the intent: "Make sure the build remains reproducible"

This means there's instability in the order things are generated. Unspurprisingly so, since glean_parser doesn't use ordereddict anywhere.

I wonder why it only started to complain now, and why our existing codegen regression tests didn't trip this earlier...

Comment 4

[BMO Automation]

Attachment: [mozilla/glean_parser] Bug 1820334 - Sort generated Metrics, Pings and Tags (#560)

Comment 5

[Intermittent Failures Robot]

20 failures in 748 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 13
• autoland: 7

Platform and build breakdown:

• diff: 12
  • opt: 12
• macosx1015-64-shippable-qr: 1
  • opt: 1
• macosx1015-64-qr: 7
  • debug: 5
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1820334&startday=2023-03-09&endday=2023-03-09&tree=all

Comment 6

[Intermittent Failures Robot]

73 failures in 3966 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 66
• autoland: 7

Platform and build breakdown:

• diff: 65
  • opt: 65
• macosx1015-64-qr: 7
  • debug: 5
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1820334&startday=2023-03-06&endday=2023-03-12&tree=all

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1820189

Comment 9

[Mike Hommey [:glandium]]

Another side effect of bug 1820189 is that top-level $objdir/backend.mk too is not deterministic (because it's fed with glean stuff too), which means cascades into the glean headers being rebuilt when the build backend is regenerated, which, since the headers are not deterministic either, cascades into a ton of C++ being rebuilt too.

Comment 10

[Mike Hommey [:glandium]]

(In reply to Chris H-C :chutten from comment #3)

I wonder why it only started to complain now, and why our existing codegen regression tests didn't trip this earlier...

The use of sets in metrics_index.py added in https://phabricator.services.mozilla.com/D171576 surely doesn't help.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1820334 - Make the metrics_yamls and ping_yamls lists deterministically ordered.

Comment 12

[Intermittent Failures Robot]

15 failures in 865 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15

Platform and build breakdown:

• diff: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1820334&startday=2023-03-15&endday=2023-03-15&tree=all

Comment 13

[Intermittent Failures Robot]

72 failures in 3761 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #14 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 72

Platform and build breakdown:

• diff: 72
  • opt: 72

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1820334&startday=2023-03-13&endday=2023-03-19&tree=all

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/29cf43c71b23
Make the metrics_yamls and ping_yamls lists deterministically ordered. r=chutten,nalexander

Comment 15

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/29cf43c71b23

Comment 16

[Mike Hommey [:glandium]]

Comment on attachment 9323408 [details]
Bug 1820334 - Make the metrics_yamls and ping_yamls lists deterministically ordered.

Beta/Release Uplift Approval Request

• User impact if declined: Faster builds on automation because of better cache hits. Faster incremental builds locally. And this makes the shippable Linux builds reproducible again.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Build-time change, and very limited in scope.
• String changes made/needed: N/A
• Is Android affected?: Yes

Comment 17

[Dianna Smith [:diannaS]]

Comment on attachment 9323408 [details]
Bug 1820334 - Make the metrics_yamls and ping_yamls lists deterministically ordered.

Approved for 112.0b5

Comment 18

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/ee35489f5569

Comment 19

[Intermittent Failures Robot]

12 failures in 4161 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• toolchains: 4
• autoland: 1

Platform and build breakdown:

• diff: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1820334&startday=2023-03-20&endday=2023-03-26&tree=all

Comment 20

[Jan-Erik Rediger [:janerik]]

I suspect the remaining non-deterministic parts are because glean_parser doesn't keep an order on metrics. I might be able to look into this shortly.

Edit: Scratch the first part. I don't actually know. I still plan to look into it.

Comment 21

[Jan-Erik Rediger [:janerik]]

Hm, I see one tiny issue, though I'm not sure anymore this actually caused problems in my local test.
Looking at the failures it doesn't seem to be still failing anymore, all the failures are form the day before the fix landed actually.

Guess I accidentally reopened this. No work to be done here.