1597903 - Improve error reporting for the diffoscope tasks

Status: NEW

(Firefox Build System :: Task Configuration, task, P3)

Description

[Mike Hommey [:glandium]]

cf. bug 1597901, the thing that pops to sheriffs is not really helpful (Downloaded image is corrupted: (HTTP code 404) no such image) because it's actually irrelevant to the actual error which is that the builds differ.

So we should add an error message that shows up on treeherder, possibly with a link to the diff artifact if possible.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1597903 - Add an error message when differences are found in diffoscope.

Comment 2

[Mike Hommey [:glandium]]

Let's start with just an error message. I'll leave the bug open to think more about maybe ways we could munge diff.txt to generate something useful for sheriffs to copy/paste when filing bugs.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd6b79c74b23
Add an error message when differences are found in diffoscope. r=froydnj

Comment 4

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/cd6b79c74b23

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1597903 - Improve error reporting on diffoscope tasks.

Somehow parse the diff output to print out slightly more unique error messages.

With this change, the error for bug 1597901 becomes:
TEST-UNEXPECTED-FAIL | firefox/libmozavutil.so differs. See the diff.html or diff.txt artifact

And the error for bug 1601150 becomes:
TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact

Which is not great, but better than the status quo.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1597903 - When a build differs, check if the generated-files differ.

With this change, the error for bug 1601150 becomes:
TEST-UNEXPECTED-FAIL | i686-unknown-linux-gnu/release/build/cranelift-codegen-37dab135a3db8b7a/out/inst_builder.rs differs. See the generated-files.diff.html or generated-files.diff.txt artifact
TEST-UNEXPECTED-FAIL | i686-unknown-linux-gnu/release/build/cranelift-codegen-37dab135a3db8b7a/out/opcodes.rs differs. See the generated-files.diff.html or generated-files.diff.txt artifact
TEST-UNEXPECTED-FAIL | i686-unknown-linux-gnu/release/build/cranelift-codegen-e483dcad1bdf8c4a/out/inst_builder.rs differs. See the generated-files.diff.html or generated-files.diff.txt artifact
TEST-UNEXPECTED-FAIL | i686-unknown-linux-gnu/release/build/cranelift-codegen-e483dcad1bdf8c4a/out/opcodes.rs differs. See the generated-files.diff.html or generated-files.diff.txt artifact

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6435902a44ef
Improve error reporting on diffoscope tasks. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/ae2503b461f3
When a build differs, check if the generated-files differ. r=froydnj

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/6435902a44ef
https://hg.mozilla.org/mozilla-central/rev/ae2503b461f3

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:glandium, maybe it's time to close this bug?

Comment 10

[Mike Hommey [:glandium]]

That's still something that needs to be looked into, but it's not on my more immediate list of things to do.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:ahal, maybe it's time to close this bug?