Closed Bug 1600952 Opened 2 months ago Closed 2 months ago

Perma TEST-UNEXPECTED-FAIL | Builds differ. See the diff.html or diff.txt artifact

Categories

(Firefox Build System :: Task Configuration, defect)

defect
Not set

Tracking

(firefox-esr68 unaffected, firefox71 unaffected, firefox72 fixed, firefox73 verified)

VERIFIED FIXED
mozilla73
Tracking Status
firefox-esr68 --- unaffected
firefox71 --- unaffected
firefox72 --- fixed
firefox73 --- verified

People

(Reporter: malexandru, Assigned: glandium)

Details

Attachments

(2 files)

Central as Late Beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=ac6cd17d6342d211ff009f21ccf396e364a9b9d9&searchStr=diffoscope%2Copt%2Cdiff-reproducible-linux32%2C%28linux32%29

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=279306573&repo=try&lineNumber=171

[task 2019-12-03T12:26:16.784Z] + case "$ORIG_URL" in
[task 2019-12-03T12:26:16.784Z] + OMNIJAR=omni.ja
[task 2019-12-03T12:26:16.784Z] + POST=true
[task 2019-12-03T12:26:16.784Z] + for option in "$@"
[task 2019-12-03T12:26:16.784Z] + case "$option" in
[task 2019-12-03T12:26:16.784Z] + POST=fail
[task 2019-12-03T12:26:16.784Z] + '[' -n '' ']'
[task 2019-12-03T12:26:16.784Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata --exclude-command .--line-numbers --exclude '/*freeblpriv3.chk' --exclude '/*nssdbm3.chk' --exclude '**/*softokn3.chk' a b
[task 2019-12-03T12:26:16.847Z] | | 0% ETA: --:--:--
[task 2019-12-03T12:26:16.847Z] 2019-12-03 12:26:16 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
[task 2019-12-03T12:26:16.884Z] | | 0% ETA: --:--:--
[task 2019-12-03T12:26:16.884Z] | | 0% application.ini ETA: 0:20:11
[task 2019-12-03T12:26:16.888Z] | | 0% browser ETA: 0:17:52
[task 2019-12-03T12:26:16.896Z] | | 0% chrome ETA: 1 day, 5:57:06
[task 2019-12-03T12:26:16.896Z] | | 0% default16.png ETA: 1 day, 12:00:01
[task 2019-12-03T12:26:16.896Z] | | 0% default32.png ETA: 1 day, 12:06:21
[task 2019-12-03T12:26:16.896Z] | | 0% default48.png ETA: 1 day, 12:05:27
[task 2019-12-03T12:26:16.897Z] | | 0% default64.png ETA: 1 day, 11:58:12
[task 2019-12-03T12:26:16.897Z] | | 0% ETA: 1 day, 11:42:48
[task 2019-12-03T12:26:16.897Z] | | 0% ETA: 1 day, 11:45:36
[task 2019-12-03T12:26:16.897Z] | | 0% ETA: 1 day, 11:45:49
[task 2019-12-03T12:26:16.897Z] | | 0% chrome.manifest ETA: 1 day, 11:47:40
[task 2019-12-03T12:26:16.897Z] | | 0% …ashreporter-override.ini ETA: 1 day, 11:52:58
[task 2019-12-03T12:26:16.900Z] | | 0% features ETA: 1 day, 11:55:11
[task 2019-12-03T12:26:16.901Z] | | 0% …autofill@mozilla.org.xpi ETA: 6 days, 21:04:11
[task 2019-12-03T12:26:16.903Z] | | 0% …eenshots@mozilla.org.xpi ETA: 3 days, 7:14:30
[task 2019-12-03T12:26:16.904Z] | | 0% …reporter@mozilla.org.xpi ETA: 1 day, 18:49:55
[task 2019-12-03T12:26:16.905Z] | | 0% webcompat@mozilla.org.xpi ETA: 1 day, 18:33:43
[task 2019-12-03T12:26:16.905Z] | | 0% ETA: 1 day, 17:50:39
[task 2019-12-03T12:26:16.929Z] | | 0% omni.ja ETA: 1 day, 17:56:20
[task 2019-12-03T12:26:16.929Z] | | 0% ETA: 0:39:19
[task 2019-12-03T12:26:16.929Z] | | 0% chrome.manifest ETA: 0:39:23
[task 2019-12-03T12:26:16.931Z] | | 0% crashreporter ETA: 0:39:28
[task 2019-12-03T12:26:16.931Z] | | 0% crashreporter.ini ETA: 0:00:51
[task 2019-12-03T12:26:16.936Z] | | 0% defaults ETA: 0:00:50
[task 2019-12-03T12:26:16.936Z] | | 0% ETA: 0:00:52
[task 2019-12-03T12:26:16.936Z] | | 0% ETA: 0:00:52
[task 2019-12-03T12:26:16.936Z] | | 0% dependentlibs.list ETA: 0:00:52
[task 2019-12-03T12:26:16.936Z] | | 0% firefox ETA: 0:00:52
[task 2019-12-03T12:26:16.938Z] | | 0% firefox-bin ETA: 0:00:50
[task 2019-12-03T12:26:16.942Z] | | 0% fonts ETA: 0:00:16
[task 2019-12-03T12:26:16.942Z] | | 0% ETA: 0:00:17
[task 2019-12-03T12:26:16.948Z] | | 0% gmp-clearkey ETA: 0:00:17
[task 2019-12-03T12:26:16.948Z] | | 0% manifest.json ETA: 0:00:18
[task 2019-12-03T12:26:16.948Z] | | 0% ETA: 0:00:18
[task 2019-12-03T12:26:16.949Z] | | 0% ETA: 0:00:18
[task 2019-12-03T12:26:16.951Z] | | 0% gtk2 ETA: 0:00:18
[task 2019-12-03T12:26:16.951Z] | | 0% ETA: 0:00:19
[task 2019-12-03T12:26:16.953Z] | | 0% icons ETA: 0:00:19
[task 2019-12-03T12:26:16.953Z] | | 0% ETA: 0:00:19
[task 2019-12-03T12:26:16.955Z] | | 0% libfreeblpriv3.so ETA: 0:00:19
[task 2019-12-03T12:26:16.955Z] | | 0% liblgpllibs.so ETA: 0:00:12
[task 2019-12-03T12:26:16.957Z] | | 0% libmozavcodec.so ETA: 0:00:11
[task 2019-12-03T12:26:16.958Z] | | 1% libmozavutil.so ETA: 0:00:10
[task 2019-12-03T12:26:16.958Z] | | 1% libmozgtk.so ETA: 0:00:09
[task 2019-12-03T12:26:16.960Z] | | 1% libmozsandbox.so ETA: 0:00:09
[task 2019-12-03T12:26:16.962Z] | | 1% libmozsqlite3.so ETA: 0:00:08
[task 2019-12-03T12:26:16.962Z] | | 2% libmozwayland.so ETA: 0:00:05
[task 2019-12-03T12:26:16.963Z] | | 2% libnspr4.so ETA: 0:00:05
[task 2019-12-03T12:26:16.965Z] | | 2% libnss3.so ETA: 0:00:04
[task 2019-12-03T12:26:16.966Z] | | 2% libnssckbi.so ETA: 0:00:04
[task 2019-12-03T12:26:16.968Z] | | 3% libnssdbm3.so ETA: 0:00:03
[task 2019-12-03T12:26:16.969Z] | | 3% libnssutil3.so ETA: 0:00:03
[task 2019-12-03T12:26:16.969Z] | | 3% libplc4.so ETA: 0:00:03
[task 2019-12-03T12:26:16.970Z] | | 3% libplds4.so ETA: 0:00:03
[task 2019-12-03T12:26:16.971Z] | | 3% libsmime3.so ETA: 0:00:03
[task 2019-12-03T12:26:16.972Z] | | 3% libsoftokn3.so ETA: 0:00:03
[task 2019-12-03T12:26:16.974Z] | | 3% libssl3.so ETA: 0:00:03
[task 2019-12-03T12:27:13.141Z] |# | 3% libxul.so ETA: 0:00:03
[task 2019-12-03T12:27:13.157Z] |# | 7% …ox/libxul.so [.gnu.hash] ETA: 0:12:11
[task 2019-12-03T12:27:13.248Z] |## | 10% …efox/libxul.so [.dynstr] ETA: 0:08:20
[task 2019-12-03T12:27:13.615Z] |### | 13% …ul.so [.elfhack.data.v0] ETA: 0:06:14
[task 2019-12-03T12:27:13.641Z] |#### | 16% …ul.so [.elfhack.text.v0] ETA: 0:04:56
[task 2019-12-03T12:27:13.663Z] |##### | 19% …irefox/libxul.so [.init] ETA: 0:04:01
[task 2019-12-03T12:27:13.886Z] |##### | 21% …firefox/libxul.so [.plt] ETA: 0:03:21
[task 2019-12-03T12:27:13.911Z] |###### | 24% …fox/libxul.so [.plt.got] ETA: 0:02:51
[task 2019-12-03T12:32:53.459Z] |####### | 27% …irefox/libxul.so [.text] ETA: 0:02:27
[task 2019-12-03T12:32:53.556Z] |######## | 30% …fox/libxul.so [text_env] ETA: 0:14:47
[task 2019-12-03T12:32:53.575Z] |######### | 33% …irefox/libxul.so [.fini] ETA: 0:12:54
[task 2019-12-03T12:33:04.226Z] |######### | 36% …efox/libxul.so [.rodata] ETA: 0:11:20
[task 2019-12-03T12:33:05.313Z] |########## | 39% …ibxul.so [.eh_frame_hdr] ETA: 0:10:16
[task 2019-12-03T12:33:16.416Z] |########### | 42% …ox/libxul.so [.eh_frame] ETA: 0:09:06
[task 2019-12-03T12:33:16.475Z] |############ | 45% …l.so [.gcc_except_table] ETA: 0:08:17
[task 2019-12-03T12:33:16.491Z] |############# | 48% …refox/libxul.so [.tdata] ETA: 0:07:21
[task 2019-12-03T12:33:16.502Z] |############# | 51% …irefox/libxul.so [.tbss] ETA: 0:06:32
[task 2019-12-03T12:33:16.514Z] |############## | 54% …/libxul.so [.init_array] ETA: 0:05:48
[task 2019-12-03T12:33:16.525Z] |############### | 57% …/libxul.so [.fini_array] ETA: 0:05:08
[task 2019-12-03T12:33:17.641Z] |################ | 60% …libxul.so [.data.rel.ro] ETA: 0:04:33
[task 2019-12-03T12:33:17.657Z] |################# | 63% …firefox/libxul.so [.got] ETA: 0:04:01
[task 2019-12-03T12:33:17.678Z] |################# | 66% …fox/libxul.so [.got.plt] ETA: 0:03:31
[task 2019-12-03T12:33:17.760Z] |################## | 69% …irefox/libxul.so [.data] ETA: 0:03:04
[task 2019-12-03T12:33:17.772Z] |################### | 72% …firefox/libxul.so [.bss] ETA: 0:02:39
[task 2019-12-03T12:33:17.784Z] |#################### | 75% …fox/libxul.so [.comment] ETA: 0:02:17
[task 2019-12-03T12:33:17.795Z] |##################### | 78% …bxul.so [.gnu_debuglink] ETA: 0:01:55
[task 2019-12-03T12:33:17.807Z] |##################### | 81% …ox/libxul.so [.shstrtab] ETA: 0:01:36
[task 2019-12-03T12:33:17.807Z] |###################### | 84% ETA: 0:01:18
[task 2019-12-03T12:33:17.810Z] |###################### | 84% minidump-analyzer ETA: 0:01:18
[task 2019-12-03T12:33:17.824Z] |####################### | 85% omni.ja ETA: 0:01:13
[task 2019-12-03T12:33:17.826Z] |########################## | 99% pingsender ETA: 0:00:03
[task 2019-12-03T12:33:17.826Z] |########################## | 99% platform.ini ETA: 0:00:01
[task 2019-12-03T12:33:17.829Z] |########################## | 99% plugin-container ETA: 0:00:01
[task 2019-12-03T12:33:17.829Z] |########################## | 99% precomplete ETA: 0:00:00
[task 2019-12-03T12:33:17.829Z] |########################## | 99% removed-files ETA: 0:00:00
[task 2019-12-03T12:33:17.829Z] |########################## | 99% update-settings.ini ETA: 0:00:00
[task 2019-12-03T12:33:17.831Z] |########################## | 99% updater ETA: 0:00:00
[task 2019-12-03T12:33:17.832Z] |########################## | 99% updater.ini ETA: 0:00:00
[task 2019-12-03T12:33:17.832Z] |###########################| 100% ETA: 0:00:00
[task 2019-12-03T12:33:17.832Z] |###########################| 100% ETA: 0:00:00
[task 2019-12-03T12:33:17.832Z] |###########################| 100% ETA: 0:00:00
[task 2019-12-03T12:33:55.053Z] |###########################| 100% Time: 0:07:00
[task 2019-12-03T12:33:55.053Z] + fail
[task 2019-12-03T12:33:55.053Z] + set +x
[task 2019-12-03T12:33:55.053Z] TEST-UNEXPECTED-FAIL | Builds differ. See the diff.html or diff.txt artifact
[taskcluster 2019-12-03 12:33:55.365Z] === Task Finished ===
[taskcluster 2019-12-03 12:34:03.342Z] Unsuccessful task run with exit code: 1 completed in 493.407 seconds

It's not only late beta. While it passed for the early beta directly pushed before the late beta in comment 0, this early beta sim pushed later also failed it https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=diffoscope&revision=136a7e6a5f841c05de5cbb440fdf39d5120c6574

Flags: needinfo?(mh+mozilla)

That is not supposed to run on beta/release, per https://searchfox.org/mozilla-central/source/taskcluster/ci/build/linux.yml#649. And since you're pushing to try, it runs, but it doesn't matter.

But I now realize that the diff task itself doesn't have the same setup, and I guess that means it would run on beta/release too... Tom, do you confirm? (does a task that has no run-on-projects and depends on one "win"?)

Flags: needinfo?(mh+mozilla) → needinfo?(mozilla)
Summary: Perma Late Beta TEST-UNEXPECTED-FAIL | Builds differ. See the diff.html or diff.txt artifact when Gecko 73 merges to Beta on 06-01-20 → Perma TEST-UNEXPECTED-FAIL | Builds differ. See the diff.html or diff.txt artifact

The diffs on autoland are different.

Oh wait, autoland is perma-fail now. When did that start?

Flags: needinfo?(aryx.bugmail)

(In reply to Mike Hommey [:glandium] (high latency) from comment #2)

But I now realize that the diff task itself doesn't have the same setup, and I guess that means it would run on beta/release too... Tom, do you confirm? (does a task that has no run-on-projects and depends on one "win"?)

Yes. run-on-projects controls what gets added to the set of target tasks, but everything they depend on is pulled into the target graph (which then has optimizations applied to it).

That is not supposed to run on beta/release, per https://searchfox.org/mozilla-central/source/taskcluster/ci/build/linux.yml#649. And since you're pushing to try, it runs,

This actually is not true. The beta simulations use the release_simulation target task method, which evaluates run-on-projects with mozilla-beta to more closely simulate what will happen when things are pushed to beta.

Flags: needinfo?(mozilla)
Assignee: nobody → mh+mozilla
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/b02e985f2130
Do not run the diff-reproducible-linux32 task on beta/release as intended in bug 1577212. r=tomprince

Comment on attachment 9113388 [details]
Bug 1600952 - Do not run the diff-reproducible-linux32 task on beta/release as intended in bug 1577212.

Beta/Release Uplift Approval Request

  • User impact if declined: The diff-reproducible-linux32 task is known to fail on beta/release (bug 1596350). Bug 1577212 intended the task to be disabled on beta/release, but missed the necessary bits for that to actually happen, which this adds.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Taskcluster configuration only.
  • String changes made/needed: N/A
Attachment #9113388 - Flags: approval-mozilla-beta?
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73

Comment on attachment 9113388 [details]
Bug 1600952 - Do not run the diff-reproducible-linux32 task on beta/release as intended in bug 1577212.

disable a CI task that fails on beta, approved for 72.0b3

Attachment #9113388 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/ff6b73da39fe
Followup for bug 1600952: run on trunk, not mozilla-central.
You need to log in before you can comment on or make changes to this bug.