1703887 - Perma macOS dom/media/test/reftest/bipbop_300_215kbps.mp4.lastframe.html == dom/media/test/reftest/bipbop_300_215kbps.mp4.lastframe-ref.html | image comparison, max difference: 95, number of differing pixels: 38260 (MacOS 10.15 issue)

Status: NEW

(Core :: Audio/Video: Playback, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=335853364&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/H36FEJawTPCMf6efM6J8RA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/H36FEJawTPCMf6efM6J8RA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2021-04-08T16:04:15.687Z] 16:04:15     INFO - REFTEST TEST-LOAD | http://localhost:50478/1617897852393/3/reftest/bipbop_300_215kbps.mp4.lastframe-ref.html | 2 / 8 (25%)
[task 2021-04-08T16:04:15.911Z] 16:04:15     INFO - REFTEST INFO | REFTEST fuzzy test (55, 4173) <= (95, 38260) <= (80, 4417)
[task 2021-04-08T16:04:16.165Z] 16:04:16     INFO - [Child 2154, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7496
[task 2021-04-08T16:04:16.389Z] 16:04:16     INFO - REFTEST TEST-UNEXPECTED-FAIL | dom/media/test/reftest/bipbop_300_215kbps.mp4.lastframe.html == dom/media/test/reftest/bipbop_300_215kbps.mp4.lastframe-ref.html | image comparison, max difference: 95, number of differing pixels: 38260```

Comment 1

[Natalia Csoregi [:nataliaCs]]

Jeff, any chance you could take a quick look over this?
Thank you.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

Ok, yeah, this is sort of expected. Why is this tier2?

Comment 3

[Cosmin Sabou [:CosminS]]

Hi Jeff, the jobs that run on trees are split into 3 categories, tier 1,2 and 3. For tier 1 failures we backout directly, for tier 2 failures we file a bug, add the regressor and ni de dev (as it's the case here) and tier 3 are not followed by the sheriffs and don't show up on TH by default.

Comment 4

[Intermittent Failures Robot]

37 failures in 707 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 36
• mozilla-central: 1

Platform and build breakdown:

• macosx1015-64-qr: 36
  • opt: 8
  • debug: 28
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-08&endday=2021-04-08&tree=all

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

It's tier 2 because macOS 10.15 got newly added and is still in the stabilization phase.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1703724

Comment 7

[Intermittent Failures Robot]

26 failures in 758 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 3
• try: 2

Platform and build breakdown:

• macosx1015-64-qr: 23
  • debug: 16
  • opt: 7
• macosx1015-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-09&endday=2021-04-09&tree=all

Comment 8

[Intermittent Failures Robot]

31 failures in 191 pushes (0.162 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 20
• mozilla-central: 9
• try: 2

Platform and build breakdown:

• macosx1015-64-qr: 27
  • debug: 20
  • opt: 7
• macosx1015-64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-10&endday=2021-04-10&tree=all

Comment 9

[Intermittent Failures Robot]

116 failures in 3854 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 92
• mozilla-central: 19
• try: 5

Platform and build breakdown:

• macosx1015-64-qr: 105
  • debug: 77
  • opt: 28
• macosx1015-64-shippable-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 10

[Intermittent Failures Robot]

44 failures in 779 pushes (0.056 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 34
• mozilla-central: 9
• try: 1

Platform and build breakdown:

• macosx1015-64-qr: 40
  • debug: 18
  • opt: 22
• macosx1015-64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-12&endday=2021-04-12&tree=all

Comment 11

[Intermittent Failures Robot]

62 failures in 649 pushes (0.096 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 49
• mozilla-central: 9
• try: 4

Platform and build breakdown:

• macosx1015-64-qr: 57
  • opt: 29
  • debug: 28
• macosx1015-64-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-13&endday=2021-04-13&tree=all

Comment 12

[Intermittent Failures Robot]

78 failures in 1076 pushes (0.072 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 70
• mozilla-central: 7
• try: 1

Platform and build breakdown:

• macosx1015-64-qr: 75
  • opt: 38
  • debug: 37
• macosx1015-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-14&endday=2021-04-14&tree=all

Comment 13

[Intermittent Failures Robot]

75 failures in 870 pushes (0.086 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 58
• mozilla-central: 12
• try: 5

Platform and build breakdown:

• macosx1015-64-qr: 70
  • opt: 33
  • debug: 37
• macosx1015-64-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-15&endday=2021-04-15&tree=all

Comment 14

[Intermittent Failures Robot]

56 failures in 648 pushes (0.086 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 44
• mozilla-central: 10
• try: 2

Platform and build breakdown:

• macosx1015-64-qr: 50
  • debug: 30
  • opt: 20
• macosx1015-64-shippable-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-16&endday=2021-04-16&tree=all

Comment 15

[Intermittent Failures Robot]

35 failures in 149 pushes (0.235 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 27
• mozilla-central: 6
• try: 2

Platform and build breakdown:

• macosx1015-64-qr: 31
  • opt: 9
  • debug: 22
• macosx1015-64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-17&endday=2021-04-17&tree=all

Comment 16

[Natalia Csoregi [:nataliaCs]]

Update:

There have been 359 failures within the last 7 days on OS X 10.15 WebRender and OS X 10.15 WebRender Shippable.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=336903159&repo=autoland&lineNumber=11047

Given the failure rate, should we disable this test until a fix shall be in place?

Comment 17

[Intermittent Failures Robot]

364 failures in 4255 pushes (0.086 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 291
• mozilla-central: 57
• try: 16

Platform and build breakdown:

• macosx1015-64-qr: 334
  • debug: 180
  • opt: 154
• macosx1015-64-shippable-qr: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 18

[Cosmin Sabou [:CosminS]]

Attachment: Bug 1703887 - Disable bipbop_300_215kbps.mp4.lastframe.html on mac for perma failures. r=jmaher,jrmuizel

Comment 19

[Cosmin Sabou [:CosminS]]

This is now tier 1 failure as macOS 10.15 was upgraded to tier 1 in Bug 1705839.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/05871e4f57ec
Disable bipbop_300_215kbps.mp4.lastframe.html on mac for perma failures. r=jmaher DONTBUILD

Comment 21

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/05871e4f57ec

Comment 22

[Intermittent Failures Robot]

38 failures in 663 pushes (0.057 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 9
• try: 5
• mozilla-beta: 3

Platform and build breakdown:

• macosx1015-64-qr: 31
  • opt: 9
  • debug: 22
• macosx1015-64-shippable-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-19&endday=2021-04-19&tree=all

Comment 23

[Intermittent Failures Robot]

51 failures in 3224 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #14 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 21
• mozilla-central: 9
• try: 6
• mozilla-beta: 15

Platform and build breakdown:

• macosx1015-64-qr: 31
  • opt: 9
  • debug: 22
• macosx1015-64-shippable-qr: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1703887&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 24

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/1085e53c5e1b

Comment 25

[Jim Mathies [:jimm]]

Do we care about this? Seemed like a pretty bad failure before the disable.

Comment 26

[Jim Mathies [:jimm]]

Pixel values are getting the wrong gamma applied to them.