1880704 - Permafailing QA test_pdf_download | AssertionError('Mismatching screenshots for test_pdf_download_base.png')

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=447548115&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LhaogxCARNevpIIx6lvEsg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-02-17T01:11:42.761Z] TEST-START | test_pdf_download
[task 2024-02-17T01:12:51.849Z] Wait for consent form: timed out, maybe it is not here
[task 2024-02-17T01:13:02.026Z] Saving screenshot 'screenshot_fail.png' to '/home/ubuntu/tasks/task_170812943797888/builds/worker/artifacts/screenshot_fail.png'
[task 2024-02-17T01:13:02.146Z] Saving screenshot 'screenshot_fail_parent.png' to '/home/ubuntu/tasks/task_170812943797888/builds/worker/artifacts/screenshot_fail_parent.png'
[task 2024-02-17T01:13:02.295Z] TEST-UNEXPECTED-FAIL | test_pdf_download | AssertionError('Mismatching screenshots for test_pdf_download_base.png')
[task 2024-02-17T01:13:02.295Z] TEST-INFO took 79533ms
[task 2024-02-17T01:13:02.308Z] Traceback (most recent call last):
[task 2024-02-17T01:13:02.308Z]   File "/home/ubuntu/tasks/task_170812943797888/fetches/basic_tests.py", line 77, in __init__
[task 2024-02-17T01:13:02.308Z]     rv = getattr(self, m)(self._expectations[m])
[task 2024-02-17T01:13:02.308Z]   File "/home/ubuntu/tasks/task_170812943797888/fetches/qa_tests.py", line 506, in test_pdf_download
[task 2024-02-17T01:13:02.308Z]     self.assert_rendering(exp, self._driver)
[task 2024-02-17T01:13:02.308Z]   File "/home/ubuntu/tasks/task_170812943797888/fetches/basic_tests.py", line 193, in assert_rendering
[task 2024-02-17T01:13:02.308Z]     assert diff.getbbox() is None, "Mismatching screenshots for {}".format(
[task 2024-02-17T01:13:02.308Z] AssertionError: Mismatching screenshots for test_pdf_download_base.png
[task 2024-02-17T01:13:05.882Z] Exiting with False
[task 2024-02-17T01:13:05.882Z] SUITE-END | took 247s

Comment 1

[Natalia Csoregi [:nataliaCs]]

:gerard-majax hi, can you take a look at this?
Thank you.

Comment 2

[Natalia Csoregi [:nataliaCs]]

Another mismatching screenshots failure: TEST-UNEXPECTED-FAIL | test_pdf_zoom | AssertionError('Mismatching screenshots for test_pdf_zoom_p1_400p.png')

Comment 3

[Intermittent Failures Robot]

18 failures in 910 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 18

Platform and build breakdown:

• linux64-snap-stable: 6
  • debug: 5
  • opt: 1
• linux64-snap-esr: 10
  • debug: 9
  • opt: 1
• linux64-snap-beta: 2
  • opt: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1880704&startday=2024-02-16&endday=2024-02-16&tree=all

Comment 4

[:gerard-majax]

Sorry, I was expecting a bit of intermittence, but not that much and not perma failing. ...

Comment 5

[Intermittent Failures Robot]

24 failures in 3125 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #41 most frequent failure this week.

Repository breakdown:

• mozilla-central: 24

Platform and build breakdown:

• linux64-snap-stable: 8
  • debug: 6
  • opt: 2
• linux64-snap-esr: 12
  • debug: 10
  • opt: 2
• linux64-snap-beta: 4
  • opt: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1880704&startday=2024-02-12&endday=2024-02-18&tree=all

Comment 6

[:gerard-majax]

Attachment: Bug 1880704 - Change how we wait PDF.js loading for QA tests r?#releng-reviewers!,mboldan!

Comment 7

[:gerard-majax]

Should be better, although not yet perfect https://treeherder.mozilla.org/jobs?repo=try&revision=d460f47ae1e59462ea377c5fad4072c8ea8dd05e

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b06a44eea69a
Change how we wait PDF.js loading for QA tests r=mboldan

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1866798

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b06a44eea69a

Comment 11

[Sandor Molnar[:smolnar]]

@Gerard, this is still perma-failing

L.E. Failure log #2

Comment 12

[Intermittent Failures Robot]

15 failures in 541 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15

Platform and build breakdown:

• linux64-snap-stable: 12
  • debug: 11
  • opt: 1
• linux64-snap-beta: 1
  • debug: 1
• linux64-snap-esr: 2
  • opt: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1880704&startday=2024-02-20&endday=2024-02-20&tree=all

Comment 13

[:gerard-majax]

Thanks, I dont understand ...

Comment 14

[:gerard-majax]

OK let's try:

• allowing try to re-use mozilla-central built snap packages
• use canvas for getting the rendered PDF instead of relying on the screenshot from selenium, most if not all of the failures I saw are just toolbar elements

Comment 15

[:gerard-majax]

Attachment: Bug 1880704 - Allow downloading Snap package from mozilla-central r?mboldan!

Comment 16

[:gerard-majax]

Attachment: Bug 1880704 - Crop more PDF rendering and wait differently for rendering r?mboldan!

Depends on D202338

Comment 17

[:gerard-majax]

canvas was a pain to deal with, so cropping more and a few other adjustments:

• https://treeherder.mozilla.org/jobs?repo=try&revision=079d04689d8320e74e4375bdafb8305867360a6c
• https://treeherder.mozilla.org/jobs?repo=try&revision=2aeb677bda4a2bee53ee24195832bbf5c798394e

Should be more reliable

Comment 18

[Intermittent Failures Robot]

35 failures in 3382 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #43 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 35

Platform and build breakdown:

• linux64-snap-stable: 20
  • debug: 15
  • opt: 5
• linux64-snap-esr: 12
  • debug: 8
  • opt: 4
• linux64-snap-beta: 2
  • debug: 2
• linux64-snap-nightly: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1880704&startday=2024-02-19&endday=2024-02-25&tree=all

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/554b19b4f7ac
Allow downloading Snap package from mozilla-central r=mboldan
https://hg.mozilla.org/integration/autoland/rev/9218ead49f78
Crop more PDF rendering and wait differently for rendering r=mboldan

Comment 20

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/554b19b4f7ac
https://hg.mozilla.org/mozilla-central/rev/9218ead49f78