Open Bug 1777035 Opened 2 years ago Updated 1 day ago

Intermittent (Frequent on esr115) toolkit/components/passwordmgr/test/mochitest/test_autocomplete_password_generation_confirm.html | single tracking bug

Categories

(Toolkit :: Password Manager, defect, P3)

defect

Tracking

()

People

(Reporter: jmaher, Assigned: joschmidt, NeedInfo)

References

Details

(Keywords: intermittent-failure, intermittent-testcase, leave-open, Whiteboard: [fxcm-tech-debt][stockwell unknown])

Attachments

(8 files, 1 obsolete file)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
No description provided.

Additional information about this bug failures and frequency patterns can be found by running: ./mach test-info failure-report --bug 1777035

Severity: normal → S3

This bug has spiked in failures especially on linux asan since the patch in https://bugzilla.mozilla.org/show_bug.cgi?id=1833987#c23 landed.
A before/after failure rate can be seen here and here - https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2023-06-01&endday=2023-06-08&tree=trunk&failurehash=all&bug=1777035
Johannes, please have a look over this one too. Thank you.

Flags: needinfo?(joschmidt)
Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fa74019b770a
Disable test_autocomplete_password_generation_confirm.html on linux for frequent failures at least until a fix is in place. r=credential-management-reviewers,sgalich
Whiteboard: [stockwell disable-recommended] → [stockwell disabled]

Thank you for disabling! I have not managed to fix this yet.

Flags: needinfo?(joschmidt)
Assignee: nobody → joschmidt
Whiteboard: [stockwell disabled] → [stockwell disabled fxcm-tech-debt]
Duplicate of this bug: 1721900
Whiteboard: [stockwell disabled fxcm-tech-debt] → [stockwell disabled][fxcm-tech-debt]

There have been 83 total failures in the last 7 days.
There are:

  • 19 failures on macosx1015-64-qr debug and opt
  • 2 failures on macosx1015-64-shippable-qr opt
  • 6 failures on windows10-64-2009-qr opt
  • 7 failures on windows11-32-2009-qr opt and debug
  • 30 failures on windows11-64-2009-asan-qr opt
  • 19 failures on windows11-64-2009-qr debug

Recent failure log.

Johannes, is there any chance you could look into this?
It has an extremely high frequency on macOS and Win ( left after the linux disable) - and it is also present on release trees ( 55 failures in the last 7 days on esr115).

Should we disable this test completely? Do you have time to look into this?
Thank you.

Flags: needinfo?(joschmidt)

This is taking longer than expected. Can't give an ETA yet. If not ready until tonight, I'd propose we disable it completely for now

Flags: needinfo?(joschmidt)
Attachment #9343871 - Attachment description: WIP: - Bug 1777035 - fix intermittent test_autocomplete_password_generation_confirm.html → Bug 1777035 - fix intermittent test_autocomplete_password_generation_confirm.html - r?#credential-management-reviewers!
Pushed by joschmidt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7af2b04879fc
fix intermittent test_autocomplete_password_generation_confirm.html - r=credential-management-reviewers,jneuberger

Hi Natalia, I hope this is a fix. I will monitor the failure rate and if it goes down, I'd re-enable the test on linux, too. Otherwise lets disable completely.

(In reply to Johannes from comment #82)

Hi Natalia, I hope this is a fix. I will monitor the failure rate and if it goes down, I'd re-enable the test on linux, too. Otherwise lets disable completely.

Thank you for taking the time to put up a patch!

Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/23e906719520
re-enable test_autocomplete_password_generation_confirm.html after a potential fix. a=test-only
Whiteboard: [stockwell disabled][fxcm-tech-debt] → [fxcm-tech-debt]

This still has a consistent failure rate on linux as seen here. Let's see how it goes on mac and windows but it might end up being disabled everywhere.
Backout link: https://hg.mozilla.org/integration/autoland/rev/b0f00c8e055766816efa02a825aa5c1f7bd11be9

Whiteboard: [fxcm-tech-debt] → [stockwell disabled][fxcm-tech-debt]

On windows the failure rate looks something like this. On retriggers in failed 8 times out of 11.

I'm working on a new fix

Attachment #9344242 - Attachment description: WIP: - Bug 1777035 - fix intermittent test_autocomplete_password_generation_confirm.html → Bug 1777035 - fix intermittent test_autocomplete_password_generation_confirm.html - r?#credential-management-reviewers!

Update

There have been 80 total failures within the last 7 days:

  • 26 failures on OS X 10.15 WebRender opt/debug
  • 2 failures on OS X 10.15 WebRender Shippable opt
  • 6 failures on windows10-64-2009-qr debug
  • 5 failures on Windows 11 x86 22H2 WebRender debug
  • 32 failures on Windows 11 x64 22H2 asan WebRender opt
  • 1 failure on Windows 11 x64 22H2 CCov WebRender opt
  • 8 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=424090605&repo=autoland&lineNumber=13254

Whiteboard: [stockwell disabled][fxcm-tech-debt] → [stockwell disabled][fxcm-tech-debt][stockwell needswork:owner]

There have been 91 total failures in the last 7 days.
There are:

  • 24 failures on macosx1015-64-qr opt and debug
  • 4 failures on macosx1015-64-shippable-qr opt
  • 3 failures on windows10-64-2009-qr debug
  • 10 failures on windows11-32-2009-qr debug
  • 36 failures on windows11-64-2009-asan-qr opt
  • 14 failures on windows11-64-2009-qr debug

Recent failure log.

Johannes, do you have any updates? Thank you

Flags: needinfo?(joschmidt)

Hi, my patch is still in review

Flags: needinfo?(joschmidt)

Hi Johannes! Your patch is accepted. Can you please land it, maybe we can get rid of these failures.
Thank you!

Flags: needinfo?(joschmidt)

There have been 60 total failures in the last 7 days.
There are:

  • 12 failures on macosx1015-64-qr debug
  • 2 failures on macosx1015-64-shippable-qr opt
  • 2 failures on windows10-64-2009-qr opt
  • 8 failures on windows11-32-2009-qr opt
  • 27 failures on windows11-64-2009-asan-qr opt
  • 9 failures on windows11-64-2009-qr debug

Hi Johannes! Your patch is accepted. Can you please land it? Thank you!

Any chance to get the patch landed soon to resolve the frequent failures?

Next thing on my agenda!

Flags: needinfo?(joschmidt)
Pushed by joschmidt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7a41636ba8ee
fix intermittent test_autocomplete_password_generation_confirm.html - r=credential-management-reviewers,sgalich
https://hg.mozilla.org/integration/autoland/rev/0e00e863ca4c
flatten function calls - r=credential-management-reviewers,sgalich
https://hg.mozilla.org/integration/autoland/rev/d50213d1862b
test in between password field - r=credential-management-reviewers,sgalich
https://hg.mozilla.org/integration/autoland/rev/3bbd6a67ae15
check all other password fields to be masked - r=credential-management-reviewers,sgalich
https://hg.mozilla.org/integration/autoland/rev/ab58e27cf5b6
re-enable test_autocomplete_password_generation_confirm.html on linux - r=credential-management-reviewers,sgalich

Looks like this is still failing pretty frequently

Flags: needinfo?(joschmidt)
Whiteboard: [stockwell disabled][fxcm-tech-debt][stockwell unknown] → [stockwell disabled][fxcm-tech-debt][stockwell needswork:owner]
Whiteboard: [stockwell disabled][fxcm-tech-debt][stockwell needswork:owner] → [fxcm-tech-debt][stockwell needswork:owner]

I hope above patch will fix this intermittent. At least it will definitively ease debugging.

Flags: needinfo?(joschmidt)
Pushed by joschmidt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1eeeded0c0b6
and 1852745 - modernize test_autocomplete_password_generation_confirm.html - r=credential-management-reviewers,dimi
No longer depends on: 1633064, 1633541, 1690358

Here is a range of backfills and retriggers on esr115. I stopped with the backfills at that point but it seems that this is going on for some time.
The jobs fail with this failure line: TEST-UNEXPECTED-FAIL | toolkit/components/passwordmgr/test/mochitest/test_autocomplete_password_generation_confirm.html | Checking array values: Check all rows are correct - got 2, expected 1

Hi Sergey! Hi Johannes! Can you please take a look at this? Maybe you can figure out what is going on here.
Thank you!

Flags: needinfo?(joschmidt)
Summary: Intermittent toolkit/components/passwordmgr/test/mochitest/test_autocomplete_password_generation_confirm.html | single tracking bug → Intermittent (Frequent on esr115) toolkit/components/passwordmgr/test/mochitest/test_autocomplete_password_generation_confirm.html | single tracking bug
Flags: needinfo?(sgalich)
Flags: needinfo?(sgalich)
Attachment #9385570 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: