1851497 - Windows 11 Mochitests with conditioned profile perma failing with Left over worker: https://s.0cf.io/sw.js?id=c859531b-3ae0-f3ad-0025-2175efd7e4ca (scope: https://s.0cf.io/

Status: RESOLVED FIXED

(Core :: DOM: Service Workers, defect, P2)

Description

[Narcis Beleuzu [:NarcisB]]

Starting with this push, the Windows 11 Mochitests with conditioned profile are perma failing with Left over worker: https://s.0cf.io/sw.js?id=c859531b-3ae0-f3ad-0025-2175efd7e4ca (scope: https://s.0cf.io/

Log link: https://treeherder.mozilla.org/logviewer?job_id=427925350&repo=autoland&lineNumber=2643

Log snippet:

[task 2023-09-04T14:06:44.465Z] 14:06:44     INFO - TEST-PASS | dom/serviceworkers/test/test_abrupt_completion.html | Correct response text 
[task 2023-09-04T14:06:44.465Z] 14:06:44     INFO - add_task | Leaving testFetchHandler
[task 2023-09-04T14:06:44.465Z] 14:06:44     INFO - Buffered messages finished
[task 2023-09-04T14:06:44.466Z] 14:06:44     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_abrupt_completion.html | Left over worker: https://s.0cf.io/sw.js?id=c859531b-3ae0-f3ad-0025-2175efd7e4ca (scope: https://s.0cf.io/) 
[task 2023-09-04T14:06:44.466Z] 14:06:44     INFO -     SimpleTest.ok@SimpleTest/SimpleTest.js:424:16
[task 2023-09-04T14:06:44.466Z] 14:06:44     INFO -     afterCleanup@SimpleTest/SimpleTest.js:1481:20
[task 2023-09-04T14:06:44.466Z] 14:06:44     INFO -     async*executeCleanupFunction@SimpleTest/SimpleTest.js:1521:7
[task 2023-09-04T14:06:44.467Z] 14:06:44     INFO - GECKO(3816) | MEMORY STAT | vsize 2103691MB | vsizeMaxContiguous 87514148MB | residentFast 54MB | heapAllocated 8MB
[task 2023-09-04T14:06:44.467Z] 14:06:44    ERROR - TEST-UNEXPECTED-FAIL | /tests/dom/serviceworkers/test/test_abrupt_completion.html logged result after SimpleTest.finish(): Left over worker: https://s.0cf.io/sw.js?id=c859531b-3ae0-f3ad-0025-2175efd7e4ca (scope: https://s.0cf.io/)
[task 2023-09-04T14:06:44.468Z] 14:06:44     INFO - TEST-OK | dom/serviceworkers/test/test_abrupt_completion.html | took 1516ms

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

These permanent failures on Windows 11 with a conditioned profile (= profile gets upgraded once per day, and clones of it used for testing to identify issues with persistent data) started today after the latest conditioned profile had been created.

The mentioned host s.0cf.io is known for spreading unwanted and/or malicious software. It's not mentioned in the source code of mozilla-central. The changes between the push for which the previous conditioned profile got generated shows no related changes, and retriggers of successful runs hit this failure.

addons in the new conditioned profile - the same addons and version as for the previous conditioned profile:

[e-mail] - 1.0.1
[e-mail] - 1.0.0
[e-mail] - 39.0.1
[e-mail] - 1.5.1
[e-mail] - 118.0.0
[e-mail] - 1.3
[e-mail] - 2.0.0
[e-mail] - 1.4
[e-mail] - 1.3
[e-mail] - 1.6
[e-mail] - 1.4
[e-mail] - 1.2
[e-mail] - 1.2
[e-mail] - 1.4
[e-mail] - 1.12

The issue should be the AlternativeServices.txt file found in the profile folder root. It has these 2 additions in the new profile

line 239
https:ps.0cf.io:443:.:^partitionKey=%28https%2Cebay.com%29:3 0 19604 https:ps.0cf.io:443:ps.0cf.io:443::n:1693916739:h3:y:1693829595:n:^partitionKey=%28https%2Cebay.com%29:|n:y:

line 595
https:s.0cf.io:443:.:^partitionKey=%28https%2Cebay.com%29:3 0 19604 https:s.0cf.io:443:s.0cf.io:443::n:1693916831:h3:y:1693829595:n:^partitionKey=%28https%2Cebay.com%29:|n:y:

Files with the conditioned profiles ("Artifacts") tab:

• new
• old

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1840135 touched the code related to AlternativeServices.txt but got backed out before it made it to central.

Any insight what's going on here?

Comment 3

[Dana Keeler (she/her) [:keeler]]

I think what that's saying is when the profile visited ebay.com, it (ebay) included content from ps.0cf.io/s.0cf.io, and when it (the profile) connected to those sites, they a) installed a service worker and b) sent an Alt-Svc header (which is basically just "you can reach this service at this other location"). I don't think there's anything particularly sinister here, other than the online ad ecosystem being kinda scummy.

My sense is that test_abrupt_completion.html isn't sound with conditioned profiles, precisely because other sites could register serviceworkers that it would consider to be leftover.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1851497 - skip dom/serviceworkers/test/test_abrupt_completion.html with condition profiles. r=jmaher

Because conditioned profiles retain profile data, a service worker installed
by an ebay.com page (but for a different host) got detected as unexpected
service worker by this test.

Comment 5

[Intermittent Failures Robot]

69 failures in 653 pushes (0.106 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 62
• mozilla-central: 7

Platform and build breakdown:

• windows11-64-2009-qr: 31
  • opt: 31
• windows11-64-2009-shippable-qr: 37
  • opt: 37
• windows11-32-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-07&endday=2023-09-07&tree=all

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

(I commented on the PR at https://phabricator.services.mozilla.com/D187656#6212009 about how this is actually the mochitest checks for leftover ServiceWorkers that's sad and test_abrupt_completion, although it has other problems, is not really involved in the problem, just the first test that gets run in the process.)

Comment 7

[Intermittent Failures Robot]

71 failures in 682 pushes (0.104 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15
• autoland: 56

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 43
  • opt: 43
• windows11-64-2009-qr: 28
  • opt: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-08&endday=2023-09-08&tree=all

Comment 8

[Intermittent Failures Robot]

47 failures in 125 pushes (0.376 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31
• mozilla-central: 16

Platform and build breakdown:

• windows11-64-2009-qr: 15
  • opt: 15
• windows11-64-2009-shippable-qr: 32
  • opt: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-09&endday=2023-09-09&tree=all

Comment 9

[Intermittent Failures Robot]

421 failures in 3267 pushes (0.129 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 333
• mozilla-central: 88

Platform and build breakdown:

• windows11-64-2009-qr: 171
  • opt: 171
• windows11-64-2009-shippable-qr: 249
  • opt: 249
• windows11-32-2009-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 10

[Intermittent Failures Robot]

68 failures in 545 pushes (0.125 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 53
• mozilla-central: 11
• try: 4

Platform and build breakdown:

• windows11-64-2009-qr: 33
  • opt: 33
• windows11-64-2009-shippable-qr: 35
  • opt: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-11&endday=2023-09-11&tree=all

Comment 11

[Intermittent Failures Robot]

84 failures in 559 pushes (0.15 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 17
• autoland: 66
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-shippable-qr: 49
  • opt: 49
• windows11-64-2009-qr: 34
  • opt: 34
• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-12&endday=2023-09-12&tree=all

Comment 12

[Cosmin Sabou [:CosminS]]

Hi, any chance something could be done about these failures? There are 618 total failures since this bug was filled and it's failing at a large rate: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2023-08-14&endday=2023-09-13&tree=trunk&failurehash=all&bug=1851497

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

Yes, I'll put up a patch along the lines of what I said in comment 6. Taking the bug.

Comment 14

[Cosmin Sabou [:CosminS]]

Thank you. Much appreciated.

Comment 15

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Bug 1851497 - Expect the new SW in conditioned profiles.

The conditioned profile has picked up a new ServiceWorker due to a
change in the sites the profile is run against. We need to ignore this
in the SimpleTest.js automatic check that makes sure tests didn't fail
to clean up after themselves. (A test that fails to clean up after
itself is potentially broken in multiple ways.)

As shown in the log in comment 0 of the bug, the SW script ends up with
a cache-busting query parameter that looks like a UUID
(fake example: sw.js?ima-fake-uuid) that seems likely to be unstable
because it is either used to ensure the SW gets updated or because it's
a tracking parameter. So now we check for our list of expected SWs to
be prefixes of the actual scope we find.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/299706d11bb7
Expect the new SW in conditioned profiles. r=jmarshall

Comment 17

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/299706d11bb7

Comment 18

[Intermittent Failures Robot]

52 failures in 662 pushes (0.079 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 44
• mozilla-central: 8

Platform and build breakdown:

• windows11-64-2009-qr: 22
  • opt: 22
• windows11-64-2009-shippable-qr: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-13&endday=2023-09-13&tree=all

Comment 19

[Intermittent Failures Robot]

204 failures in 3337 pushes (0.061 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 163
• mozilla-central: 36
• try: 4
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 89
  • opt: 89
• windows11-64-2009-shippable-qr: 114
  • opt: 114
• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851497&startday=2023-09-11&endday=2023-09-17&tree=all