1541192 - Intermittent /service-workers/service-worker/update-no-cache-request-headers.https.html | headers in no-cache mode - assert_equals: expected (string) "etag" but got (undefined) undefined

Status: RESOLVED INCOMPLETE

(Core :: DOM: Service Workers, defect, P3)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: rgurzau [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=237630702&repo=mozilla-central

https://queue.taskcluster.net/v1/task/EXUM0xGnRhaQDQUvi_Qq9w/runs/0/artifacts/public/logs/live_backing.log

17:29:09 INFO - TEST-START | /service-workers/service-worker/update-no-cache-request-headers.https.html
17:29:09 INFO - Closing window 356482285569
17:29:09 INFO - PID 5076 | [Parent 7732, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [Child 620, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [Child 620, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [Child 620, Main Thread] WARNING: A runnable was posted to a worker that is already shutting down!: file z:/build/build/src/dom/workers/WorkerPrivate.cpp, line 1355
17:29:09 INFO - PID 5076 | [Child 620, Main Thread] WARNING: Failed to dispatch offline status change event!: file z:/build/build/src/dom/workers/WorkerPrivate.cpp, line 1822
17:29:09 INFO - PID 5076 | [Parent 7732, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | ++DOCSHELL 000001EC2554D800 == 1 [pid = 8292] [id = {68abae30-9e22-4894-96e0-3a62b501c053}]
17:29:09 INFO - PID 5076 | [GPU 688, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [GPU 688, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | ++DOMWINDOW == 1 (000001EC2B6FD020) [pid = 8292] [serial = 1] [outer = 0000000000000000]
17:29:09 INFO - PID 5076 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
17:29:09 INFO - PID 5076 | [GPU 688, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [Parent 7732, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [Parent 7732, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | [Parent 7732, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
17:29:09 INFO - PID 5076 | ++DOMWINDOW == 2 (000001EC2B6E5800) [pid = 8292] [serial = 2] [outer = 000001EC2B6FD020]
17:29:10 INFO - PID 5076 | --DOCSHELL 0000023E8B04E000 == 0 [pid = 620] [id = {defae4cb-2f00-4e7f-b194-98716a2b91aa}] [url = https://web-platform.test:8443/service-workers/service-worker/update-missing-import-scripts.https.html]
17:29:10 INFO - PID 5076 | --DOMWINDOW == 3 (0000023E91709020) [pid = 620] [serial = 1] [outer = 0000000000000000] [url = https://web-platform.test:8443/service-workers/service-worker/update-missing-import-scripts.https.html]
17:29:10 INFO - PID 5076 | --DOMWINDOW == 2 (0000023E917CA800) [pid = 620] [serial = 4] [outer = 0000000000000000] [url = https://web-platform.test:8443/service-workers/service-worker/update-missing-import-scripts.https.html]
17:29:10 INFO - PID 5076 | --DOMWINDOW == 1 (0000023E911E8800) [pid = 620] [serial = 2] [outer = 0000000000000000] [url = about:blank]
17:29:10 INFO - PID 5076 | --DOMWINDOW == 0 (0000023E917C1000) [pid = 620] [serial = 3] [outer = 0000000000000000] [url = about:blank]
17:29:10 INFO - PID 5076 | ++DOMWINDOW == 3 (000001EC2CED4800) [pid = 8292] [serial = 3] [outer = 000001EC2B6FD020]
17:29:10 INFO - PID 5076 | nsStringStats
17:29:10 INFO - PID 5076 | => mAllocCount: 9258
17:29:10 INFO - PID 5076 | => mReallocCount: 0
17:29:10 INFO - PID 5076 | => mFreeCount: 9258
17:29:10 INFO - PID 5076 | => mShareCount: 10781
17:29:10 INFO - PID 5076 | => mAdoptCount: 544
17:29:10 INFO - PID 5076 | => mAdoptFreeCount: 598
17:29:10 INFO - PID 5076 | => Process ID: 620, Thread ID: 8072
17:29:10 INFO - PID 5076 | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to c:\users\task_1554223313\appdata\local\temp\tmpqpfwnl.mozrunner\runtests_leaks_9696_tab_pid7056.log
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file z:/build/build/src/xpcom/base/nsSystemInfo.cpp, line 111
17:29:10 INFO - PID 5076 | [Parent 7732, Main Thread] WARNING: '!parent', file z:/build/build/src/netwerk/ipc/NeckoParent.cpp, line 955
17:29:10 INFO - PID 5076 | ++DOMWINDOW == 4 (000001EC2CEDE400) [pid = 8292] [serial = 4] [outer = 000001EC2B6FD020]
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: NS_ENSURE_SUCCESS_VOID(rv) failed with result 0x80070057: file z:/build/build/src/toolkit/components/antitracking/AntiTrackingCommon.cpp, line 1724
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: '!window', file z:/build/build/src/dom/cache/CacheStorage.cpp, line 572
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: '!window', file z:/build/build/src/dom/cache/CacheStorage.cpp, line 572
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: '!window', file z:/build/build/src/dom/cache/CacheStorage.cpp, line 572
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: NS_ENSURE_SUCCESS_VOID(rv) failed with result 0x80070057: file z:/build/build/src/toolkit/components/antitracking/AntiTrackingCommon.cpp, line 1724
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: '!window', file z:/build/build/src/dom/cache/CacheStorage.cpp, line 572
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: '!window', file z:/build/build/src/dom/cache/CacheStorage.cpp, line 572
17:29:10 INFO - PID 5076 | [Child 8292, Main Thread] WARNING: '!window', file z:/build/build/src/dom/cache/CacheStorage.cpp, line 572
17:29:10 INFO - PID 5076 | [Parent 7732, Main Thread] WARNING: '!wind[[C[hild 7208,C[Child 7056, MChild 1200, Main Thread] WARNING: '!window', file z:/obuild/buiawld/srhild 5204in Thread] WARNING: ''!window', file z:/ Main Thread] cb,WARNING: '!winuild/ b,uil file z:/build/buildd/srcdow', file z:M///build/build/srcain Thread] WARNING: '!window', /dom/cache/CadcheStorage.cpp,f osm/clache/CacheStorage.cpp, lin/dom/cache/CacheStorage.cpp, line 5eri7i2
17:29:10 INFO - PID 5076 | le n z:/buic/dom/c5ld/build/src/dom/cache/Ce72
17:29:10 INFO - PID 5076 | ache/CacheSt acheStorage.cpp, li5orage.cpp, line 572
17:29:10 INFO - PID 5076 | 72
17:29:10 INFO - PID 5076 | ne 572
17:29:10 INFO -
17:29:10 INFO - TEST-UNEXPECTED-FAIL | /service-workers/service-worker/update-no-cache-request-headers.https.html | headers in no-cache mode - assert_equals: expected (string) "etag" but got (undefined) undefined
17:29:10 INFO - @https://web-platform.test:8443/service-workers/service-worker/update-no-cache-request-headers.https.html:45:3
17:29:10 INFO - async*Test.prototype.step@https://web-platform.test:8443/resources/testharness.js:1587:25
17:29:10 INFO - promise_test/tests.promise_tests</<@https://web-platform.test:8443/resources/testharness.js:591:36
17:29:10 INFO - promise_test/tests.promise_tests<@https://web-platform.test:8443/resources/testharness.js:590:20
17:29:10 INFO - promise callback*promise_test@https://web-platform.test:8443/resources/testharness.js:589:51
17:29:10 INFO - @https://web-platform.test:8443/service-workers/service-worker/update-no-cache-request-headers.https.html:20:1
17:29:10 INFO - TEST-OK | /service-workers/service-worker/update-no-cache-request-headers.https.html | took 725ms

Comment 1

[Bogdan Tara[:bogdan_tara | bogdant]]

Seeing this on Tier 1:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=238361369&repo=mozilla-inbound&lineNumber=13180

12:16:28 INFO - TEST-START | /service-workers/service-worker/update-no-cache-request-headers.https.html
12:16:28 INFO - Closing window 354334801921
12:16:28 INFO - PID 4676 | [Parent 3952, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 3940, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 3940, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO -
12:16:28 INFO - TEST-UNEXPECTED-FAIL | /service-workers/service-worker/update-no-cache-request-headers.https.html | headers in no-cache mode - assert_equals: expected (string) "etag" but got (undefined) undefined
12:16:28 INFO - @https://web-platform.test:8443/service-workers/service-worker/update-no-cache-request-headers.https.html:45:3
12:16:28 INFO - asyncTest.prototype.step@https://web-platform.test:8443/resources/testharness.js:1587:25
12:16:28 INFO - promise_test/tests.promise_tests</<@https://web-platform.test:8443/resources/testharness.js:591:36
12:16:28 INFO - promise_test/tests.promise_tests<@https://web-platform.test:8443/resources/testharness.js:590:20
12:16:28 INFO - promise callbackpromise_test@https://web-platform.test:8443/resources/testharness.js:589:51
12:16:28 INFO - @https://web-platform.test:8443/service-workers/service-worker/update-no-cache-request-headers.https.html:20:1
12:16:28 INFO - TEST-OK | /service-workers/service-worker/update-no-cache-request-headers.https.html | took 275ms
12:16:28 INFO - PID 4676 | 1554466588795 Marionette INFO Stopped listening on port 2828
12:16:28 INFO - PID 4676 | JavaScript error: resource:///modules/AsyncTabSwitcher.jsm, line 255: TypeError: browser.frameLoader is null
12:16:28 INFO - PID 4676 | [Parent 3952, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 4596, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 4596, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Parent 3952, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 5320, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 5320, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, li[Child 5356, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 5356, Chrome_ChildThread] WARNING: pipe error: 109: file z:/tas[Child 3780, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Child 3780, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:28 INFO - PID 4676 | [Parent 3952, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1554458632/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
12:16:29 INFO - PID 4676 | [Child 1464, Chrome_ChildThread] WARNING: pipe error: 109: file
12:16:29 INFO - Browser exited with return code 0

Comment 2

[Intermittent Failures Robot]

9 failures in 2843 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• mozilla-inbound: 1

Platform breakdown:

• windows10-64-qr: 8
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 3

[Intermittent Failures Robot]

9 failures in 3025 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• autoland: 2
• try: 1

Platform breakdown:

• windows10-64-qr: 6
• windows7-32-shippable: 2
• windows10-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 4

[Intermittent Failures Robot]

7 failures in 2673 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform breakdown:

• windows10-64-qr: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 5

[Intermittent Failures Robot]

4 failures in 3031 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 1

Platform breakdown:

• windows10-64-qr: 2
• windows10-64-ccov: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 6

[Intermittent Failures Robot]

8 failures in 4131 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• try: 3
• mozilla-inbound: 1

Platform breakdown:

• windows10-64-qr: 4
• android-em-7-0-x86: 1
• windows10-64: 1
• windows7-32-shippable: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 7

[Intermittent Failures Robot]

10 failures in 4705 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 2
• mozilla-inbound: 4
• mozilla-central: 3

Platform breakdown:

• windows7-32-shippable: 7
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 8

[Intermittent Failures Robot]

14 failures in 4658 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 6
• mozilla-beta: 1
• autoland: 4
• mozilla-inbound: 2

Platform breakdown:

• windows7-32-shippable: 8
• windows7-32: 4
• windows7-32-devedition: 1
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 9

[Razvan Maries]

There are 24 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-05-19&endday=2019-05-26&tree=trunk&bug=1541192

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248376993&repo=mozilla-inbound&lineNumber=12788

23:54:42 INFO - TEST-START | /service-workers/service-worker/update-no-cache-request-headers.https.html
23:54:42 INFO - Closing window 356482285569
23:54:42 INFO - PID 772 | [Parent 5276, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 2244, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 2244, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO -
23:54:42 INFO - TEST-UNEXPECTED-FAIL | /service-workers/service-worker/update-no-cache-request-headers.https.html | headers in no-cache mode - assert_equals: expected (string) "etag" but got (undefined) undefined
23:54:42 INFO - @https://web-platform.test:8443/service-workers/service-worker/update-no-cache-request-headers.https.html:45:3
23:54:42 INFO - asyncTest.prototype.step@https://web-platform.test:8443/resources/testharness.js:1587:25
23:54:42 INFO - promise_test/tests.promise_tests</<@https://web-platform.test:8443/resources/testharness.js:591:36
23:54:42 INFO - promise_test/tests.promise_tests<@https://web-platform.test:8443/resources/testharness.js:590:20
23:54:42 INFO - promise callbackpromise_test@https://web-platform.test:8443/resources/testharness.js:589:51
23:54:42 INFO - @https://web-platform.test:8443/service-workers/service-worker/update-no-cache-request-headers.https.html:20:1
23:54:42 INFO - TEST-OK | /service-workers/service-worker/update-no-cache-request-headers.https.html | took 338ms
23:54:42 INFO - PID 772 | 1558828482687 Marionette INFO Stopped listening on port 2828
23:54:42 INFO - PID 772 | [Parent 5276, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 5880, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 5880, Chrome_ChildThread] WARNING: pipe error: 109: file z:/[Parent 5276, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 6136, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 5772, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 5772, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:42 INFO - PID 772 | [Child 3664, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1558818059/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
23:54:43 INFO - PID 772 | [Child 3664, Chrome_ChildThread] WARNING: pipe error: 109: file
23:54:43 INFO - Browser exited with return code 0
23:54:43 INFO - PROCESS LEAKS None
23:54:43 INFO - Closing logging queue
23:54:43 INFO - queue closed
23:54:43 INFO - Setting up ssl
23:54:43 INFO - certutil |
23:54:43 INFO - certutil |
23:54:43 INFO - certutil |
23:54:43 INFO - Certificate Nickname Trust Attributes
23:54:43 INFO - SSL,S/MIME,JAR/XPI
23:54:43 INFO -
23:54:43 INFO - web-platform-tests CT,,
23:54:43 INFO -
23:54:43 INFO - Application command: Z:\task_1558820837\build\application\firefox\firefox.exe -marionette about:blank --wait-for-browser -profile c:\users\task_1558820837\appdata\local\temp\tmpyxbcgh.mozrunner
23:54:43 INFO - Starting runner
23:54:44 INFO - PID 5392 | 1558828484257 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
23:54:44 INFO - PID 5392 | 1558828484258 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: telemetry
23:54:44 INFO - PID 5392 | 1558828484258 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
23:54:44 INFO - PID 5392 | 1558828484258 [e-mail] WARN Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
23:54:45 INFO - PID 5392 | 1558828485876 Marionette INFO Listening on port 2828

Andrew, could you please assign this to someone?

Comment 10

[Intermittent Failures Robot]

26 failures in 4730 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 5
• autoland: 17
• try: 3
• mozilla-central: 1

Platform breakdown:

• windows7-32-shippable: 24
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 11

[Intermittent Failures Robot]

26 failures in 4337 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• mozilla-inbound: 6
• mozilla-central: 3
• try: 1

Platform breakdown:

• windows7-32-shippable: 22
• windows10-64-qr: 2
• windows7-32-devedition: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-05-27&endday=2019-06-02&tree=all

Comment 12

[Natalia Csoregi [:nataliaCs]]

Update: There have been 31 failures within the last 7 days:

• 2 failures on Windows 10 x64 QuantumRender debug
• 1 failure on Windows 7 debug
• 28 failures on Windows 7 Shippable opt

e.g. recent log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249856471&repo=autoland&lineNumber=12850

Comment 13

[Andrew Overholt [:overholt]]

Things will change after we land bug 1231208, so let's disable this until then.

Comment 14

[Intermittent Failures Robot]

25 failures in 5197 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• mozilla-inbound: 6
• try: 3
• mozilla-beta: 1
• mozilla-central: 3

Platform breakdown:

• windows7-32-shippable: 20
• windows10-64-qr: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 15

[Intermittent Failures Robot]

17 failures in 4753 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-beta: 1
• mozilla-esr68: 2
• autoland: 10
• mozilla-inbound: 2

Platform breakdown:

• windows7-32-devedition: 1
• windows7-32-shippable: 12
• windows10-64-shippable: 1
• windows10-64: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 16

[Intermittent Failures Robot]

3 failures in 2581 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-esr68: 1
• mozilla-central: 1

Platform breakdown:

• windows7-32-shippable: 1
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-06-17&endday=2019-06-23&tree=all

Comment 17

[Intermittent Failures Robot]

4 failures in 4409 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 3

Platform breakdown:

• windows10-64-qr: 3
• linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 18

[Intermittent Failures Robot]

2 failures in 3501 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-esr68: 1

Platform breakdown:

• windows10-64: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 19

[Intermittent Failures Robot]

7 failures in 4710 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2
• mozilla-inbound: 1
• try: 1

Platform breakdown:

• windows10-64: 1
• windows10-64-qr: 2
• windows7-32: 2
• windows7-32-devedition: 1
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 20

[Intermittent Failures Robot]

5 failures in 4298 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2
• mozilla-esr68: 1
• try: 1

Platform breakdown:

• windows7-32-shippable: 2
• linux64-ccov: 1
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 21

[Intermittent Failures Robot]

3 failures in 4608 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• try: 2

Platform breakdown:

• windows10-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 22

[Intermittent Failures Robot]

3 failures in 4439 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1
• mozilla-release: 1
• try: 1

Platform breakdown:

• windows7-32-shippable: 1
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 23

[Intermittent Failures Robot]

10 failures in 4666 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 2
• mozilla-release: 1
• mozilla-inbound: 1
• try: 1

Platform breakdown:

• windows7-32: 5
• linux64-ccov: 2
• windows7-32-shippable: 1
• windows10-64: 1
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 24

[Intermittent Failures Robot]

12 failures in 4839 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-beta: 1

Platform breakdown:

• windows7-32: 9
• windows10-64: 2
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 25

[Intermittent Failures Robot]

11 failures in 5073 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 2
• mozilla-inbound: 1
• mozilla-beta: 2
• try: 1

Platform breakdown:

• windows10-64-qr: 5
• linux64-ccov: 3
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 26

[Intermittent Failures Robot]

9 failures in 5047 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-inbound: 1
• try: 1
• autoland: 4
• mozilla-beta: 1

Platform breakdown:

• windows10-64-ccov: 1
• windows7-32: 5
• windows10-64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 27

[Intermittent Failures Robot]

8 failures in 4408 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 3
• mozilla-inbound: 1

Platform breakdown:

• windows7-32: 5
• windows10-64-qr: 2
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-09-02&endday=2019-09-08&tree=all

Comment 28

[Intermittent Failures Robot]

4 failures in 4637 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1
• mozilla-central: 1
• try: 1
• autoland: 1

Platform breakdown:

• windows7-32-shippable: 1
• linux64-qr: 1
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-09-09&endday=2019-09-15&tree=all

Comment 29

[Intermittent Failures Robot]

4 failures in 4350 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform breakdown:

• windows10-64: 3
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 30

[Intermittent Failures Robot]

1 failures in 4530 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 31

[Intermittent Failures Robot]

1 failures in 4292 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-09-30&endday=2019-10-06&tree=all

Comment 32

[Intermittent Failures Robot]

34 failures in 3801 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 16
• mozilla-central: 13
• try: 5

Platform breakdown:

• macosx1014-64: 4
• windows10-64: 10
• macosx1014-64-shippable: 6
• windows10-aarch64: 9
• windows10-64-qr: 2
• linux64-qr: 1
• linux64-shippable: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 33

[Alexandru Michis [:malexandru]]

There is a spike of 35 failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-10-08&endday=2019-10-15&tree=trunk&bug=1541192

Affected platforms are:

• linux64-qr opt, linux64-shippable opt
• macosx1014-64 debug, macosx1014-64-shippable opt
• windows10-64 debug, windows10-64-qr debug, windows10-aarch64 opt

Jens, could you redirect this to someone who can take a look at it?

Comment 34

[Jens Stutte [:jstutte]]

@Perry: please triage.

Comment 35

[Andrew Sutherland [:asuth] (he/him)]

I believe comment 13 may have been in error due to confusion during a bulk triage session.

The check in question appears to be validating step 19.2.2.1 of https://fetch.spec.whatwg.org/#http-network-or-cache-fetch: "If storedResponse’s header list contains ETag, then append If-None-Match with its value to httpRequest’s header list."

Our logic to set this appears to live at https://searchfox.org/mozilla-central/rev/ebe492edacc75bb122a2b380e4cafcca3470864c/netwerk/protocol/http/nsHttpChannel.cpp#4515

Our current meta settings for the test at https://searchfox.org/mozilla-central/source/testing/web-platform/meta/service-workers/service-worker/update-no-cache-request-headers.https.html.ini make a number of failures okay, but it does seem like there is fundamentally an underlying intermittent-ness in the test.

The most likely problem is that the necko HTTP layer is returning a cached response when it should not. One notable thing is that the script file at https://searchfox.org/mozilla-central/source/testing/web-platform/tests/service-workers/service-worker/resources/test-request-headers-worker.py always returns a header of the form "ETag: etag". The semantics of "If-None-Match" are on the server to validate the Etag and return a 200 if it's changed. In this case, the server always returns the same ETag, but will also return a 200 because the server is a simple python script that doesn't validate the ETag at all. But it seems conceivable our network layer might have some optimization that assumes the server would obey the etag semantics. Alternately, there might be some edge case where the original request is still sufficiently alive from necko's perspective that we de-duplicate to the already-returned channel. Or alternately some kind of wasteful update() suppression is in effect. This definitely would benefit from an rr/pernosco trace.

Clearing Perry needinfo since Jan triaged appropriately and this is slightly lower than the set of SW e10s regressions that just happened. I'll quickly see if I can convince pernosco to generate a trace though.

Comment 36

[Cosmin Sabou [:CosminS]]

Latest windows aarch64 failures are from: https://hg.mozilla.org/mozilla-central/rev/25c4c2685699da87e8859aa66f650977fb013cf1#l477.9

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&selectedJob=272001358&searchStr=windows%2C10%2Caarch64%2Copt%2Cweb%2Cplatform%2Ctests%2Ctest-windows10-aarch64%2Fopt-web-platform-tests-e10s-11%2Cw%28wpt11%29&revision=25c4c2685699da87e8859aa66f650977fb013cf1

Comment 37

[Intermittent Failures Robot]

48 failures in 3964 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-inbound: 5
• try: 2
• autoland: 31
• mozilla-central: 10

Platform breakdown:

• windows10-64: 25
• macosx1014-64: 12
• macosx1014-64-shippable: 4
• windows10-64-ccov: 2
• windows7-32: 1
• windows10-aarch64: 3
• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 38

[Intermittent Failures Robot]

41 failures in 4182 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #43 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 24
• mozilla-central: 12
• mozilla-inbound: 2
• try: 3

Platform breakdown:

• macosx1014-64: 17
• macosx1014-64-shippable: 5
• linux64: 1
• windows10-aarch64: 12
• windows7-32: 4
• linux64-qr: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 39

[Razvan Maries]

There are 39 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-10-21&endday=2019-10-28&tree=trunk&bug=1541192

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=273210669&repo=autoland&lineNumber=7252

Comment 40

[Intermittent Failures Robot]

38 failures in 4314 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #50 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 28
• mozilla-central: 7
• mozilla-release: 1
• mozilla-inbound: 1
• try: 1

Platform breakdown:

• macosx1014-64-shippable: 13
• linux64: 2
• windows10-aarch64: 6
• windows7-32: 10
• macosx1014-64: 6
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-10-28&endday=2019-11-03&tree=all

Comment 41

[Intermittent Failures Robot]

23 failures in 4850 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 19
• mozilla-esr68: 1
• mozilla-central: 2
• mozilla-release: 1

Platform breakdown:

• macosx1014-64-shippable: 8
• macosx1014-64: 10
• windows10-64-shippable: 2
• windows7-32: 1
• linux64-shippable-qr: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 42

[Intermittent Failures Robot]

26 failures in 4207 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 21
• mozilla-central: 3
• mozilla-inbound: 1

Platform breakdown:

• windows10-64-qr: 1
• macosx1014-64: 6
• windows7-32: 7
• macosx1014-64-shippable: 8
• linux64: 3
• linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-11-11&endday=2019-11-17&tree=all

Comment 43

[Alexandru Michis [:malexandru]]

Jens,
There have been 26 failures in the last 7 days on all platforms, could you please take a look?
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=276692084&repo=autoland&lineNumber=7165

Comment 44

[Jens Stutte [:jstutte]]

Perry, please check if "Clearing Perry needinfo since Jan triaged appropriately and this is slightly lower than the set of SW e10s regressions that just happened." still applies or if you can move over to this soon.

Comment 45

[Intermittent Failures Robot]

22 failures in 5444 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18
• try: 1
• mozilla-central: 2
• mozilla-esr68: 1

Platform breakdown:

• macosx1014-64-shippable: 12
• macosx1014-64: 7
• linux64: 2
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-11-18&endday=2019-11-24&tree=all

Comment 46

[Intermittent Failures Robot]

21 failures in 4331 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18
• try: 1
• mozilla-central: 2

Platform breakdown:

• macosx1014-64: 7
• windows7-32: 4
• macosx1014-64-shippable: 7
• linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 47

[Perry Jiang [:perry] [no longer employee, use ni?]]

I tried to figure out what could be wrong without being able to reproduce the failure and agree with Andrew's comment 35. In particular, it seems that all the setting of HTTP headers and consulting the cache is delegated to the networking layer, which is what the test is mostly concerned about. This is still a bug but I think it may make sense to change this to P3 because it doesn't appear to be a very breaking issue and is difficult to get more details on without a recording of a failure. So I'd say that "this is slightly lower than the set of SW e10s regressions" still applies.

Comment 48

[Intermittent Failures Robot]

16 failures in 5144 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• try: 3
• mozilla-central: 2

Platform breakdown:

• macosx1014-64-shippable: 4
• windows7-32: 5
• linux64: 2
• macosx1014-64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-12-02&endday=2019-12-08&tree=all

Comment 49

[Intermittent Failures Robot]

1 failures in 4867 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 50

[Intermittent Failures Robot]

1 failures in 4582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 51

[Intermittent Failures Robot]

1 failures in 1365 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 52

[Intermittent Failures Robot]

1 failures in 4045 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2020-02-17&endday=2020-02-23&tree=all

Comment 53

[Intermittent Failures Robot]

1 failures in 5483 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2020-03-02&endday=2020-03-08&tree=all

Comment 54

[Intermittent Failures Robot]

1 failures in 5423 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 55

[Intermittent Failures Robot]

1 failures in 4035 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 56

[Intermittent Failures Robot]

1 failures in 4516 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1541192&startday=2020-06-29&endday=2020-07-05&tree=all

Comment 57

[Jens Stutte [:jstutte]]

Bulk closing some old intermittents.