Open Bug 1434754 Opened 6 years ago Updated 1 year ago

Intermittent TEST-UNEXPECTED-TIMEOUT | /WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html | expected OK

Categories

(Core :: DOM: Web Crypto, defect, P5)

defect

Tracking

()

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure, leave-open, Whiteboard: [stockwell disabled][domsecurity-intermittent])

Attachments

(1 file, 6 obsolete files)

In the last 7 days, there have been 31 failures.

All the failures are on windows10-64-ccov, debug build type.

An example of a recent log file:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=164121454&lineNumber=1061

And the relevant part of the log:
12:20:33     INFO - TEST-START | /WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html
12:20:34     INFO - PID 4700 | ++DOCSHELL 000002534EAE0800 == 4 [pid = 2672] [id = {582462c8-199b-40d0-b244-b4943c1f3543}]
12:20:34     INFO - PID 4700 | ++DOMWINDOW == 7 (000002534E039000) [pid = 2672] [serial = 15] [outer = 0000000000000000]
12:20:34     INFO - PID 4700 | ++DOMWINDOW == 8 (000002534E78A800) [pid = 2672] [serial = 16] [outer = 000002534E039000]
12:20:34     INFO - PID 4700 | ++DOMWINDOW == 9 (000002534E039400) [pid = 2672] [serial = 17] [outer = 000002534E039000]
12:20:39     INFO - PID 4700 | [Parent 4700, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
12:20:48     INFO - PID 4700 | --DOCSHELL 000002534E470800 == 3 [pid = 2672] [id = {5c893b35-b404-45ee-be5d-753e5f373d05}]
12:20:50     INFO - PID 4700 | --DOMWINDOW == 8 (000002534E03BC00) [pid = 2672] [serial = 12] [outer = 0000000000000000] [url = https://web-platform.test:8443/WebCryptoAPI/derive_bits_keys/ecdh_keys.https.worker.html]
12:20:56     INFO - PID 4700 | --DOMWINDOW == 7 (000002534E78A400) [pid = 2672] [serial = 14] [outer = 0000000000000000] [url = about:blank]
12:20:56     INFO - PID 4700 | --DOMWINDOW == 6 (000002534E78A800) [pid = 2672] [serial = 16] [outer = 0000000000000000] [url = about:blank]
12:21:23     INFO - TEST-UNEXPECTED-TIMEOUT | /WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html | expected OK
12:21:23     INFO - TEST-INFO took 50063ms
12:21:26     INFO - PID 4700 | [Parent 4700, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
12:22:24     INFO - Browser exited with return code 572
12:22:24  WARNING - u'runner_teardown': ()
12:22:25     INFO - Setting up ssl
12:22:25     INFO - certutil | 
12:22:25     INFO - certutil | 
12:22:26     INFO - certutil | 
12:22:26     INFO - Certificate Nickname                                         Trust Attributes
12:22:26     INFO -                                                              SSL,S/MIME,JAR/XPI
12:22:26     INFO - 
12:22:26     INFO - web-platform-tests                                           CT,, 
12:22:26     INFO - 
12:22:26     INFO - Application command: Z:\task_1519474331\build\application\firefox\firefox.exe -marionette about:blank -profile c:\users\genericworker\appdata\local\temp\tmpcxtghj.mozrunner
12:22:26     INFO - Starting runner
12:22:26     INFO - PID 7848 | [7848, Main Thread] WARNING: Failed to load startupcache file correctly, removing!: file z:/build/build/src/startupcache/StartupCache.cpp, line 217
12:22:30     INFO - PID 7848 | ++DOCSHELL 0000020089A76000 == 1 [pid = 7848] [id = {f5f6b1fb-2489-4879-8e09-ea6a5a375557}]
12:22:30     INFO - PID 7848 | ++DOMWINDOW == 1 (000002008AC63000) [pid = 7848] [serial = 1] [outer = 0000000000000000]
12:22:30     INFO - PID 7848 | ++DOMWINDOW == 2 (000002008AC64000) [pid = 7848] [serial = 2] [outer = 000002008AC63000]
12:22:30     INFO - PID 7848 | ++DOCSHELL 0000020089A8D800 == 2 [pid = 7848] [id = {76bbfbba-3325-4f9a-ad50-5aac22f6536d}]
12:22:30     INFO - PID 7848 | ++DOMWINDOW == 3 (000002008ACF2000) [pid = 7848] [serial = 3] [outer = 0000000000000000]
12:22:30     INFO - PID 7848 | ++DOMWINDOW == 4 (000002008ACF3800) [pid = 7848] [serial = 4] [outer = 000002008ACF2000]
12:22:31     INFO - PID 7848 | ++DOCSHELL 000002008C682800 == 3 [pid = 7848] [id = {371f2bab-20aa-4eda-909b-dcd6b169dbd0}]
12:22:31     INFO - PID 7848 | ++DOMWINDOW == 5 (000002008850CC00) [pid = 7848] [serial = 5] [outer = 0000000000000000]
12:22:31     INFO - PID 7848 | ++DOMWINDOW == 6 (000002008ACF7800) [pid = 7848] [serial = 6] [outer = 000002008850CC00]
12:22:31     INFO - PID 7848 | ++DOMWINDOW == 7 (000002008AC60C00) [pid = 7848] [serial = 7] [outer = 000002008ACF2000]
12:22:32     INFO - PID 7848 | ++DOCSHELL 000002008CCA2000 == 4 [pid = 7848] [id = {cf36bbdb-2021-49ad-a4a0-18aab6f0d1bc}]
12:22:32     INFO - PID 7848 | ++DOMWINDOW == 8 (000002008C6B1C00) [pid = 7848] [serial = 8] [outer = 0000000000000000]

:jgraham could you please take a look?
Flags: needinfo?(james)
Whiteboard: [stockwell needswork]
we should disable this for win10 ccov, it is often the case that ccov builds cause tests to behave much differently, luckily it is only a few there.
Component: web-platform-tests → DOM: Security
Flags: needinfo?(james)
Product: Testing → Core
Version: Version 3 → unspecified
now that this is in the right component... :)

:ckerschb, could you help validate that this test is ok to disable on windows code coverage builds?
Flags: needinfo?(ckerschb)
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #7)
> now that this is in the right component... :)
> 
> :ckerschb, could you help validate that this test is ok to disable on
> windows code coverage builds?

302 to :ttaubert who is in a better position to answer that question.
Flags: needinfo?(ckerschb) → needinfo?(ttaubert)
If disabling it only on win ccov build is a quick fix, that'd be fine with me. It's still included in general Windows runs and other platforms.
Flags: needinfo?(ttaubert)
Whiteboard: [stockwell needswork] → [stockwell needswork][domsecurity-intermittent]
Assignee: nobody → aiakab
Attachment #8956663 - Flags: review?(jmaher)
Comment on attachment 8956663 [details] [diff] [review]
Disabled On Windows 10 ccov and debug

Review of attachment 8956663 [details] [diff] [review]:
-----------------------------------------------------------------

just a small tweak, otherwise this looks good.

::: testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html.ini
@@ +1,3 @@
> +[hkdf.https.worker.html]
> +  disabled:
> +    if ccov and debug and (os == "win") and (version == "10.0.15063"): https://bugzilla.mozilla.org/show_bug.cgi?id=1434754

we could simplify this and just do:
if ccov and (os == "win")

the reason for this is that ccov is debug for windows and we only run on windows 10.  I do not see us having an option to run on non debug, or running on other versions of windows.
Attachment #8956663 - Flags: review?(jmaher) → review-
Attachment #8957559 - Flags: review?(jmaher)
Attachment #8956663 - Attachment is obsolete: true
Attachment #8957559 - Flags: review?(jmaher) → review+
This Bug has failed 36 times on Windows 10 ccov platform affecting debug build type. 
Link to a recent log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=168625485&lineNumber=1056
Part of that bug:
21:17:29     INFO - TEST-UNEXPECTED-TIMEOUT | /WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html | expected OK
21:17:29     INFO - TEST-INFO took 50061ms
21:17:36     INFO - PID 4188 | [Parent 4188, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
21:18:30     INFO - Browser exited with return code 572
21:18:30  WARNING - u'runner_teardown': ()
21:18:31     INFO - Setting up ssl
21:18:31     INFO - certutil | 
21:18:31     INFO - certutil | 
21:18:31     INFO - certutil | 
21:18:31     INFO - Certificate Nickname                                         Trust Attributes
21:18:31     INFO -                                                              SSL,S/MIME,JAR/XPI
21:18:31     INFO - 
21:18:31     INFO - web-platform-tests                                           CT,, 

:jmaher should we land the disabling patch?
Flags: needinfo?(jmaher)
yes, please land this.
Flags: needinfo?(jmaher)
The attached patch is missing the author information. Please update it to include all necessary commit information and re-request checkin.
Flags: needinfo?(aiakab)
Keywords: checkin-needed
I did the update. Please check if the author information is present.
Attachment #8962011 - Flags: review?(jmaher)
Flags: needinfo?(aiakab)
Keywords: checkin-needed
Comment on attachment 8962011 [details] [diff] [review]
Disabled on Windows 10 ccov platform

Review of attachment 8962011 [details] [diff] [review]:
-----------------------------------------------------------------

when I have more time I will look into if ccov is available.

::: testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html.ini
@@ +1,3 @@
> +[hkdf.https.worker.html]
> +  disabled:
> +    if ccov and (os == "win"): https://bugzilla.mozilla.org/show_bug.cgi?id=1434754

I am not sure if we have 'ccov' available to use for web-platform-tests, I am not finding prior art to compare against.
Comment on attachment 8962011 [details] [diff] [review]
Disabled on Windows 10 ccov platform

from what I can tell 'ccov' will work in web-platform-tests.
Attachment #8962011 - Flags: review?(jmaher) → review+
Attachment #8957559 - Attachment is obsolete: true
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/41afc070bb71
disabling hkdf.https.worker.html on Windows 10 ccov platforms r=jmaher
Whiteboard: [stockwell needswork][domsecurity-intermittent] → [stockwell disabled][domsecurity-intermittent]
Updates:
There have been 31 total failures in the last 7 days.
Fails only on windows10-64-ccov / debug.

Recent log file and snippet with the failure:
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=173373765&lineNumber=1055

22:08:38     INFO - PID 5544 | [Parent 5544, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
22:08:49     INFO - PID 5544 | --DOCSHELL 000001FBA1654000 == 3 [pid = 5760] [id = {234e04fb-cab6-4208-bb94-0a51ec6291f9}]
22:08:51     INFO - PID 5544 | --DOMWINDOW == 8 (000001FBA9093000) [pid = 5760] [serial = 12] [outer = 0000000000000000] [url = https://web-platform.test:8443/WebCryptoAPI/derive_bits_keys/ecdh_keys.https.worker.html]
22:08:57     INFO - PID 5544 | --DOMWINDOW == 7 (000001FBA9094400) [pid = 5760] [serial = 16] [outer = 0000000000000000] [url = about:blank]
22:08:57     INFO - PID 5544 | --DOMWINDOW == 6 (000001FBA9093800) [pid = 5760] [serial = 14] [outer = 0000000000000000] [url = about:blank]
22:09:22     INFO - TEST-UNEXPECTED-TIMEOUT | /WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html | expected OK
22:09:22     INFO - TEST-INFO took 50065ms
22:09:33     INFO - PID 5544 | [Parent 5544, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
22:10:24     INFO - Browser exited with return code 572
22:10:24  WARNING - u'runner_teardown': ()

:jmaher, could you please have a look? It has been disabled Windows 10 ccov , but it seems to continue to fail on that.
Flags: needinfo?(jmaher)
Tiberius, this is interesting and took me a while to figure out.  This test case doesn't exist in-tree instead it is autogenerated, we need to delete the landed .ini file and add the disabled clause to:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.js.ini

Tiberius, do you want to do that?
Flags: needinfo?(jmaher) → needinfo?(toros)
I was a bit reluctant to proceed without guidance as I don't want to do more harm than good on this.
So, my approach on this was:

- I've modified testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.js.ini by adding this to my local file:
disabled:
    if ccov and (os == "win"): https://bugzilla.mozilla.org/show_bug.cgi?id=1434754

- to remove and replace the patch
hg qnew bug1434754
hg rm testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html.ini
hg edit testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.js.ini
hg qref
hg qref -e (to add the commit message)

- after the new patch is created on my local machine I could select the file and attach it to the bug.

:jmaher , do you think this is the proper way to proceed on this? Thank you.
Flags: needinfo?(jmaher)
Flags: needinfo?(toros)
Tiberius, that looks like the proper way, looking forward to the review.
Flags: needinfo?(jmaher)
Attached patch skipccov_bug1434754 (obsolete) — Splinter Review
Attachment #8968758 - Flags: review?(jmaher)
Comment on attachment 8968758 [details] [diff] [review]
skipccov_bug1434754

Review of attachment 8968758 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.js.ini
@@ +1,3 @@
>  [hkdf.https.worker.html]
>    disabled:
>      if ccov and (os == "win"): https://bugzilla.mozilla.org/show_bug.cgi?id=1434754

you might have a secondary patch somewhere else because this disabled line is not being added to your patch.
Attachment #8968758 - Flags: review?(jmaher) → review-
Attached patch disablewinccov_bug1434754 (obsolete) — Splinter Review
Attachment #8969580 - Flags: review?(gbrown)
Comment on attachment 8969580 [details] [diff] [review]
disablewinccov_bug1434754

Review of attachment 8969580 [details] [diff] [review]:
-----------------------------------------------------------------

I can verify that the condition (if ccov and ...) is correct, but I am unfamiliar with the need for 2 ini files. If you are pretty sure you know what you are doing, go ahead with check-in. Otherwise, double check with jgraham.

::: testing/web-platform/meta/WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html.ini
@@ +1,1 @@
> +[hkdf.https.worker.html]

Are you sure you need this file? What does it do?
Attachment #8969580 - Flags: review?(gbrown) → review+
Geoff, taking in consideration Comment 28 I wanted to remove the condition from the first .ini (hkdf.https.worker.html.ini) file and add it to the the correct one (hkdf.https.worker.js.ini). Thank you for the review.
Attachment #8962011 - Attachment is obsolete: true
Attachment #8968758 - Attachment is obsolete: true
Attached patch disabled test on win10 ccov (obsolete) — Splinter Review
Attachment #8969580 - Attachment is obsolete: true
Attachment #8970875 - Flags: review?(jmaher)
Attachment #8970875 - Attachment is obsolete: true
Attachment #8970875 - Flags: review?(jmaher)
Attachment #8970878 - Flags: review?(jmaher)
Attachment #8970878 - Flags: review?(jmaher) → review+
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/06cba34e7186
Disable /WebCryptoAPI/derive_bits_keys/hkdf.https.worker.html on windows10 ccov. r=jmaher
Keywords: checkin-needed
Component: DOM: Security → DOM: Web Crypto
Assignee: aiakab → nobody
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.