1553797 - Perma - late beta - TEST-UNEXPECTED-FAIL | services/common/tests/unit/test_blocklist_onecrl.js | xpcshell return code: 0 when Gecko 69 merges to Beta on 19-07-01

Status: VERIFIED FIXED

(Cloud Services :: Firefox: Common, defect, P1)

Description

[Daniel Varga [:dvarga]]

[Tracking Requested - why for this release]:

Central-as-beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=e4badc6e8243638ecba464c3ae8b49a6626a502f&selectedJob=247949834

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=247949834&repo=try&lineNumber=2460

[task 2019-05-23T11:42:56.897Z] 11:42:56     INFO -  TEST-START | services/common/tests/unit/test_blocklist_onecrl.js
[task 2019-05-23T11:42:58.795Z] 11:42:58  WARNING -  TEST-UNEXPECTED-FAIL | services/common/tests/unit/test_blocklist_onecrl.js | xpcshell return code: 0
[task 2019-05-23T11:42:58.796Z] 11:42:58     INFO -  TEST-INFO took 1896ms
[task 2019-05-23T11:42:58.796Z] 11:42:58     INFO -  >>>>>>>
[task 2019-05-23T11:42:58.797Z] 11:42:58     INFO -  PID 16398 | [16398, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2561
[task 2019-05-23T11:42:58.798Z] 11:42:58     INFO -  PID 16398 | [16398, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, kKnownEsrVersion) failed with result 0x80004002: file /builds/worker/workspace/build/src/toolkit/components/resistfingerprinting/nsRFPService.cpp, line 662
[task 2019-05-23T11:42:58.798Z] 11:42:58     INFO -  PID 16398 | [16398, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file /builds/worker/workspace/build/src/extensions/permissions/nsPermissionManager.cpp, line 2906
[task 2019-05-23T11:42:58.799Z] 11:42:58     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-05-23T11:42:58.799Z] 11:42:58     INFO -  (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2019-05-23T11:42:58.800Z] 11:42:58     INFO -  (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-05-23T11:42:58.800Z] 11:42:58     INFO -  running event loop
[task 2019-05-23T11:42:58.800Z] 11:42:58     INFO -  services/common/tests/unit/test_blocklist_onecrl.js | Starting test_uses_a_custom_signer
[task 2019-05-23T11:42:58.800Z] 11:42:58     INFO -  (xpcshell/head.js) | test test_uses_a_custom_signer pending (2)
[task 2019-05-23T11:42:58.801Z] 11:42:58     INFO -  TEST-PASS | services/common/tests/unit/test_blocklist_onecrl.js | test_uses_a_custom_signer - [test_uses_a_custom_signer : 11] "onecrl.content-signature.mozilla.org" != "remote-settings.content-signature.mozilla.org"
[task 2019-05-23T11:42:58.801Z] 11:42:58     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2019-05-23T11:42:58.801Z] 11:42:58     INFO -  (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2019-05-23T11:42:58.802Z] 11:42:58     INFO -  (xpcshell/head.js) | test test_uses_a_custom_signer finished (2)
[task 2019-05-23T11:42:58.803Z] 11:42:58     INFO -  services/common/tests/unit/test_blocklist_onecrl.js | Starting test_has_initial_dump
[task 2019-05-23T11:42:58.803Z] 11:42:58     INFO -  (xpcshell/head.js) | test test_has_initial_dump pending (2)
[task 2019-05-23T11:42:58.804Z] 11:42:58     INFO -  (xpcshell/head.js) | test run_next_test 1 finished (2)
[task 2019-05-23T11:42:58.804Z] 11:42:58     INFO -  TEST-PASS | services/common/tests/unit/test_blocklist_onecrl.js | test_has_initial_dump - [test_has_initial_dump : 15] true == true
[task 2019-05-23T11:42:58.808Z] 11:42:58     INFO -  (xpcshell/head.js) | test run_next_test 2 pending (2)
[task 2019-05-23T11:42:58.809Z] 11:42:58     INFO -  (xpcshell/head.js) | test test_has_initial_dump finished (2)
[task 2019-05-23T11:42:58.809Z] 11:42:58     INFO -  services/common/tests/unit/test_blocklist_onecrl.js | Starting test_default_jexl_filter_is_used
[task 2019-05-23T11:42:58.810Z] 11:42:58     INFO -  (xpcshell/head.js) | test test_default_jexl_filter_is_used pending (2)

Comment 1

[Mathieu Leplatre [:leplatrem]]

This test fails because of this:
https://hg.mozilla.org/mozilla-central/rev/ee7e04b12dd3#l4.290

The test assumes that revocations.txt exists in the profile, and apparently in this case it doesn't.

If it makes sense, we could add a safety check, but I'd like to get a confirmation from the PSM team that it's acceptable for this file to be missing in some situations.

Comment 2

[Mathieu Leplatre [:leplatrem]]

I could reproduce the failure locally by forcing MOZ_NEW_CERT_STORAGE
https://searchfox.org/mozilla-central/rev/952521e6164ddffa3f34bc8cfa5a81afc5b859c4/old-configure.in#1695-1698

But the test is supposed to be skipped when MOZ_NEW_CERT_STORAGE is true:
https://hg.mozilla.org/mozilla-central/rev/ee7e04b12dd3#l4.285

I'm not entirely sure it's related to my patch. I think I would need some help from Dana and Myk, who introduced this in https://bugzilla.mozilla.org/show_bug.cgi?id=1547877

Thanks 🙏

Comment 3

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1553797 - Fix OneCRL tests when MOZ_NEW_CERT_STORAGE is disabled

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

That looks correct.

Comment 5

[Mathieu Leplatre [:leplatrem]]

Is there a way to run a try build for this patch on the beta tree?

Comment 6

[Mathieu Leplatre [:leplatrem]]

When I try to compile Firefox locally by forcing MOZ_NEW_CERT_STORAGE=0 in old-configure.in, I get this error NSSCertDBTrustDomain.h:17:12: fatal error: 'nsICertBlocklist.h' file not found.

I'd like to at least be able to test the patch somewhere before requesting landing :)

Comment 7

[Mark Goodwin [:mgoodwin]]

(In reply to Mathieu Leplatre [:leplatrem] from comment #6)

When I try to compile Firefox locally by forcing MOZ_NEW_CERT_STORAGE=0 in old-configure.in, I get this error NSSCertDBTrustDomain.h:17:12: fatal error: 'nsICertBlocklist.h' file not found.

I'd like to at least be able to test the patch somewhere before requesting landing :)

This would be because nsICertBlocklist.h isn't generated; if you've done previous builds there might be old stuff hanging around (see https://searchfox.org/mozilla-beta/source/security/manager/ssl/moz.build#184) maybe you need to clobber?

Comment 8

[Myk Melez [:myk] [@mykmelez]]

(In reply to Mathieu Leplatre [:leplatrem] from comment #6)

When I try to compile Firefox locally by forcing MOZ_NEW_CERT_STORAGE=0 in old-configure.in, I get this error NSSCertDBTrustDomain.h:17:12: fatal error: 'nsICertBlocklist.h' file not found.

I built beta, made this change, then built again:

diff --git a/old-configure.in b/old-configure.in
index ae37bffd5153d..60adaae960d24 100644
--- a/old-configure.in
+++ b/old-configure.in
@@ -1687,20 +1687,20 @@ if test "$BUILDING_RELEASE"; then
   # Override value in defines.sh, if any
   EARLY_BETA_OR_EARLIER=
 elif test "$EARLY_BETA_OR_EARLIER"; then
   AC_DEFINE(EARLY_BETA_OR_EARLIER)
 fi
 AC_SUBST(EARLY_BETA_OR_EARLIER)
 
 
-if test "$EARLY_BETA_OR_EARLIER"; then
-    MOZ_NEW_CERT_STORAGE=1
-    AC_DEFINE(MOZ_NEW_CERT_STORAGE)
-fi
+# if test "$EARLY_BETA_OR_EARLIER"; then
+#     MOZ_NEW_CERT_STORAGE=1
+#     AC_DEFINE(MOZ_NEW_CERT_STORAGE)
+# fi
 AC_SUBST(MOZ_NEW_CERT_STORAGE)
 
 # Allow someone to change MOZ_APP_NAME and MOZ_APP_BASENAME in mozconfig
 MOZ_ARG_WITH_STRING(app-name,
 [--with-app-name=APPNAME sets MOZ_APP_NAME to APPNAME],
 WITH_APP_NAME=$withval,
 )

And I didn't see the same failure, although I did see this one:

Undefined symbols for architecture x86_64:
  "construct_cert_storage(nsISupports*, nsID const&, void**)", referenced from:
      mozilla::xpcom::CreateInstanceImpl(mozilla::xpcom::ModuleID, nsISupports*, nsID const&, void**) in StaticComponents.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

However, after clobbering and reverting the change, I was unable to reproduce the problem, even after reapplying/reverting the change and rebuilding repeatedly (without clobbering in between builds). So I don't know why the change originally failed, nor why the build failure I saw is different from the one you saw.

Nevertheless, as @mgoodwin notes, you might resolve it by clobbering.

Comment 9

[Mathieu Leplatre [:leplatrem]]

I tried this morning to modify old-configure.in on trunk, clobber and build, and still fatal error: 'nsICertBlocklist.h' file not found.

If I checkout FIREFOX_BETA_68_BASE, there's no MOZ_NEW_CERT_STORAGE in old-configure.in. So I modified toolkit/moz.configure, clobbered and recompiled. Then had other errors like use of undeclared identifier 'SK_R16_MASK'

I added this test to be thorough, but start to regret it...

If you have a local build with the old storage, could you please try the attached patch please?

Comment 10

[Myk Melez [:myk] [@mykmelez]]

(In reply to Mathieu Leplatre [:leplatrem] from comment #9)

I tried this morning to modify old-configure.in on trunk, clobber and build, and still fatal error: 'nsICertBlocklist.h' file not found.

It'd be easier to reproduce this failure if you provided the specific changes you're making to old-configure.in.

If I checkout FIREFOX_BETA_68_BASE, there's no MOZ_NEW_CERT_STORAGE in old-configure.in. So I modified toolkit/moz.configure, clobbered and recompiled. Then had other errors like use of undeclared identifier 'SK_R16_MASK'

I added this test to be thorough, but start to regret it...

If you have a local build with the old storage, could you please try the attached patch please?

If I apply the patch in comment 8 to the tip of mozilla-central and also apply your patch in https://phabricator.services.mozilla.com/D32331, then build and run the test services/common/tests/unit/test_blocklist_onecrl.js, it succeeds.

Comment 11

[Intermittent Failures Robot]

1 failures in 4730 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• macosx1010-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1553797&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 12

[Mathieu Leplatre [:leplatrem]]

It'd be easier to reproduce this failure if you provided the specific changes you're making to old-configure.in.

Oh sorry :/ Basically I commented the few lines that sets MOZ_NEW_CERT_STORAGE to 1, like you did.

If I apply the patch in comment 8 to the tip of mozilla-central and also apply your patch in https://phabricator.services.mozilla.com/D32331, then build and run the test services/common/tests/unit/test_blocklist_onecrl.js, it succeeds.

Ok, excellent news :) Let's land this then!

Thank you!

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7c8a87a7e190
Fix OneCRL tests when MOZ_NEW_CERT_STORAGE is disabled r=keeler

Comment 14

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/7c8a87a7e190

Comment 15

[Cristina Coroiu [:ccoroiu]]

Verified as fixed with this beta sim:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=3beb3e0943823cac308727d2e7234fe05a341688

Comment 16

[Intermittent Failures Robot]

2 failures in 4337 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows7-32-shippable: 1
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1553797&startday=2019-05-27&endday=2019-06-02&tree=all