1670712 - Perma abort: certificate for hg.mozilla.org has unexpected fingerprint sha256:ff:e7:8d:93:e9:56:3c:c0:19:fc:00:4c:18:b9:86:e5:08:e5:10:f5:e2:ea:48:e8:22:d3:a3:3a:ca:99:c3:4c

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=318396417&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PTqXBh-WQW6MDzkN3mUy-Q/runs/0/artifacts/public/logs/live_backing.log

---

[vcs 2020-10-12T15:36:50.418Z] Abort: certificate for hg.mozilla.org has unexpected fingerprint sha256:ff:e7:8d:93:e9:56:3c:c0:19:fc:00:4c:18:b9:86:e5:08:e5:10:f5:e2:ea:48:e8:22:d3:a3:3a:ca:99:c3:4c
[vcs 2020-10-12T15:36:50.418Z] abort: certificate for hg.mozilla.org has unexpected fingerprint sha256:ff:e7:8d:93:e9:56:3c:c0:19:fc:00:4c:18:b9:86:e5:08:e5:10:f5:e2:ea:48:e8:22:d3:a3:3a:ca:99:c3:4c
[vcs 2020-10-12T15:36:50.418Z] (check hostsecurity configuration)
[taskcluster 2020-10-12T15:36:50.436Z]    Exit Code: 255
[taskcluster 2020-10-12T15:36:50.436Z]    User Time: 920ms
[taskcluster 2020-10-12T15:36:50.436Z]  Kernel Time: 3.16s
[taskcluster 2020-10-12T15:36:50.436Z]    Wall Time: 6.560897513s
[taskcluster 2020-10-12T15:36:50.436Z]       Result: FAILED
[taskcluster 2020-10-12T15:36:50.436Z] === Task Finished ===
[taskcluster 2020-10-12T15:36:50.436Z] Task Duration: 6.562953204s
[taskcluster 2020-10-12T15:36:50.436Z] [mounts] Preserving cache: Moving "/home/cltbld/tasks/task_1602511779/checkouts" to "/home/cltbld/caches/cZxh_GYNR_-ArMKAwrlUYw"
[taskcluster:error] exit status 255```

Comment 1

[Razvan Maries]

Connor, could you take a look?

Comment 2

[Connor Sheehan [:sheehan]]

Created https://github.com/mozilla-releng/build-puppet/pull/688 which should fix this.

Comment 3

[(Away)]

It looks like Windows and Mac machines haven't yet picked up the new fingerprint, is it expected that they'd have a longer lead time? https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=run&fromchange=fd3271b959b9eef4d457d1995f9adf901336d27a

Comment 4

[Aki Sasaki (not active)]

Currently:

OCC required additional action to deploy {gecko,comm}-{1,2,3}/b-win2012 images with the new fingerprint. Levels 1 and 2 (untrusted) are now rebuilt and are going green. Level 3 requires a CoT key. :markco doesn't have this key and doesn't know how to create a new image with the trusted CoT key. This means that our trusted level 3 builders are stuck on the old hg.m.o fingerprint, which resulted in the Firefox 82.0 RC graph being stuck.

I saw three potential solutions:

1. Roll back the fingerprint change. Because the new images have both the new and previous fingerprints, they should still work. Because the previous image has the previous fingerprint, those should start working again. However, the fingerprint will expire tomorrow. This is not a good solution after midnight UTC tonight.
2. Find the current CoT private key, figure out how to add it to the image (populate the taskcluster secret? what format should this be in?), and rerun the level 3 image builder task. This may need to wait for :grenade.
3. Generate a new CoT keypair, add it to scriptworker.constants, roll out a new scriptworker release + k8s images, then populate the new builder images with the new private key.

It looks like we're doing a combination of (1) and (2). :dhouse rolled back the fingerprint change to unblock Firefox 82.0 RC. And :markco will check with :grenade on how to roll out the images with the existing CoT key. (3) can be a fallback option if we no longer have the existing CoT key. We likely won't be able to fully resolve this issue until tomorrow (October 13), but we have some workarounds.

[EDIT] After this is done, we should update the documentation about how to roll out a new hg.m.o fingerprint with these additional steps.

Related: the artifact metadata RFC describes how we can obsolete the Chain of Trust keys, simplifying maintenance.

Comment 5

[:dhouse]

Attachment: mac workers update

I'm checking+updating the mac workers. I had not updated the fingerprint.

Comment 6

[Aki Sasaki (not active)]

The fingerprint expires at noon UTC on Tuesday, October 13. That gives us a few hours for :grenade to rebuild the image once he gets in, and then for us to flip the hg.m.o fingerprint to the new fingerprint. We might end up with an hour or more of tree closure, depending on timings.

Comment 7

[Aki Sasaki (not active)]

(Not sure if those are the right priority/severity. I'm looking for a soon-to-be-blocker level.)

Comment 8

[Aki Sasaki (not active)]

So far:

• gecko-3/b-win2012
• gecko-t/t-win10-64-source
• releng-hardware/gecko-3-t-osx-1014

need the new hg.m.o fingerprint.

(We flipped back to the new cert to identify those busted workerTypes. Now we've flipped back to the old cert, in case we need an 82.0rc build 2. We should fix the above workers then reenable the new cert, ideally before noon UTC.)

Comment 9

[Intermittent Failures Robot]

17 failures in 622 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 7
• mozilla-central: 8
• try: 2

Platform breakdown:

• linux64-shippable: 9
• windows2012-32-shippable: 2
• macosx1014-64-shippable: 2
• windows2012-64-shippable: 2
• windows10-64: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-12&endday=2020-10-12&tree=all

Comment 10

[Rob Thijssen [:grenade (EET/UTC+0300)]]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #8)

• are we sure gecko-3/b-win2012 is actually being used? i'm rebuilding amis anyway, but was under the impression that we had migrated all win builds to run as cross compiles on linux.
• gecko-t/t-win10-64-source was a custom build by releng (not built or maintained by relops). i don't know who made it. the best i can do there is rebuild gecko-t/t-win10-64 and update worker managers gecko-t/t-win10-64-source definition to use gecko-t/t-win10-64 amis. please shout if you know more about the origin of this worker type and any customisations it has.

Comment 11

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: updated gecko-3-b-win2012 amis

i have patched the gecko-3/b-win2012 worker configuration with amis built today using the new hg fingerprint.

the attached patch shows the changes i made with new and old ami-ids

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: patch

diff'ed the right way round this time...

Comment 13

[:dhouse]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #8)

So far:

• gecko-3/b-win2012
• gecko-t/t-win10-64-source
• releng-hardware/gecko-3-t-osx-1014

need the new hg.m.o fingerprint.

for releng-hardware/gecko-3-t-osx-1014, I rebooted and applied the updated configuration and verified they all would not fail on the new cert fingerprint (new config disabled pinning to the fingerprint).

(We flipped back to the new cert to identify those busted workerTypes. Now we've flipped back to the old cert, in case we need an 82.0rc build 2. We should fix the above workers then reenable the new cert, ideally before noon UTC.)

We flipped back to the new cert at 11:36utc. The state of 82RC and 81.0.2 were confirmed as past needing the old cert fingerprint.

Comment 14

[(Away)]

• are we sure gecko-3/b-win2012 is actually being used? i'm rebuilding amis anyway, but was under the impression that we had migrated all win builds to run as cross compiles on linux.

There are still a few build flavors that have not yet been migrated to cross compiles (and even then we'll want to keep at least one flavor on native builds, to monitor for bustage affecting local development).

Comment 15

[:dhouse]

Mark/Rob comm-3-b-win2012 failing may be related. rjl noted it in #firefox-ci (at 6:59, and follow-up at 7:13 pacific):

rjl

    if you're looking for busted hg.mo workers comm-3-b-win2012 

yeah those are still failing after a rerun.

Comment 16

[Rob Lemley [:rjl]]

(In reply to Dave House [:dhouse] from comment #15)

Mark/Rob comm-3-b-win2012 failing may be related. rjl noted it in #firefox-ci (at 6:59, and follow-up at 7:13 pacific):

rjl

    if you're looking for busted hg.mo workers comm-3-b-win2012 

yeah those are still failing after a rerun.

https://firefoxci.taskcluster-artifacts.net/fpov-VnKSnO8V0CSw6p6SA/0/public/logs/live_backing.log

That's from the nightly build, but we've got a beta and a release blocked on this.

(In reply to :dmajor from comment #14)

• are we sure gecko-3/b-win2012 is actually being used? i'm rebuilding amis anyway, but was under the impression that we had migrated all win builds to run as cross compiles on linux.

There are still a few build flavors that have not yet been migrated to cross compiles (and even then we'll want to keep at least one flavor on native builds, to monitor for bustage affecting local development).

MSI repacks run on b-2012 workers.

Comment 17

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: gecko-3/b-win2012 images with updated hg cert pin

Comment 18

[Rob Thijssen [:grenade (EET/UTC+0300)]]

the patch in comment 17 above makes my earlier jerry-rig patch stick. earlier, i only modified the worker configuration in worker manager. i know nothing about comm-3-b-win2012 but if it was based on gecko-3-b-win2012 and is trusted to use the same cot keys, it's definition should be updated to match the updtaed gecko-3-b-win2012 amis. eg:

• us-east-1: ami-09b9b07cb12f168f8
• us-west-1: ami-08c572f9f8b57d655
• us-west-2: ami-0e5ce0795a0efb58e

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/3c591b1d79a7
gecko-3/b-win2012 images with updated hg cert pin r=aki

Comment 20

[Aki Sasaki (not active)]

(In reply to Rob Thijssen [:grenade (EET/UTC+0300)] from comment #10)

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #8)

• are we sure gecko-3/b-win2012 is actually being used? i'm rebuilding amis anyway, but was under the impression that we had migrated all win builds to run as cross compiles on linux.

As mentioned in Slack, yes, we use it for PGO profile generation and MSI repackages.

• gecko-t/t-win10-64-source was a custom build by releng (not built or maintained by relops). i don't know who made it. the best i can do there is rebuild gecko-t/t-win10-64 and update worker managers gecko-t/t-win10-64-source definition to use gecko-t/t-win10-64 amis. please shout if you know more about the origin of this worker type and any customisations it has.

Looks like this is the same image that we use for t-win10-64, so if we have updated that AMI with the new fingerprint we should be good.

Comment 21

[Aki Sasaki (not active)]

Aiui, gecko-3/b-win2012 should be all good. t-win10-64-source may still need some attention, but aiui those are all tier 2 tasks; relops will look at that tomorrow.

Comment 22

[Cosmin Sabou [:CosminS]]

So the failures on t-win10-64-source look something like this on tree: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&searchStr=mbu&tochange=7cde57e7688b0b6bf9d87321a37c87b4dcf14b87&fromchange=ab495477524d6c7e527cfe344ea7d0c74bd54d7c&selectedTaskRun=QEjgcEsjTpKt2ZvU8pLjsA.0

Comment 23

[Intermittent Failures Robot]

23 failures in 681 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-esr78: 4
• try: 8
• mozilla-release: 2
• autoland: 9

Platform breakdown:

• windows2012-32-shippable: 5
• windows2012-64-shippable: 5
• windows2012-32-devedition: 2
• windows2012-64-devedition: 2
• macosx1014-64-qr: 1
• windows10-64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-13&endday=2020-10-13&tree=all

Comment 24

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

gecko-1/b-win2012 will also need the new fingerprint. Bpgo(run) for central-as-beta simulations are failing.

Comment 25

[Aki Sasaki (not active)]

Hm. gecko-1/b-win2012 were working the other day, with the new fingerprint. We may have a good AMI that we need to point at in ci-configuration.

Comment 26

[Mark Cornmesser [:markco]]

Attachment: Bug 1670712 - Update 2012 L2 builders AMIS. r?aki

Comment 27

[Mark Cornmesser [:markco]]

Attachment: Bug 1670712 - Update 2012 L2 builders AMIs r?aki

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/2d709690e13b
Update 2012 L2 builders AMIs r=aki

Comment 29

[Intermittent Failures Robot]

89 failures in 845 pushes (0.105 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 50
• try: 22
• mozilla-central: 17

Platform breakdown:

• windows10-64: 66
• windows10-64-qr: 6
• windows2012-64: 5
• windows2012-32-shippable: 6
• windows2012-64-shippable: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-14&endday=2020-10-14&tree=all

Comment 30

[Mark Cornmesser [:markco]]

What is the status of the gecko-1/b-win2012 builders?

Comment 31

[Aki Sasaki (not active)]

I see some missing y:/ errors here: https://firefox-ci-tc.services.mozilla.com/tasks/fS3h92UFRjiBm78ttr8kkw/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FfS3h92UFRjiBm78ttr8kkw%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=GYJkxrllQymKwEeW8RLdGg.0&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&searchStr=windows
We probably have to do the same y:/ fix for gecko-1 as we did for gecko-3.

[EDIT] yeah, grenade's patch included /dev/sdc fixes; mcornmesser's patch didn't.

Comment 32

[Aki Sasaki (not active)]

gecko-t/t-win10-64-source still appears broken: https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-t/worker-types/t-win10-64-source
That probably means we need to bump the AMI for occ-t-win10-64-current as we did occ-b-win2012{,-trusted}-current .

Comment 33

[Mark Cornmesser [:markco]]

Attachment: Bug 1670712 - /dev/sdc fixes 2012 L1 and L2 builders config. r?#releng-reviewers

Comment 34

[Mark Cornmesser [:markco]]

Submitted above patch to address comment 31.

gecko-t/t-win10-64-source AMIs are currently rebuilding .

Comment 35

[Rob Lemley [:rjl]]

Attachment: Bug 1670712 - /dev/sdc fixes for comm-1-b-win2012 workers. r?#releng-reviewers

Comment 36

[Intermittent Failures Robot]

49 failures in 714 pushes (0.069 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 26
• mozilla-central: 12
• try: 11

Platform breakdown:

• windows10-64: 48
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-15&endday=2020-10-15&tree=all

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/13c0e04c32a4
/dev/sdc fixes 2012 L1 and L2 builders config. r=releng-reviewers,aki
https://hg.mozilla.org/ci/ci-configuration/rev/4cbabd7b8f42
/dev/sdc fixes for comm-1-b-win2012 workers. r=releng-reviewers,aki

Comment 38

[Intermittent Failures Robot]

43 failures in 655 pushes (0.066 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 14
• try: 8
• autoland: 21

Platform breakdown:

• windows10-64: 38
• android-em-7-0-x86_64: 1
• macosx1014-64-qr: 1
• linux1804-64-qr: 1
• windows10-64-qr: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-16&endday=2020-10-16&tree=all

Comment 39

[Andreea Pavel [:apavel]]

No more failures here since October 17th.

Comment 40

[Intermittent Failures Robot]

234 failures in 3694 pushes (0.063 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 117
• mozilla-central: 55
• try: 56
• mozilla-esr78: 4
• mozilla-release: 2

Platform breakdown:

• linux64-shippable: 9
• windows2012-32-shippable: 13
• macosx1014-64-shippable: 2
• windows2012-64-shippable: 13
• windows10-64: 174
• macosx1014-64: 1
• windows2012-32-devedition: 2
• windows2012-64-devedition: 2
• macosx1014-64-qr: 2
• windows10-64-qr: 7
• windows2012-64: 5
• linux1804-64: 2
• android-em-7-0-x86_64: 1
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 41

[Joel Maher ( :jmaher ) (UTC -8)]

this hasn't ran on central since the 16th, and a few runs on autoland today (19th) are failing, I get this on try as a perma fail.

Comment 42

[Andrew Halberstadt [:ahal]]

Unsure if this is known, but just posting in case it's not.. All the Windows source-test tasks are still hitting this:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2Csource-test-python

They are typically using this docker image:
https://searchfox.org/mozilla-central/source/taskcluster/docker/lint

and the t-win10-64-source worker type.

Comment 43

[Mark Cornmesser [:markco]]

(In reply to Joel Maher ( :jmaher ) (UTC -0400) from comment #41)

this hasn't ran on central since the 16th, and a few runs on autoland today (19th) are failing, I get this on try as a perma fail.

Could you point me to some of the failures, please?

(In reply to Andrew Halberstadt [:ahal] from comment #42)

Unsure if this is known, but just posting in case it's not.. All the Windows source-test tasks are still hitting this:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2Csource-test-python

They are typically using this docker image:
https://searchfox.org/mozilla-central/source/taskcluster/docker/lint

and the t-win10-64-source worker type.
We are tracking that in Bug 1671731 .

Comment 44

[Joel Maher ( :jmaher ) (UTC -8)]

some examples:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=win%2Ctryselect&revision=729d218d29585492b1c0c07d8bf0233402c5c231

Comment 45

[Mark Cornmesser [:markco]]

(In reply to Joel Maher ( :jmaher ) (UTC -0400) from comment #44)

some examples:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=win%2Ctryselect&revision=729d218d29585492b1c0c07d8bf0233402c5c231

Specifically we are still seeing the failures on t-win10-64-source and not the Windows builders? We a re currently trying to figure out why the 10-64 AMIs are failing to build. I am going to try to sync up with grenade tonight (his morning) and get plan regarding this issue.

Comment 46

[Intermittent Failures Robot]

70 failures in 590 pushes (0.119 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 29
• mozilla-beta: 20
• try: 15
• mozilla-central: 6

Platform breakdown:

• windows10-64: 68
• linux1804-64: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-19&endday=2020-10-19&tree=all

Comment 47

[Aki Sasaki (not active)]

https://hg.mozilla.org/ci/ci-configuration/annotate/tip/worker-images.yml#l246 hasn't been updated with new AMIs. (occ-t-win10-64-current points at occ-t-win10-64-pre). I'm guessing if we update those AMIs we'll green up the busted t-win10-64-source tasks.

Comment 48

[Mark Cornmesser [:markco]]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #47)

https://hg.mozilla.org/ci/ci-configuration/annotate/tip/worker-images.yml#l246 hasn't been updated with new AMIs. (occ-t-win10-64-current points at occ-t-win10-64-pre). I'm guessing if we update those AMIs we'll green up the busted t-win10-64-source tasks.

The building the AMI is failing. Rob is currently debugging the process.

Comment 49

[Intermittent Failures Robot]

39 failures in 746 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 32
• mozilla-central: 7

Platform breakdown:

• windows10-64: 37
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-20&endday=2020-10-20&tree=all

Comment 50

[Intermittent Failures Robot]

107 failures in 878 pushes (0.122 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 79
• mozilla-central: 20
• try: 8

Platform breakdown:

• windows10-64: 103
• macosx1014-64-qr: 1
• linux1804-64-asan: 1
• windows2012-32: 1
• windows2012-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-21&endday=2020-10-21&tree=all

Comment 51

[Intermittent Failures Robot]

108 failures in 865 pushes (0.125 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 65
• mozilla-central: 24
• try: 3
• mozilla-esr78: 16

Platform breakdown:

• windows10-64: 105
• macosx1014-64: 1
• windows10-64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-22&endday=2020-10-22&tree=all

Comment 52

[Intermittent Failures Robot]

86 failures in 877 pushes (0.098 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 60
• mozilla-central: 24
• mozilla-beta: 1
• try: 1

Platform breakdown:

• windows10-64: 84
• windows10-64-qr: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-23&endday=2020-10-23&tree=all

Comment 53

[Intermittent Failures Robot]

44 failures in 189 pushes (0.233 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 25
• mozilla-central: 19

Platform breakdown:

• windows10-64: 44

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-24&endday=2020-10-24&tree=all

Comment 54

[Intermittent Failures Robot]

458 failures in 4230 pushes (0.108 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 291
• mozilla-beta: 21
• try: 27
• mozilla-central: 103
• mozilla-esr78: 16

Platform breakdown:

• windows10-64: 445
• linux1804-64: 1
• macosx1014-64: 3
• windows10-64-qr: 5
• macosx1014-64-qr: 1
• windows2012-32: 1
• windows2012-64: 1
• linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 55

[Intermittent Failures Robot]

77 failures in 692 pushes (0.111 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 48
• mozilla-central: 2
• try: 27

Platform breakdown:

• windows10-64: 75
• windows10-64-qr: 1
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-26&endday=2020-10-26&tree=all

Comment 56

[Andreea Pavel [:apavel]]

Hi guys, are there any updates here?

There are 429 total failures in the last 7 days on windows10-64 opt (few exceptions on windows10-64-qr release), all of them in source-test-python-[something] test suite.
Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319822761&repo=mozilla-central&lineNumber=82
https://firefox-ci-tc.services.mozilla.com/tasks/RBbL0InzTl2cmPiPFiweFg

[setup 2020-10-27T04:57:03.053Z] run-task started in Z:\task_1603774457
[vcs 2020-10-27T04:57:03.055Z] WARNING: vcs checkout path (./build/src) not in cache or volume; performance will likely suffer
[vcs 2020-10-27T04:57:03.055Z] WARNING: HG_STORE_PATH (y:/hg-shared) not in cache or volume; performance will likely suffer
[vcs 2020-10-27T04:57:03.055Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2020-10-27T04:57:04.068Z] Unable to retrieve hgmointernal config using the secret service; falling back to public hg.mozilla.org service
[vcs 2020-10-27T04:57:04.068Z] executing ['C:\Program Files\Mercurial\hg.exe', 'robustcheckout', '--sharebase', 'y:\hg-shared', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '46a0e993f8bb4eeca12cbfbe21efd19c75592d35', 'https://hg.mozilla.org/mozilla-central', 'Z:\task_1603774457\build\src']
[vcs 2020-10-27T04:57:04.155Z] (using Mercurial 4.7.1)
[vcs 2020-10-27T04:57:04.518Z] ensuring https://hg.mozilla.org/mozilla-central@46a0e993f8bb4eeca12cbfbe21efd19c75592d35 is available at Z:\task_1603774457\build\src
[vcs 2020-10-27T04:57:04.518Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["c5.2xlarge"], "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.35700011253356934}, {"extraOptions": ["c5.2xlarge"], "lowerIsBetter": true, "name": "overall_nopull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.35700011253356934}, {"extraOptions": ["c5.2xlarge"], "lowerIsBetter": true, "name": "overall_nopull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.35700011253356934}, {"extraOptions": ["c5.2xlarge"], "lowerIsBetter": true, "name": "overall_nopull_populatedwdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.35700011253356934}]}
[vcs 2020-10-27T04:57:04.518Z] Traceback (most recent call last):
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\scmutil.pyc", line 161, in callcatch
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\dispatch.pyc", line 344, in _runcatchfunc
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\dispatch.pyc", line 984, in _dispatch
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\dispatch.pyc", line 730, in runcommand
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\dispatch.pyc", line 992, in _runcommand
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\dispatch.pyc", line 981, in <lambda>
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\util.pyc", line 1528, in check
[vcs 2020-10-27T04:57:04.518Z] File "C:/mozilla-build/robustcheckout.py", line 283, in robustcheckout
[vcs 2020-10-27T04:57:04.518Z] File "C:/mozilla-build/robustcheckout.py", line 544, in _docheckout
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\hg.pyc", line 189, in peer
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\hg.pyc", line 163, in _peerorrepo
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\httppeer.pyc", line 988, in instance
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\httppeer.pyc", line 952, in makepeer
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\httppeer.pyc", line 872, in performhandshake
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\httppeer.pyc", line 311, in sendrequest
[vcs 2020-10-27T04:57:04.518Z] File "urllib2.pyc", line 429, in open
[vcs 2020-10-27T04:57:04.518Z] File "urllib2.pyc", line 447, in _open
[vcs 2020-10-27T04:57:04.518Z] File "urllib2.pyc", line 407, in _call_chain
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\url.pyc", line 391, in https_open
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\keepalive.pyc", line 240, in do_open
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\url.pyc", line 377, in _start_transaction
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\keepalive.pyc", line 343, in _start_transaction
[vcs 2020-10-27T04:57:04.518Z] File "httplib.pyc", line 1038, in endheaders
[vcs 2020-10-27T04:57:04.518Z] File "httplib.pyc", line 882, in _send_output
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\url.pyc", line 161, in _sendfile
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\keepalive.pyc", line 563, in safesend
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\url.pyc", line 365, in connect
[vcs 2020-10-27T04:57:04.518Z] File "mercurial\sslutil.pyc", line 858, in validatesocket
[vcs 2020-10-27T04:57:04.518Z] Abort: certificate for hg.mozilla.org has unexpected fingerprint sha256:ff:e7:8d:93:e9:56:3c:c0:19:fc:00:4c:18:b9:86:e5:08:e5:10:f5:e2:ea:48:e8:22:d3:a3:3a:ca:99:c3:4c
[vcs 2020-10-27T04:57:04.518Z] abort: certificate for hg.mozilla.org has unexpected fingerprint sha256:ff:e7:8d:93:e9:56:3c:c0:19:fc:00:4c:18:b9:86:e5:08:e5:10:f5:e2:ea:48:e8:22:d3:a3:3a:ca:99:c3:4c
[vcs 2020-10-27T04:57:04.518Z] (check hostsecurity configuration)
[taskcluster 2020-10-27T04:57:04.556Z] Exit Code: 255
[taskcluster 2020-10-27T04:57:04.556Z] User Time: 0s
[taskcluster 2020-10-27T04:57:04.556Z] Kernel Time: 15.625ms
[taskcluster 2020-10-27T04:57:04.556Z] Wall Time: 1.9704544s
[taskcluster 2020-10-27T04:57:04.556Z] Result: FAILED
[taskcluster 2020-10-27T04:57:04.556Z] === Task Finished ===
[taskcluster 2020-10-27T04:57:04.556Z] Task Duration: 1.9704544s
[taskcluster 2020-10-27T04:57:04.556Z] [mounts] Preserving cache: Moving "Z:\task_1603774457\build" to "Z:\caches\UOLeAnYuS7Wtqd9mjNLhBA"
[taskcluster 2020-10-27T04:57:04.556Z] [mounts] Denying task_1603774457 access to 'Z:\caches\UOLeAnYuS7Wtqd9mjNLhBA'
[taskcluster 2020-10-27T04:57:04.643Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RBbL0InzTl2cmPiPFiweFg/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2021-10-27T04:47:00.980Z
[taskcluster:error] exit status 255

Comment 57

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

See discussion in bug 1671731.

Comment 58

[Intermittent Failures Robot]

60 failures in 865 pushes (0.069 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 24
• autoland: 32
• try: 1
• mozilla-beta: 2
• mozilla-release: 1

Platform breakdown:

• windows10-64: 57
• windows10-64-qr: 2
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-27&endday=2020-10-27&tree=all

Comment 59

[Intermittent Failures Robot]

153 failures in 4275 pushes (0.036 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 88
• mozilla-central: 32
• try: 29
• mozilla-beta: 3
• mozilla-release: 1

Platform breakdown:

• windows10-64: 146
• windows10-64-qr: 3
• macosx1014-64: 2
• linux64-shippable: 1
• macosx1014-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 60

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Is there anything left to be done in this bug? Latest such failure in CI was last week Wednesday.

Comment 61

[Intermittent Failures Robot]

1 failures in 4128 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1670712&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 62

[:dhouse]

Since Oct. 28, the only repeats of this have been on a single macos test worker, t-mojave-r7-440. This worker was stuck on an old puppet configuration without the hg new fingerprint. I've fixed it (and it is using the latest config now) so this will not repeat because of that worker.

Comment 63

[Intermittent Failures Robot]

2 failures in 4040 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1670712&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 64

[Intermittent Failures Robot]

2 failures in 3421 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1670712&startday=2020-12-14&endday=2020-12-20&tree=all