1462534 - ESR52 build problems - emevoucher signing, missing logs on treeherder

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Something odd is happening on ESR52 and might be fallout from various python upgrades. The current tip is at https://hg.mozilla.org/releases/mozilla-esr52/rev/b71294fa76350589b14335982dddde96a85d138f. The mac & windows compile jobs are failing, and treeherder doesn't get the logs.

Missing logs:
I think they're still being uploaded, eg https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-esr52-macosx64/1526605972/mozilla-esr52-macosx64-bm86-build1-build1.txt.gz, but something goes wrong with getting the message to Treeherder. Maybe pulse ?

Signing:
End of the log goes like
18:55:34     INFO -  Usage: signtool.py [options] file [file ...]
18:55:34     INFO -  If no include patterns are specified, all files will be considered. -i/-x only
18:55:34     INFO -  have effect when signing entire directories.
18:55:34     INFO -  signtool.py: error: invalid format: emevoucher
which is weird, because we still support emevoucher based on search hits in tools and puppet.  See also bug 1381719 about disabling EME vouchers on ESR52.

Maybe related - Bug 1462429 contains some buildbot scheduler weirdness from about 06:40 Pacific on the 17th.

Any ideas on this Ben ?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

re the logs, I see a lot of mess in my browser console about
  TypeError: "({title:"Buildername"}) is not a function"
and logs don't seem to work a week ago either.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: fix allowed formats in signtool.py

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #0)
> Signing:
> End of the log goes like
> 18:55:34     INFO -  Usage: signtool.py [options] file [file ...]
> 18:55:34     INFO -  If no include patterns are specified, all files will be
> considered. -i/-x only
> 18:55:34     INFO -  have effect when signing entire directories.
> 18:55:34     INFO -  signtool.py: error: invalid format: emevoucher
> which is weird, because we still support emevoucher based on search hits in
> tools and puppet.  See also bug 1381719 about disabling EME vouchers on
> ESR52.

It looks like this is caused by a missing comma in https://hg.mozilla.org/build/tools/rev/366c73410d4e#l3.14 actually, wheh!

Comment 3

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8976942 [details] [diff] [review]
fix allowed formats in signtool.py

Review of attachment 8976942 [details] [diff] [review]:
-----------------------------------------------------------------

That's a fallout of bug 1409091. Sorry about this. I thought my testing covered this. It turned out I forgot to test the "old" signtool, which is not used anymore in scriptworker. Thank you for the investigation, guys!

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)
> re the logs, I see a lot of mess in my browser console about
>   TypeError: "({title:"Buildername"}) is not a function"
> and logs don't seem to work a week ago either.

I spoke with the Treeherder folks, it looks like this a regression on their side: https://bugzilla.mozilla.org/show_bug.cgi?id=1462625

Comment 5

[[e-mail] (:bhearsum)]

My triggered job worked, looks like we're good here!

Comment 6

[Intermittent Failures Robot]

6 failures in 639 pushes (0.009 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr52: 6

Platform breakdown:
* windowsxp: 2
* windows8-64: 2
* osx-10-7: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1462534&startday=2018-05-14&endday=2018-05-20&tree=trunk