1260383 - TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | JavascriptException: JavascriptException: TypeError: ums.activeUpdate is null

Status: RESOLVED WORKSFORME

(Testing :: Firefox UI Tests, defect)

Description

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Failure in https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=53d6e6648f97&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=944909. Cleared on re-run, so maybe some one-time hiccup?

 09:16:52     INFO -  ERROR: Error verifying signature.
 09:16:52     INFO -  ERROR: Not all signatures were verified.
 09:16:52     INFO -  ERROR: Error verifying signature.
 09:16:52     INFO -  ERROR: Not all signatures were verified.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #0)
>  09:16:52     INFO -  ERROR: Error verifying signature.
>  09:16:52     INFO -  ERROR: Not all signatures were verified.
>  09:16:52     INFO -  ERROR: Error verifying signature.
>  09:16:52     INFO -  ERROR: Not all signatures were verified.

Those failures come from the application updater. Robert, mind helping us to understand what this could mean and why it happened only once?

Comment 2

[Robert Strong (they/them - no direct email)]

The signature used to sign the mar file didn't match the signature of the mar file. We verify the signature code in the in tree build xpcshell tests and there were no failures there. Perhaps the mar your test downloaded didn't have the correct signature.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm. Rebuilding the same job passed a couple of minutes after the first one has been run. So here the download locations as given by the application update log:

fail: http://download.cdn.mozilla.net/pub/firefox/releases/46.0b5/update/mac/zh-TW/firefox-46.0b5.complete.mar
pass: http://download.cdn.mozilla.net/pub/firefox/releases/46.0b5/update/mac/zh-TW/firefox-46.0b5.complete.mar

So would that mean that the download mirror served us a mar file which didn't had a good signature?

Comment 4

[Rail Aliiev [:rail]]

I just verified both files using https://github.com/mozilla/build-mar/ and they look good

mar.py --verify --list  --keyfile ~/work/mozilla/repos/mozilla/release/docker/funsize-balrog-submitter/release.pubkey firefox-46.0b5.complete.mar 2>&1 |head 
Verification OK (RSA-PKCS1-SHA1)
SIZE    MODE    NAME   
1358    0664    updatev2.manifest
1385    0664    updatev3.manifest
4361    0644    Contents/_CodeSignature/CodeResources
242     0644    Contents/Resources/zh_TW.lproj/InfoPlist.strings
387     0644    Contents/Resources/webapprt/webapprt.ini
69744   0644    Contents/Resources/webapprt/omni.ja
21506   0755    Contents/Resources/webapprt-stub
512     0644    Contents/Resources/updater.ini

I'm not sure what was the root clause of that error. Could be the CDN, could be the proxy.

I see you have HTTP headers in the logs, it's worth asking cloud ops to look at them.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jeremy, would you mind to have a look at those HTTP logs? Is there anything suspicious you can see in regards of broken downloaded MAR files?

The signature check which failed happened here:
https://firefox-ui-tests.s3.amazonaws.com/be2f54f0-c2f6-458c-afd6-47df0cda1f33/http.log

Comment 6

[Jeremy Orem [:oremj]]

It looks like this file has never been modified, so what is there now is what has always been there. The etag in the response also matches the etag of the current file.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This failure didn't occur anymore. So it seems like it was a one-time networking issue. Lets close the bug as WFM. Feel free to reopen when it starts to happen again more than once.