TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_hotfix_cert.js | xpcshell return code: 0

RESOLVED FIXED

Status

()

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Fallen, Assigned: mossop)

Tracking

({intermittent-failure, regression})

45 Branch
intermittent-failure, regression
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

This has been occurring on all comm trees lately. I first thought it might be fallout from addons signing, but unfortunately it is the dreaded nsIHttpServer not working after a request or two which I have also been getting in bug 1190012.

What happens is that the first request succeeds, but all other requests just time out. As a result, the addon download never finishes and the whole test times out.

http://mxr.mozilla.org/comm-central/source/mozilla/toolkit/mozapps/extensions/test/xpcshell/test_hotfix_cert.js

I've tested this with the latest comm-central clone, where I then went into test_hotfix_cert, commented out all tasks but the first and made an exact copy of the first task. This makes the test timeout as well.

I then copied the httpserver initialization into the beginning of tryInstallHotfix, so that on each task a new httpserver is started and stopped. This makes the modified test work again.

The bug must therefore be some interaction between nsIHttpServer, comm-central, and xpcshell. I spent a few days trying to debug this in bug 1190012 but I really have no idea how to fix it.

Patrick, as you do reviews for Core/Networking, would you be able to help out in trying to figure out the problem here? If you have some ideas on what debugging I can enable or what the problem can be, please let me know. Or maybe you can defer the needinfo to someone else.
Flags: needinfo?(mcmanus)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Flags: needinfo?(mcmanus) → needinfo?(jduell.mcbugs)

Comment 6

3 years ago
Fun fact: if you run this test on an IB build, it fails immediately in the second test (no timeout).

Whether that means it's more or less broken there, I don't know.
For the record:

http://hg.mozilla.org/comm-central/pushloghtml?fromchange=dd26276d7402&tochange=bdb1708f8b8c
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a69094e0f2a4&tochange=e537a1ba501b

WORKS:
c-c: dd26276d7402
m-c: a69094e0f2a4

FAILS:
c-c: bdb1708f8b8c
m-c: e537a1ba501b


The c-c changesets are close together and pretty unrelated, there are a lot of m-c changesets. I don't even know if it is pure coincidence. Maybe someone with a fast machine could build with these changesets, make sure they correctly work/fail, and then bisect the m-c changesets.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Keywords: regression
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* comm-central: 6

Platform breakdown:
* windowsxp: 2
* windows7-32: 2
* linux64: 2

For more details, see:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1199907&startday=2015-09-28&endday=2015-10-04&tree=all

Comment 32

3 years ago
Might this be related to the cache patch in bug 11706464 (now backed out)? Let's see if it works now.

Comment 33

3 years ago
(In reply to aleth [:aleth] from comment #32)
> Might this be related to the cache patch in bug 11706464 (now backed out)?
> Let's see if it works now.

Nope, still failing (at least locally).
9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* comm-beta: 9

Platform breakdown:
* windowsxp: 2
* osx-10-10: 2
* linux64: 2
* windows7-32: 1
* osx-10-6: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1199907&startday=2015-10-05&endday=2015-10-11&tree=all
 0:02.58 PROCESS_OUTPUT: Thread-1 (pid:29609) "1447099520991	addons.productaddons	WARN	Failed downloading XM, status: 0, reason: error"
 0:02.58 PROCESS_OUTPUT: Thread-1 (pid:29609) "1447099520995	addons.manager	WARN	Failed to update system addons: Error: Failed downloading XML from http://%(server)s/dummy-system-addons.xml
....


...

So the url isn't expanded to a real server. Do we use
http://mxr.mozilla.org/comm-central/source/mozilla/testing/talos/talos/config.py#144
The above might be unrelated, or at least irrelevant.

For the failure we have now on trunk, it's a matter if singing. If i build with add-on signing enabled it works, if I build without it the test fails.
Depends on: 1168571
Created attachment 8687713 [details] [diff] [review]
bug1199907_hotfix_cert_test.patch

Disable the test for when addon-signing is disabled. 

This is a bit concerning though. I don't really understand this enough to say if it's just the test that's wrong or if hotfixinig is just not working when addon signing is disabled. If it's not just a test failure, ff dev edition (and thunderbird) can't be hotfixed.

xref bug 1210995
Assignee: nobody → mkmelin+mozilla
Status: NEW → ASSIGNED
Attachment #8687713 - Flags: review?(dtownsend)
(Assignee)

Comment 38

3 years ago
Comment on attachment 8687713 [details] [diff] [review]
bug1199907_hotfix_cert_test.patch

Review of attachment 8687713 [details] [diff] [review]:
-----------------------------------------------------------------

Hotfixing should still be working. Hotfixes must be signed by something chaining to the AMO root if ADDON_SIGNING is enabled (dev edition) or my any cert that matches the prefs otherwise (Thunderbird). That said I'd rather we fix verifyZipSignedState to always validate the certificate for hotfix add-ons, similar to bug 1199907. Again I can do that if you want, just assign it to me.
Attachment #8687713 - Flags: review?(dtownsend) → review-
Assignee: mkmelin+mozilla → dtownsend
Component: Testing Infrastructure → Add-ons Manager
Product: Thunderbird → Toolkit
Version: 39 → 45 Branch
(Assignee)

Updated

3 years ago
Depends on: 1225629
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

3 years ago
Flags: needinfo?(jduell.mcbugs)
You need to log in before you can comment on or make changes to this bug.