Add ssltunnel to xre.zip bundle used for b2g tests

RESOLVED FIXED

Status

defect
P2
normal
RESOLVED FIXED
5 years ago
a year ago

People

(Reporter: mt, Assigned: jgriffin)

Tracking

({ateam-b2g-task})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [leave open], URL)

Attachments

(4 attachments)

(Reporter)

Description

5 years ago
Running tests that use https resources means that ssltunnel needs to be run.  It is not present in the current build of the xre.zip, so cannot run.

Bug 907770 includes the gecko-side changes for this.  That requires the xre bundle, currently at http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/cba263cef46b57585334f4b71fbbd15ce740fa4b7260571a9f7a76f8f0d6b492b93b01523cb01ee54697cc9b1de1ccc8e89ad64da95a0ea31e0f119fe744c09f to be updated.

Bug 742597 caused a bundle to be created that would probably suffice, I just want to make sure that there aren't other things here that need to be considered.
(Assignee)

Comment 1

5 years ago
The tegra bundle won't work for B2G, because it's too old, and its httpd.js won't match in-tree server.js which is currently bundled in tests.zip.

I think we should add ssltunnel from a linux32 desktop build to the existing xre.zip from tooltool.
(Assignee)

Comment 2

5 years ago
Actually, we'll need linux32, linux64, and osx versions of this.
(Assignee)

Comment 3

5 years ago
The most recent copy of xre.zip comes from xulrunner-sdk-26.  I'll take this and add ssltunnel from here:  http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/26.0/
(Assignee)

Updated

5 years ago
Assignee: nobody → jgriffin
(Assignee)

Updated

5 years ago
Depends on: 987957
Should be fine, ssltunnel hasn't changed in a while. (I wish we just had implemented this all in Python.)
(Assignee)

Comment 6

5 years ago
I'll update desktop configs in a separate patch.
Attachment #8398074 - Flags: review?(ahalberstadt)
Comment on attachment 8398074 [details] [diff] [review]
Use updated xre.zip in emulator unittests,

Review of attachment 8398074 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm.. I guess :)
Attachment #8398074 - Flags: review?(ahalberstadt) → review+
(Assignee)

Comment 8

5 years ago
Comment on attachment 8398074 [details] [diff] [review]
Use updated xre.zip in emulator unittests,

https://hg.mozilla.org/build/mozharness/rev/bce52213ed6b

Will watch cedar for bustage.
Attachment #8398074 - Flags: checked-in+
(Assignee)

Comment 9

5 years ago
That definitely didn't work:  

15:34:32     INFO -  MochitestServer : launching [u'/builds/slave/test/build/xre/bin/xpcshell', '-g', '/builds/slave/test/build/xre/bin', '-v', '170', '-f', '/builds/slave/test/build/xre/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmp60hZKU'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', './server.js']
15:34:32     INFO -  runtests.py | Server pid: 2392
15:36:02  WARNING -  TEST-UNEXPECTED-FAIL | runtests.py | Timed out while waiting for server startup.
15:37:06     INFO -  Traceback (most recent call last):
15:37:06     INFO -    File "runtestsb2g.py", line 339, in run_remote_mochitests
15:37:06     INFO -      retVal = mochitest.run_tests(options)
15:37:06     INFO -    File "runtestsb2g.py", line 117, in run_tests
15:37:06     INFO -      self.startWebServer(options)
15:37:06     INFO -    File "runtestsb2g.py", line 217, in startWebServer
15:37:06     INFO -      self.server.ensureReady(90)
15:37:06     INFO -    File "/builds/slave/test/build/tests/mochitest/runtests.py", line 190, in ensureReady
15:37:06     INFO -      sys.exit(1)
15:37:06     INFO -  SystemExit: 1
15:38:09     INFO -  Automation Error: Exception caught while running tests

I've backed out and will investigate.
(Assignee)

Comment 10

5 years ago
This failed because I failed to add httpd.js to the new xre.zip.  :(  I've fixed this, but it will require another tooltool upload.
(Assignee)

Updated

5 years ago
Depends on: 989143
in production.
(Reporter)

Comment 13

5 years ago
No screams yet?  Trying this out: https://tbpl.mozilla.org/?tree=Try&rev=e2aa932bebb7
(Assignee)

Comment 14

5 years ago
You won't see it in a try job yet; mozharness changes have to be explicitly moved into production before they take effect globally.  I'm watching some jobs on cedar (which works on pre-production changes), and if they go green, I'll put this patch into production.
(Assignee)

Comment 15

5 years ago
This patch works 80% of the time!

Seriously.

It works for most emulator tests, but unfortunately it fails on reftests running on the old r3 mac minis.  Apparently those machines are 32-bit, since this version of xre.zip fails with:

OSError: [Errno 8] Exec format error

We're phasing tests out on r3 mac minis, but we're not quite there, so I'll have to back this out and try again with a 32-bit version.
(Assignee)

Comment 16

5 years ago
(The good news is that the 64-bit xre.zip works fine and I'll use that for linux64 b2g desktop builds.)
(Assignee)

Updated

5 years ago
Depends on: 989412
(Assignee)

Comment 17

5 years ago
Landed the 32-bit xre.zip change:  https://hg.mozilla.org/build/mozharness/rev/52636b92cbfd

Watching cypress...
(Assignee)

Comment 18

5 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #17)
> Landed the 32-bit xre.zip change: 
> https://hg.mozilla.org/build/mozharness/rev/52636b92cbfd
> 
> Watching cypress...

Great success!  I'll merge to production on Monday a.m.
in production.
(Assignee)

Comment 20

5 years ago
So, the emulators are done, I now need to fix the B2G desktop build tests.
(Reporter)

Comment 21

5 years ago
The desktop builds should be OK, since they are compiling xpcshell and ssltunnel in a compatible form.
(Assignee)

Comment 22

5 years ago
Ah, right.  In that case, I'll close this; re-open if needed.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 23

5 years ago
Encountering issues with the b2g emulator: https://tbpl.mozilla.org/?tree=Try&rev=2b5bb0340054

The actual problem seems consistent:

10:35:59     INFO -  INFO | runtests.py | SSL tunnel pid: 2413
10:35:59     INFO -  Failed to init NSS:

This suggests that the xre bundle (http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/ce1ad86b21d534bd706cd70f1d763b9ca1cfd0a1ea4d519864c961e202c36ccf48bcfa9c572113076ee8b3d0770f60b8967b7620b4adaad531f47b993eb57912) doesn't contain a version of ssltunnel that runs on the emulator machines.  Given that it's a 64bit Ubuntu machine, I'm at a loss to explain this.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Updated

5 years ago
Keywords: ateam-b2g-task
Priority: -- → P2
(Assignee)

Comment 24

5 years ago
We're off the mac minis now, so will try bumping our xre.zip to 64-bit.
Attachment #8407778 - Flags: review?(ahalberstadt) → review+
(Assignee)

Comment 26

5 years ago
Comment on attachment 8407778 [details] [diff] [review]
Use a 64-bit version of xre.zip on emulator unittests,

https://hg.mozilla.org/build/mozharness/rev/2a89305f4a67
Attachment #8407778 - Flags: checked-in+
(In reply to Jonathan Griffin (:jgriffin) from comment #26)
> Comment on attachment 8407778 [details] [diff] [review]
> Use a 64-bit version of xre.zip on emulator unittests,
> 
> https://hg.mozilla.org/build/mozharness/rev/2a89305f4a67

In production
(Assignee)

Comment 28

5 years ago
Martin, can you try again?  The 64-bit version is now live.
Flags: needinfo?(martin.thomson)
(Reporter)

Comment 29

5 years ago
Bug 907770 needs a rebase.  It seems like these files see a lot of churn.  Let me see if I can't get a try run overnight.
Flags: needinfo?(martin.thomson)
(Reporter)

Comment 31

5 years ago
Looks like our old friend:  https://tbpl.mozilla.org/php/getParsedLog.php?id=37968603&tree=Try&full=1

17:39:41     INFO -  INFO | runtests.py | SSL tunnel pid: 2420
17:39:41     INFO -  Failed to init NSS: None

A few builds seem to be burning here, but it doesn't look like anything else is broken.
(Assignee)

Comment 32

5 years ago
I can reproduce this locally; it doesn't seem to be a problem with ssltunnel itself, but how we're invoking.  Digging more...
(Reporter)

Comment 33

5 years ago
I don't know if this helps, but there's a possibility that the emulator needs to use a different type of certificate DB.  Android insists on having the DB in the "new" form, the desktop tests in the "old" form.  Maybe my intelligence on this was wrong:

    if self.certdbNew:
 android and b2g use the new DB formats exclusively
		certdbPath = "sql:" + options.profilePath
		else:
		# desktop seems to use the old
		certdbPath = options.profilePath
(Reporter)

Comment 34

5 years ago
I hope that the missing comment in the above is obvious - I have to remember that tab thing.
(Assignee)

Comment 35

5 years ago
At least one of the problems (hopefully the only one) is that we're producing an invalid ssltunnel.cfg:

httpproxy:1
certdbdir:None
forward:127.0.0.1:8888
websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate
websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate
listen:self-signed.example.com:443:4443:selfsigned
...

We don't include a real certdbdir because we're not passing --certificate-path to runtestsb2g.py.  This will need a mozharness fix, and then an in-tree config fix.

With these things fixed locally, everything works for me on my machine, so we'll hope the same is true for buildbot.
(Assignee)

Comment 36

5 years ago
This mozharness part of the fix.
Attachment #8408735 - Flags: review?(ahalberstadt)
(Assignee)

Comment 37

5 years ago
The in-tree part of the fix.
Attachment #8408736 - Flags: review?(ahalberstadt)
(Assignee)

Comment 38

5 years ago
Comment on attachment 8408735 [details] [diff] [review]
Set up certificate_path processing,

Review of attachment 8408735 [details] [diff] [review]:
-----------------------------------------------------------------

ahal is on Easter break today
Attachment #8408735 - Flags: review?(ahalberstadt) → review?(aki)
(Assignee)

Updated

5 years ago
Attachment #8408736 - Flags: review?(ahalberstadt) → review+

Updated

5 years ago
Attachment #8408735 - Flags: review?(aki) → review+
(Reporter)

Comment 40

5 years ago
Should a respin of the try run work at this point?
(Assignee)

Comment 41

5 years ago
(In reply to Martin Thomson [:mt] from comment #40)
> Should a respin of the try run work at this point?

No.  I need to verify the above landing in staging on cypress, push it to production, then land the gecko bits.  I'll ping you when it's ready to re-test.
(Assignee)

Comment 42

5 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #39)
> Comment on attachment 8408735 [details] [diff] [review]
> Set up certificate_path processing,
> 
> https://hg.mozilla.org/build/mozharness/rev/d92a223fe008

in production
(Assignee)

Comment 43

5 years ago
Martin, the mozharness piece is in production now.  I haven't been able to land the gecko piece, because m-i is closed, but if you add this patch to your patch queue, you could try it out with a try job anyway:

https://bug987406.bugzilla.mozilla.org/attachment.cgi?id=8408736
Flags: needinfo?(martin.thomson)
(Assignee)

Comment 44

5 years ago
Ah, m-i just opened and I landed it:  https://hg.mozilla.org/integration/mozilla-inbound/rev/5d9fb05d9d12

You'll still need to include this patch in your queue, or rebase on top of m-i before you push to try.
(Assignee)

Updated

5 years ago
Whiteboard: [leave open]
(Reporter)

Comment 45

5 years ago
Rebase, rebase, rebase...

https://tbpl.mozilla.org/?tree=Try&rev=f7c3e60b8541
Flags: needinfo?(martin.thomson)
(Reporter)

Comment 47

5 years ago
Good news, it looks like ssltunnel runs.  Bad news, the test still times out. But it does seem to at least make progress, so it might be a test issue. Signs are good for this bug.
(Assignee)

Comment 48

5 years ago
I also saw that we're producing an ssltunnel.cfg in which a couple of lines are repeated:

websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate
websocketserver:10.0.2.2:9988
listen:*:4443:pgo server certificate

Not sure if this would cause any problem, though.
(Reporter)

Comment 50

5 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #48)
> I also saw that we're producing an ssltunnel.cfg in which a couple of lines
> are repeated:
> 
> websocketserver:10.0.2.2:9988
> listen:*:4443:pgo server certificate
> websocketserver:10.0.2.2:9988
> listen:*:4443:pgo server certificate
> 
> Not sure if this would cause any problem, though.

I wouldn't have thought so.  I don't see how that would happen using either trunk or my modified code unless there is a python bug or something equally perverse.  Especially if those are the only lines duplicated, see: https://hg.mozilla.org/try/rev/5328e2ef6ad2#l2.197 or for trunk: https://hg.mozilla.org/try/rev/5328e2ef6ad2#l2.740
(Assignee)

Comment 51

5 years ago
You're right; this was an artifact of how I was retrieving a copy of the file during testing.
(Reporter)

Comment 52

5 years ago
Dammit, I should have noticed this more quickly.  The test that fails fails because it doesn't run on e10s.  Running on desktop with --e10s causes it to time out in exactly the same way.  I'll need to disable that test on b2g for that reason.  ssltunnel is working perfectly.
(Reporter)

Comment 54

5 years ago
https://tbpl.mozilla.org/?tree=Try&rev=4f9132738b08

Yes, it works.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 55

5 years ago
\o/
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.