ssltunnel version incompatibilities cause intermittent mochitest failures on android

RESOLVED FIXED

Status

defect
RESOLVED FIXED
5 years ago
2 months ago

People

(Reporter: mgoodwin, Assigned: Callek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Reporter

Description

5 years ago
The changes to ssltunnel developed for bug 1096197 sometimes seem not to be available to try runs on Android (causing intermittent failures).

According to https://bugzilla.mozilla.org/show_bug.cgi?id=1096197#c25 - a fix for this is to extract the ssltunnel binary from a try run (e.g. https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=d32dce6b7840 ) and include that in hostutils.zip

There should be no backwards compatibility issues (we're adding a new ssltunnel host option - nothing else changes). The patch including the ssltunnel changes in bug 1096197 is r+ by jmaher
Comment hidden (Legacy TBPL/Treeherder Robot)
Can we use the bin/ssltunnel in fennec-37.0a1.en-US.android-arm.tests.zip ? Also surprised they're intermittent failures, the ssltunnel should always be the older copy.

This is either General Automation, or Buildduty.
Component: Release Automation → General Automation
QA Contact: bhearsum → catlee
Reporter

Comment 3

5 years ago
(In reply to Nick Thomas [:nthomas] from comment #2)
> Can we use the bin/ssltunnel in fennec-37.0a1.en-US.android-arm.tests.zip ?

I'm not 100% sure; this appears to have support for the failHandshake option but I'm guessing it's an arm binary and I was assuming the ssltunnel binary runs on a different host (if so, use the ssltunnel from one of the other build products from the try run)? Someone else will know about this for sure.

> Also surprised they're intermittent failures, the ssltunnel should always be
> the older copy.

well they're intermittent in that it doesn't happen for every try push - but once it happens for a job on a given push, it will happen again no matter how many times you retrigger that job (but other jobs can run fine).
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
(In reply to Mark Goodwin [:mgoodwin] from comment #3)
> (In reply to Nick Thomas [:nthomas] from comment #2)
> > Can we use the bin/ssltunnel in fennec-37.0a1.en-US.android-arm.tests.zip ?
> 
> I'm not 100% sure; this appears to have support for the failHandshake option
> but I'm guessing it's an arm binary

bin/ssltunnel from fennec-37.0a1.en-US.android-arm.tests.zip is an arm binary:

gbrown@mozpad:~/test-zip-android-nov20-2014/bin$ hexdump -C ssltunnel | head
00000000  7f 45 4c 46 01 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  02 00 28 00 01 00 00 00  f0 9b 00 00 34 00 00 00  |..(.........4...|
                ^^

> and I was assuming the ssltunnel binary
> runs on a different host 

ssltunnel in tegra-host-utils is NOT arm:

gbrown@mozpad:~/tegra-host-utils$ hexdump -C ssltunnel | head
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  02 00 3e 00 01 00 00 00  70 36 40 00 00 00 00 00  |..>.....p6@.....|
                ^^

I'm pretty sure it runs on a linux host.
OK, I had forgotten how this works. I'm wondering if a new host-utils could be created and uploaded, then
   http://mxr.mozilla.org/build/source/mozharness/configs/android/androidarm.py#19
changed in a try push with Armen's new trick (http://armenzg.blogspot.ca/2014/12/test-mozharness-changes-on-try.html), to verify that nothing else breaks. Then swap production over, then land bug 1096197.

There is other places were the current zip file is used, which we should consider too
 http://mxr.mozilla.org/build/search?string=tegra-host-utils.*742597.zip&regexp=1&find=&findi=&filter=^%5B^\0%5D*%24&hitlimit=&tree=build
The buildbot-configs entries *may* be superceded by mozharness.

Does this sound right Callek ?
Flags: needinfo?(bugspam.Callek)
Assignee

Comment 8

5 years ago
(In reply to Nick Thomas [:nthomas] from comment #7)
> OK, I had forgotten how this works. I'm wondering if a new host-utils could
> be created and uploaded, then
>   
> Does this sound right Callek ?

This does sound right.

So a few basic questions for "whoever" takes on the work to figure out:

a) Is the changes in Bug 1096197 backwards compat? (as in ssltunnel/etc won't break on older trees, with previous uses).

b) Do we need to wait to land the ssltunnel change for when we have an updated hostutils (as in do we rebuild hostutils from a try run, or do we rebuild hostutils from an m-c nightly)

c) Anything missing/changed since 2012 with regard to my rebuild-hostutils instructions in https://bugzilla.mozilla.org/show_bug.cgi?id=742597#c29 and https://bugzilla.mozilla.org/show_bug.cgi?id=742597#c34

Once the hostutils file is regenerated I'm happy to throw it to our production server, and let the testing take place. I don't forsee myself as having time (with my current priorities) to do more work than an upload at this point in time. [members of my management chain can of course redirect my priorities]

I hope that helps.
Flags: needinfo?(bugspam.Callek)
Blocks: 1092835
For bug 1092835,

a) The change is 100% backward compatible. (I only added options.)

b) The change is already landed on m-c by a different bug (bug 1100917).
Reporter

Comment 10

5 years ago
(In reply to Justin Wood (:Callek) from comment #8)
> So a few basic questions for "whoever" takes on the work to figure out:
> 
> a) Is the changes in Bug 1096197 backwards compat? (as in ssltunnel/etc
> won't break on older trees, with previous uses).

I believe so, yes. I'm simply adding a new option 'failHandshake' - so existing server_locations will be fine.

> b) Do we need to wait to land the ssltunnel change for when we have an
> updated hostutils (as in do we rebuild hostutils from a try run, or do we
> rebuild hostutils from an m-c nightly)

Either is fine from my point of view but I'd rather land the bug this is blocking sooner rather than later.

I have no idea about the others.
how do we move forward here?  It sounds like we need to update hostutils.zip.  Can we do that and use the new mozharness try stuff that Armen did?  That would be the best way to test it out; 

Armen- does mozharness try work with android and the binaries we have to download from internal servers to specify a different binary?
Callek- can you help out and make a new hostutils.zip?  Possible pull the bits from a try server build?  this seems sort of like a chicken and egg...maybe mgoodwin can land part of it up front, generate/test the new .zip, then land the rest?
Flags: needinfo?(bugspam.Callek)
Flags: needinfo?(armenzg)

Comment 12

5 years ago
(In reply to Joel Maher (:jmaher) from comment #11)
> Armen- does mozharness try work with android and the binaries we have to
> download from internal servers to specify a different binary?

Yes, Nick has been able to use it on comment 7.

The tests.zip approach might be the most sensible due to urgency of the patch, however, I wish we would put more things into tooltool since it gives us local caching.

We can also upload it to tooltool by doing this:
1 - Upload to tooltool [1]
2 - Adding to the android tooltool manifest [2]
3 - Push to try with the new manifest + point to your own user repo
4 - IIUC setup avds will download the manifest without work on your side [3]
** It will cache it for you - cache=c.get("tooltool_cache", None))
** It will unzip it for you - output_dir=c[".avds_dir"], aka ".avds_dir": "/home/cltbld/.android"
** I am not 100%
5 - We should deprecate downloading hostutils

I will be attaching a patch today to bug 1088208 in case anyone wants to run mozharness locally for Android jobs [5]

[1] https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_to_upload_to_tooltool
[2] https://hg.mozilla.org/releases/mozilla-beta/file/0b7106ef79d2/testing/config/tooltool-manifests
[3] modify testing/mozharness/mozharness.json
[4] https://hg.mozilla.org/build/mozharness/file/fb6218e85361/scripts/android_emulator_unittest.py#l496
[5] https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer#Steps
Flags: needinfo?(armenzg)
Blocks: 1111188
Assignee

Comment 13

5 years ago
(In reply to Joel Maher (:jmaher) from comment #11)
> Callek- can you help out and make a new hostutils.zip?  Possible pull the
> bits from a try server build?  this seems sort of like a chicken and
> egg...maybe mgoodwin can land part of it up front, generate/test the new
> .zip, then land the rest?

I can't this week (PTO) and depending on team prior's may not next week.

HOWEVER my notes linked to above should be sufficient for anyone to create a new .zip for us, that I am perfectly willing to stage for someone to run against a try build (with the new mozharness stuff that lets you hit this against try) for whoever needs it.

The PTO will still limit my own availability but thats far less time.
Flags: needinfo?(bugspam.Callek)
I landed a workaround (bug 1092835) to remove dependency on this bug, so we can proceed as follows:
1. Land bug 1096197.
2. Extract ssltunnel from m-c builds.
3. Create a new hostutils.zip.
4. Remove the workaround (bug 1111188).
No longer blocks: 1096197, 1092835
mgoodwin, do you want attempt building a new hostutils.zip?  I know you have additional changes, so now might be the time :)  Just make sure to rebase before pushing to try.  Or maybe land a partial fix (the ssltunnel related stuff) so we can pull from an integration build instead of a try build.
Flags: needinfo?(mgoodwin)
Reporter

Comment 16

5 years ago
(In reply to Joel Maher (:jmaher) from comment #15)
> mgoodwin, do you want attempt building a new hostutils.zip?  

I could do with a little hand-holding (though I'm supposed to be on parental leave)

> I know you have
> additional changes, so now might be the time :)  Just make sure to rebase
> before pushing to try.  Or maybe land a partial fix (the ssltunnel related
> stuff) so we can pull from an integration build instead of a try build.

I've a new try run here (rebased): https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=835716b67e21
Flags: needinfo?(mgoodwin)
Blocks: 1000305
Hostutils.zip is linux64 (not osx).
I have created a new hostutils.zip:
http://people.mozilla.org/~jmaher/hostutils-1109310.zip

Callek, can you do the staging, drop a link for how I can verify it and I can help verify as needed.
Flags: needinfo?(bugspam.Callek)
Assignee

Comment 19

5 years ago
(In reply to Joel Maher (:jmaher) from comment #18)
> I have created a new hostutils.zip:
> http://people.mozilla.org/~jmaher/hostutils-1109310.zip

Just to articulate, for future reference, can you identify (here) what process created this, and from what cset/tree/etc.

> Callek, can you do the staging, drop a link for how I can verify it and I
> can help verify as needed.

Done

[root@relengwebadm.private.scl3 ~]# cd /data/releng/src/talos-remote/www/tegra/
[root@relengwebadm.private.scl3 tegra]# ls -la
total 115028
drwxr-xr-x 2 root root     4096 Jul 17  2012 .
drwxr-xr-x 7 root root     4096 Aug 19 06:12 ..
-rw-r--r-- 1 root root 44795471 Jun  8  2012 tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       27 Nov 20  2013 tegra-host-utils.Darwin.742597.zip -> tegra-host-utils.742597.zip
-rw-r--r-- 1 root root 32025309 Jul 24  2012 tegra-host-utils.Linux.742597.zip
-rw-r--r-- 1 root root 40953666 Feb 24  2011 tegra-host-utils.zip
[root@relengwebadm.private.scl3 tegra]# cp ~jwood/hostutils-1109310.zip ./tegra-host-utils.1109310.zip
[root@relengwebadm.private.scl3 tegra]# ln -s ./tegra-host-utils.1109310.zip ./tegra-host-utils.Linux.1109310.zip
[root@relengwebadm.private.scl3 tegra]# ls -la
total 158208
drwxr-xr-x 2 root root     4096 Dec 22 21:14 .
drwxr-xr-x 7 root root     4096 Aug 19 06:12 ..
-rw-r----- 1 root root 44214400 Dec 22 21:14 tegra-host-utils.1109310.zip
-rw-r--r-- 1 root root 44795471 Jun  8  2012 tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       27 Nov 20  2013 tegra-host-utils.Darwin.742597.zip -> tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       30 Dec 22 21:14 tegra-host-utils.Linux.1109310.zip -> ./tegra-host-utils.1109310.zip
-rw-r--r-- 1 root root 32025309 Jul 24  2012 tegra-host-utils.Linux.742597.zip
-rw-r--r-- 1 root root 40953666 Feb 24  2011 tegra-host-utils.zip
[root@relengwebadm.private.scl3 tegra]# md5sum ./tegra-host-utils.1109310.zip
4e3cd28e1f63ebab93d082146998e075  ./tegra-host-utils.1109310.zip

# Ran ./update in /data/releng/src/talos-remote/ here

Also based on e-mail from armen, where he uploaded the previous hostutils to tooltool, I did the same here:

[root@relengwebadm.private.scl3 sha512]# pwd
/mnt/netapp/relengweb/tooltool/pvt/build/sha512
[root@relengwebadm.private.scl3 sha512]# openssl sha512 ~jwood/hostutils-1109310.zip
SHA512(/home/jwood/hostutils-1109310.zip)= 38596563a5f599c5ee24504e87db24f25753fa504a756afd4ee514383fb52f4827c6e2ea6ce921a0574b98
760b5c0ad796d30e127cf1c41bd5ee217514122764
[root@relengwebadm.private.scl3 sha512]# cp ~jwood/hostutils-1109310.zip ./38596563a5f599c5ee24504e87db24f25753fa504a756afd4ee51
4383fb52f4827c6e2ea6ce921a0574b98760b5c0ad796d30e127cf1c41bd5ee217514122764
[root@relengwebadm.private.scl3 sha512]# ls -la 38596563a5f599c5ee24504e87db24f25753fa504a756afd4ee514383fb52f4827c6e2ea6ce921a0
574b98760b5c0ad796d30e127cf1c41bd5ee217514122764
-rw-r----- 1 root 2325 44214400 Dec 22 21:20 38596563a5f599c5ee24504e87db24f25753fa504a756afd4ee514383fb52f4827c6e2ea6ce921a0574b
98760b5c0ad796d30e127cf1c41bd5ee217514122764
[root@relengwebadm.private.scl3 sha512]# ls -la 372c89f9dccaf5ee3b9d35fd1cfeb089e1e5db3ff1c04e35aa3adc8800bc61a2ae10e321f37ae7ba
b20b56e60941f91bb003bcb22035902a73d70872e7bd3282
-rw-r--r-- 1 root 2325 32025309 Dec 18 08:40 372c89f9dccaf5ee3b9d35fd1cfeb089e1e5db3ff1c04e35aa3adc8800bc61a2ae10e321f37ae7bab20b
56e60941f91bb003bcb22035902a73d70872e7bd3282

n-i to armen to verify that tooltool upload, and n-i to you for instructions on how this was generated.

The way to test is "however you test mozharness" these days, with changes to the configs for host utils: http://mxr.mozilla.org/build/search?string=host_utils_url&find=&findi=&filter=^%5B^\0%5D*%24&hitlimit=&tree=build
Flags: needinfo?(jmaher)
Flags: needinfo?(bugspam.Callek)
Flags: needinfo?(armenzg)
my process here:
* push to try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4bf5810710f
* download: tests.zip and tar.bz2 from: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jmaher@mozilla.com-f4bf5810710f/try-linux64/
* unpack those locally
* mkdir hostutils
* cp -R bin/ hostutils/bin/
* cp -R firefox/ hostutils/xre/
* rm -Rf hostutils/xre/browser/
* cd hostutils
* zip -r hostutils.zip bin/ xre/

I assume this is running for all android jobs now?  I am still a little lost on how to actually test this.
Flags: needinfo?(jmaher)

Comment 21

5 years ago
(In reply to Justin Wood (:Callek) from comment #19)
> The way to test is "however you test mozharness" these days, with changes to
> the configs for host utils:
> http://mxr.mozilla.org/build/
> search?string=host_utils_url&find=&findi=&filter=^%5B^\0%5D*%24&hitlimit=&tre
> e=build

I'm not around until the 5th.

This can be tested by:
* create a mozharness user repo
* change host_utils_url to point to the new file
* change testing/mozharness/mozharness.json to point to your user repo and push to Try

For the record, the new job will be used in all trees until we have pinning of mozharness on all trees.
Flags: needinfo?(armenzg)
ok, this is using the new hostutils.zip!

the problem is we cannot find libxpcom.so (http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/jmaher@mozilla.com-b552b1152d80/try-android-api-9/try_ubuntu64_vm_mobile_test-mochitest-1-bm113-tests1-linux64-build538.txt.gz):
2:14:32     INFO -  MochitestServer : launching [u'/builds/slave/test/build/hostutils/bin/xpcshell', '-g', '/builds/slave/test/build/hostutils/xre', '-v', '170', '-f', '/builds/slave/test/build/hostutils/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpXYQX5z.mozrunner'; const _SERVER_PORT = '8854'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/builds/slave/test/build/tests/mochitest/server.js']

this is odd because I can't find it in the bits I downloaded or on my local system.  Is there some cruft on the foopies or do we need a special version of firefox to get the libraries and xpcshell from?
Flags: needinfo?(bugspam.Callek)
ok, trying hostutils.zip again (re upload and retrigger).  what is odd is android 4 doesn't seem to be using this despite entering the hostutils in my mozharness repo.

Either way, lets get android 2.3 working, then figure out the mozharness bits to adjust to make it use the real hostutils.
Flags: needinfo?(bugspam.Callek)
ok, I have verified via try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=871f79152dd8

so please upload http://people.mozilla.org/~jmaher/hostutils-1109310.zip and make it official!
Flags: needinfo?(bugspam.Callek)
Callek, can you outline the next steps here?  Do we just need to upload the new hostutils.zip and call it good?  

Since hostutils.zip has a bugid in it, I assume we might need to update mozharness as well?
I pinged Callek about this, and Armen is out for his birthday, slacker.  ;)

Worst case, we'll get this deployed on Monday.
Flags: needinfo?(bugspam.Callek)
Assignee

Comment 28

5 years ago
I'm sorry I lost/missed that needinfo from the 31st! (I'd blame our e-mail transition, but my bugmail address never moved)

Anyway, I'll tackle any remaining steps tomorrow.
Assignee: nobody → bugspam.Callek
Flags: needinfo?(bugspam.Callek)
Assignee

Comment 29

5 years ago
[root@relengwebadm.private.scl3 tegra]# cd /data/releng/src/talos-remote/www/tegra/
[root@relengwebadm.private.scl3 tegra]# ls -la
d/hototal 158208
drwxr-xr-x 2 root root     4096 Jan 12 23:04 .
drwxr-xr-x 7 root root     4096 Aug 19 06:12 ..
-rw-r----- 1 root root 44214400 Dec 22 21:14 tegra-host-utils.1109310.zip
-rw-r--r-- 1 root root 44795471 Jun  8  2012 tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       27 Nov 20  2013 tegra-host-utils.Darwin.742597.zip -> tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       30 Dec 22 21:14 tegra-host-utils.Linux.1109310.zip -> ./tegra-host-utils.1109310.zip
-rw-r--r-- 1 root root 32025309 Jul 24  2012 tegra-host-utils.Linux.742597.zip
-rw-r--r-- 1 root root 40953666 Feb 24  2011 tegra-host-utils.zip
[root@relengwebadm.private.scl3 tegra]# cp ~jwood/hostutils-1109310.zip ./tegra-host-utils.1109310.2.zip
[root@relengwebadm.private.scl3 tegra]# ln -s ./tegra-host-utils.1109310.2.zip ./tegra-host-utils.Linux.1109310.2.zip
[root@relengwebadm.private.scl3 tegra]# ls -la
total 204084
drwxr-xr-x 2 root root     4096 Jan 12 23:05 .
drwxr-xr-x 7 root root     4096 Aug 19 06:12 ..
-rw-r----- 1 root root 46973416 Jan 12 23:05 tegra-host-utils.1109310.2.zip
-rw-r----- 1 root root 44214400 Dec 22 21:14 tegra-host-utils.1109310.zip
-rw-r--r-- 1 root root 44795471 Jun  8  2012 tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       27 Nov 20  2013 tegra-host-utils.Darwin.742597.zip -> tegra-host-utils.742597.zip
lrwxrwxrwx 1 root root       32 Jan 12 23:05 tegra-host-utils.Linux.1109310.2.zip -> ./tegra-host-utils.1109310.2.zip
lrwxrwxrwx 1 root root       30 Dec 22 21:14 tegra-host-utils.Linux.1109310.zip -> ./tegra-host-utils.1109310.zip
-rw-r--r-- 1 root root 32025309 Jul 24  2012 tegra-host-utils.Linux.742597.zip
-rw-r--r-- 1 root root 40953666 Feb 24  2011 tegra-host-utils.zip
[root@relengwebadm.private.scl3 tegra]# md5sum ./tegra-host-utils.1109310.2.zip
929fa7dccd3ebb62b5ff436269190ae7  ./tegra-host-utils.1109310.2.zip
[root@relengwebadm.private.scl3 tegra]# cd /data/releng/src/talos-remote/
[root@relengwebadm.private.scl3 talos-remote]# ./update

# update output, no errors

[root@relengwebadm.private.scl3 talos-remote]#cd /mnt/netapp/relengweb/tooltool/pvt/build/sha512/
[root@relengwebadm.private.scl3 sha512]# openssl sha512 ~jwood/hostutils-1109310.zip
SHA512(/home/jwood/hostutils-1109310.zip)= 8bdb0f093b9a40c86f0cfa50fab03e6fbbb6201e741df411872d56135912c26c39f4db041c70ad7e5a896f
ab91cda27b18eaf1e22b51fc3bd2be78577861a01d
[root@relengwebadm.private.scl3 sha512]# export SHA=`openssl sha512 ~jwood/hostutils-1109310.zip | cut -d" " -f2`
[root@relengwebadm.private.scl3 sha512]# cp ~jwood/hostutils-1109310.zip ./$SHA
[root@relengwebadm.private.scl3 sha512]# ls -la $SHA
-rw-r----- 1 root 2325 46973416 Jan 12 23:10 8bdb0f093b9a40c86f0cfa50fab03e6fbbb6201e741df411872d56135912c26c39f4db041c70ad7e5a89
6fab91cda27b18eaf1e22b51fc3bd2be78577861a01d


===

Let me know if this doesn't work out for some reason
Flags: needinfo?(jmaher)
Flags: needinfo?(bugspam.Callek)
Flags: needinfo?(armenzg)
I have a try push up here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d306158fb0ca

If this is good, we can look at a mozharness patch.
Flags: needinfo?(jmaher)

Updated

5 years ago
Flags: needinfo?(armenzg)
verified on try (https://treeherder.mozilla.org/#/jobs?repo=try&revision=d306158fb0ca - beware, I verified the old, then retriggered some jobs to verify the new hostutils, some jobs which were scheduled later got the new hostutils on the first round)
Attachment #8548282 - Flags: review?(armenzg)

Updated

5 years ago
Attachment #8548282 - Flags: review?(armenzg) → review+

Updated

5 years ago
See Also: → 1121030
Assignee

Comment 33

5 years ago
For what its worth, this caused permanent bustage on ESR31 [mochitests] with the following message:

14:03:11 INFO - Mochi-Remote INFO | Android sdk version '10'; will use this to filter manifests
14:03:11 INFO - /builds/slave/test/build/hostutils/bin/certutil: /usr/lib/x86_64-linux-gnu/libnssutil3.so: version `NSSUTIL_3.15' not found (required by /builds/slave/test/build/hostutils/bin/certutil)
14:03:11 INFO - /builds/slave/test/build/hostutils/bin/certutil: /usr/lib/x86_64-linux-gnu/libnss3.so: version `NSS_3.15' not found (required by /builds/slave/test/build/hostutils/bin/certutil)
14:03:11 INFO - /builds/slave/test/build/hostutils/bin/certutil: /usr/lib/x86_64-linux-gnu/libnss3.so: version `NSS_3.16.2' not found (required by /builds/slave/test/build/hostutils/bin/certutil)
14:03:11 WARNING - TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed 

:Kwierso decided NOT to have me backout and will be filing a bug on that issue alone.
Blocks: 1121720
Assignee

Updated

5 years ago
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Component: General Automation → General
See Also: → 1548918
You need to log in before you can comment on or make changes to this bug.