520722 - run unittests on linux64

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tracking bug, as we need unittests here to claim full support of this OS. 

I'm inclined to think we would do this after creating linux64 refimages, but could be convinced otherwise. See bug#519074c0 for details.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

moving to future until dependent bug is resolved

Comment 3

[Armen [:armenzg]]

Attachment: make linux64 on staging to run unittests (1 of 2)

Comment 4

[Armen [:armenzg]]

Attachment: make linux64 on staging to run unittests (2 of 2)

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 425993 [details] [diff] [review]
make linux64 on staging to run unittests (2 of 2)

>diff --git a/env.py b/env.py
>--- a/env.py
>+++ b/env.py
>@@ -71,16 +71,23 @@ MozillaEnvironments['win32-ref-platform'
> 
> MozillaEnvironments['linux-unittest'] = {
>     "MOZ_NO_REMOTE": '1',
>     "CVS_RSH": 'ssh',
>     "DISPLAY": ':2',
>     "NO_FAIL_ON_TEST_ERRORS": '1'   
> }
> 
>+MozillaEnvironments['linux64-unittest'] = {
>+    "MOZ_NO_REMOTE": '1',
>+    "CVS_RSH": 'ssh',
>+    "DISPLAY": ':2',
>+    "NO_FAIL_ON_TEST_ERRORS": '1'   
}

Just make this a copy of the 32-bit linux env:
..... = MozillaEnironments['linux-unittest'].copy()

>         'linux': WithProperties('%(toolsdir:-)s/breakpad/linux/minidump_stackwalk'),
>+        'linux64': WithProperties('%(toolsdir:-)s/breakpad/linux/minidump_stackwalk'),

You should point to 'linux64' here, and add a symlink in the tools repository. Just in case we need different versions for the two in the future.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 425992 [details] [diff] [review]
make linux64 on staging to run unittests (1 of 2)

>diff --git a/mozilla2-staging/config.py b/mozilla2-staging/config.py
>+        'linux64-debug': {
>+            'base_name': 'Linux x86-64 %(branch)s leak test',
>+            'mozconfig': 'linux64/%(branch)s/debug',
>+            'profiled_build': False,
>+            'builds_before_reboot': 5,
>+            'build_space': 7,
>+            'slaves': SLAVES['linux64'],
>+            'platform_objdir': OBJDIR,
>+            'env': {
>+                'MOZ_OBJDIR': OBJDIR,
>+                'DISPLAY': ':2',
>+                'LD_LIBRARY_PATH': '%s/dist/bin' % OBJDIR,
>+                'XPCOM_DEBUG_BREAK': 'stack-and-abort',
>+                'MOZ_CRASHREPORTER_NO_REPORT': '1',
>+            },
>+        },


>-BRANCHES['electrolysis']['platforms']['linux']['env']['MOZ_SYMBOLS_EXTRA_BUILDID'] = 'electrolysis'
>-BRANCHES['electrolysis']['platforms']['linux']['env']['LD_LIBRARY_PATH'] = '/tools/gcc-4.3.3/installed/lib'
>-BRANCHES['electrolysis']['platforms']['linux']['unittest-env'] = {
>-    'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+BRANCHES['electrolysis']['platforms']['linux'] = {
>+    'env': {
>+        'MOZ_SYMBOLS_EXTRA_BUILDID': 'electrolysis',
>+        'LD_LIBRARY_PATH':           '/tools/gcc-4.3.3/installed/lib'
>+    },
>+    'unittest-env': {
>+        'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+    }
>+}
>+BRANCHES['electrolysis']['platforms']['linux64'] = {
>+    'env': {
>+        'MOZ_SYMBOLS_EXTRA_BUILDID': 'electrolysis',
>+        'LD_LIBRARY_PATH':           '/tools/gcc-4.3.3/installed/lib'
>+    },
>+    'unittest-env': {
>+        'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+    }
> }

You can't do this like this -- you're overwriting the existing environment. You need to do it like the existing code, or by something like:
BRANCHES['electrolysis']['platforms']['linux']['env'].update({stuff})
BRANCHES['electrolysis']['platforms']['linux']['unittest-env'].update({stuff})

Comment 7

[Armen [:armenzg]]

Attachment: [staging] make linux64 to run unittests on staging (1 of 3)

Comment 8

[Armen [:armenzg]]

Attachment: [buildbotcustom] make linux64 to run unittests on staging (2 of 3)

Comment 9

[Armen [:armenzg]]

Attachment: [tools] make linux64 to run unittests on staging (3 of 3)

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 426299 [details] [diff] [review]
[staging] make linux64 to run unittests on staging (1 of 3) 

Looks good to me

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 426300 [details] [diff] [review]
[buildbotcustom] make linux64 to run unittests on staging (2 of 3) 

>diff --git a/misc.py b/misc.py
>--- a/misc.py
>+++ b/misc.py
>@@ -239,16 +239,19 @@ def generateBranchObjects(config, name):
>     # These dicts provides mapping between en-US dep and  nightly scheduler names
>     # to l10n dep and l10n nightly scheduler names. It's filled out just below here.
>     l10nBuilders = {}
>     l10nNightlyBuilders = {}
>     # generate a list of builders, nightly builders (names must be different)
>     # for easy access
>     for platform in config['platforms'].keys():
>         pf = config['platforms'][platform]
>+        print name
>+        print platform
>+        print dir(pf)

You probably want to get rid of these ;-). Looks fine otherwise.

Comment 12

[Armen [:armenzg]]

Comment on attachment 426301 [details] [diff] [review]
[tools] make linux64 to run unittests on staging (3 of 3) 

http://hg.mozilla.org/build/tools/rev/60964b213cda

Comment 13

[Armen [:armenzg]]

We are not landing the patches yet.
We are going to run moz2-linux-slave09 on sm03 using these puppet manifests:
http://hg.mozilla.org/users/bhearsum_mozilla.com/puppet-manifests-linux64

Comment 14

[Armen [:armenzg]]

Attachment: [staging] disable downloading symbols for Linux 64 (1 of 2)

Comment 15

[Armen [:armenzg]]

Attachment: [custom] disable downloading symbols for Linux 64 (2 of 2)

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 426713 [details] [diff] [review]
[custom] disable downloading symbols for Linux 64 (2 of 2)

You missed the second 'UnittestPackagedBuildFactory'. Also, it would be nice to update the generateCCTestBuilder function at the same time -- can you add the downloadSymbols part to all of those?

Comment 17

[Armen [:armenzg]]

Attachment: [custom] disable downloading symbols for Linux 64 (2 of 2)

Comment 18

[Armen [:armenzg]]

Attachment: [buildbot-configs] disable downloading symbols for Linux 64 (1 of 2)

This also includes the flag in production.

Comment 19

[Armen [:armenzg]]

Attachment: [custom] disable downloading symbols for Linux 64 (2 of 2) (v2)

Comment 20

[Armen [:armenzg]]

Attachment: [buildbot-configs] disable downloading symbols for Linux 64 (1 of 2) (v1)

As I mentioned in IRC I found a one line problem. Reattached and carrying forward the positive review.
-BRANCHES['electrolysis']['platforms']['linux64']['download_symbols'] = False

Comment 21

[Armen [:armenzg]]

Comment on attachment 426299 [details] [diff] [review]
[staging] make linux64 to run unittests on staging (1 of 3) 

http://hg.mozilla.org/build/buildbot-configs/rev/299ecb8cbe12

Comment 22

[Armen [:armenzg]]

Comment on attachment 426748 [details] [diff] [review]
[buildbot-configs] disable downloading symbols for Linux 64 (1 of 2) (v1)

http://hg.mozilla.org/build/buildbot-configs/rev/23a90ba9945b

Comment 23

[Armen [:armenzg]]

Comment on attachment 426300 [details] [diff] [review]
[buildbotcustom] make linux64 to run unittests on staging (2 of 3) 

http://hg.mozilla.org/build/buildbotcustom/rev/c731d2bf6b14

Comment 24

[Armen [:armenzg]]

Comment on attachment 426738 [details] [diff] [review]
[custom] disable downloading symbols for Linux 64 (2 of 2) (v2)

http://hg.mozilla.org/build/buildbotcustom/rev/98d1d71282ae

Comment 25

[Armen [:armenzg]]

Attachment: [staging] bustage fix for staging

Comment 26

[Armen [:armenzg]]

Comment on attachment 427168 [details] [diff] [review]
[staging] bustage fix for staging

http://hg.mozilla.org/build/buildbot-configs/rev/5726db2625e5

Comment 27

[Armen [:armenzg]]

moz2-linux-slave09 is running unit tests on sm03 since yesterday.
Builds are looking good. The only builds that are orange so far are:
* "mozilla-central opt test mochitest-other"
* "mozilla-central debug test mochitest-other"
I will look again tomorrow morning.

Comment 28

[Armen [:armenzg]]

These are the other oranges
* Linux x86-64 places opt test mochitest-other 
* Linux x86-64 places debug test mochitest-other
* Linux x86-64 mozilla-central opt test mochitest-other
* Linux x86-64 mozilla-central debug test mochitests-1/5

It seems that mochitest-other is consistent orange and 1/5 only happened once.

Comment 29

[Armen [:armenzg]]

To fully support Linux 64 we have to run unit tests rather than the other way around. Reversed the dependency/blockage.

Comment 30

[Armen [:armenzg]]

I have put slave09 back into the pool.

Latest updates with this bug is that we are going to be running unittests on talos machines (bug 548768). I will wait for next week's meeting to find out where do I fit in all the remaining work.

Lowering priority until then.

Comment 31

[alice nodelman [:alice] [:anode]]

talos-r3-fed64-001 has been set aside for this work.

Please note when you are done with this box so that it can be put back in the production talos pool.

Comment 32

[Armen [:armenzg]]

Reversing the dependency.

Unit tests for Linux 64 bits will be run on user desktop (talos - bug 548768) instead of build machines.

Comment 33

[Armen [:armenzg]]

Adjusting the priority.

We are first deploying unit tests on Leopard and Snow Leopard.
Later we will deploy unit tests on Fedora and Fedora 64.

There is really nothing being worked on this bug but waiting for bug 548768 to have unit tests running on Fedora 64.

Comment 34

[Armen [:armenzg]]

There is nothing really to be done in this bug.

We are currently running Fedora 64 bit unit tests on production.
We will enable unit tests on more branches once all perma-oranges are dealt with and we have the capacity of Rev3 testing machines that we need.

For anyone that wants to follow when all oranges will be done follow bug 554934.
If you want to know on which branches we will enable the unit tests follow bug 557918.