790741 - Store gecko ICS b2g builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Jonathan Griffin (:jgriffin)]

The gecko ics b2g builds that buildbot makes (labelled Armv7a ICS opt/debug in TBPL) apparently don't get stored anywhere, I see only build logs, but not the builds themselves, at e.g., http://stage.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-ics_armv7a_gecko/. 

We'll need to have these builds stored so that they can be downloaded and utilized by the emulator unittests being worked on in bug 789652.

Comment 1

[[e-mail] (:bhearsum)]

It looks like we don't run 'make upload' for these builds. This might be as simple as flipping this flag: https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/b2g_config.py#L62
and this one for debug: https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/b2g_config.py#L111

Probably should be tested in staging first, though.

Comment 2

[Rail Aliiev [:rail]]

We may also want to use/deploy different ssh keys for these uploads.

Comment 3

[Chris AtLee [:catlee]]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=767515#c8

Comment 4

[Aki Sasaki (not active)]

Attachment: wip: b2g make upload

I was able to get uploads working, albeit without a |make package-tests| step or a sendchange.

I enabled packageTests and unittest_masters, and now I'm hitting an exception in |make buildsymbols|.  I'm not entirely sure why we run |make buildsymbols| when we're not going to upload them.  The solution here could be to disable |make buildsymbols|, or to fix the exception.

OBJCOPY="" \
/builds/slave/m-cen-ics-armv7a-gecko/build/obj-b2g/_virtualenv/bin/python /builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py \
  -c --vcs-info                                          \
  --exclude="*test*" --exclude="*Test*"                           \
  -s /builds/slave/m-cen-ics-armv7a-gecko/build               \
  ./dist/host/bin/dump_syms                                                \
  ./dist/crashreporter-symbols                                   \
  ./dist/bin >                                        \
  ./dist/crashreporter-symbols/b2g-18.0a1-Android-20121002155756-arm-symbols.txt
Processing file: ./dist/bin/mangle
Unexpected error:  <type 'exceptions.OSError'>
Traceback (most recent call last):
  File "/builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py", line 791, in <module>
    main()
  File "/builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py", line 787, in main
    dumper.Process(arg)
  File "/builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py", line 451, in Process
    return self.ProcessDir(file_or_dir)
  File "/builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py", line 468, in ProcessDir
    if not self.ProcessFile(fullpath):
  File "/builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py", line 543, in ProcessFile
    self.CopyDebug(file, debug_file, guid)
  File "/builds/slave/m-cen-ics-armv7a-gecko/build/toolkit/crashreporter/tools/symbolstore.py", line 646, in CopyDebug
    if subprocess.call([self.objcopy, '--only-keep-debug', file, file_dbg]) == 0 and \
  File "/usr/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1228, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied
make: *** [buildsymbols] Error 1

Comment 5

[Aki Sasaki (not active)]

Attachment: upload build

This portion works... uploads the build.
It's missing the test zip and the sendchange.

I figure we should land this and figure out what's going on with |make buildsymbols| in the meantime.

Comment 6

[Aki Sasaki (not active)]

Attachment: (configs) disable buildsymbols, enable sendchange

This depends on attachment 667232 [details].

I'm ok with all of this except for the disable_symbols ugliness.  I tried annotating process/factory.py and followed those lines through history, but I never figured out why we want to run buildsymbols steps if we're not going to upload them.  Maybe cruft from when we ran unittests on the build machines?

Up to you if you want to land this as is, or figure out what's going on with |make buildsymbols|, or find a more elegant solution.

Comment 7

[Aki Sasaki (not active)]

Attachment: (custom) disable buildsymbols, detect b2g .tar.gz file

The good:
In staging, I ran with all 3 patches applied and got this in |make upload|:

packageUrl: 'http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-ics_armv7a_gecko/1349222573/b2g-18.0a1.en-US.eabi-arm.tar.gz'
testsUrl: 'http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-ics_armv7a_gecko/1349222573/b2g-18.0a1.en-US.eabi-arm.tests.zip'

and this in the sendchange:

['buildbot', 'sendchange', '--master', 'dev-master01.build.scl1.mozilla.com:9901', '--username', 'sendchange-unittest', '--branch', 'mozilla-central-ics_armv7a_gecko-opt-unittest', '--revision', '965f6dc789be', '--property', 'buildid:20121002170253', '--property', 'pgo_build:False', '--property', 'builduid:97d4eebcdf2b4191b4050059c7014caf', 'http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-ics_armv7a_gecko/1349222573/b2g-18.0a1.en-US.eabi-arm.tar.gz', 'http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-ics_armv7a_gecko/1349222573/b2g-18.0a1.en-US.eabi-arm.tests.zip']

which went red because there's no listener on port 9901.

But this should allow us to define unittests on the mozilla-central-ics_armv7a_gecko-opt-unittest branch on the test masters.


The bad: the disable_symbols stuff is still ugly.

Comment 8

[Armen [:armenzg]]

Comment on attachment 667232 [details]
upload build

This seems to be an obsolete patch.

The only part not appearing on the following patch is:
+            'mock_copyin_files': [
+                ('/home/cltbld/.ssh', '/home/mock_mozilla/.ssh'),
+            ],

I will keep an eye on it. I think it is needed to be able to call make upload.

Comment 9

[Armen [:armenzg]]

Comment on attachment 667252 [details] [diff] [review]
(custom) disable buildsymbols, detect b2g .tar.gz file

Review of attachment 667252 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +938,5 @@
>  
>          self.multiLocale = multiLocale
>  
>          self.addBuildSteps()
> +        if self.uploadSymbols or (not self.disableSymbols and (self.packageTests or self.leakTest)):

I have always eyed this line but never dared to touch since I could not think of a cleaner way.

hg blame will keep your name *forever*. bua bua ha ha!

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 667232 [details]
upload build

This is not an obsolete patch; note that the other patch doesn't enable uploadPackages; it just assumes this patch has been applied first.

Comment 11

[Armen [:armenzg]]

Comment on attachment 667232 [details]
upload build

I'm putting now all three patches on staging.
As soon as I see it run I will enable it on production.
I'm on release duty today and would like to be double sure before enabling it on production.

Comment 12

[Armen [:armenzg]]

This is now live in production.

Comment 13

[Armen [:armenzg]]

I'm triggering builds on Cedar to see how it comes out.

Comment 14

[Armen [:armenzg]]

Attachment: disable sendchanges

I have landed a bustage fix to disable the sendchanges since it was turning the builds orange and could cause dev distraction.

We have now started uploading the gecko builds, the checksum, the tests zip and the txt with the revision info besides the usual logs:
http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko/1349452518/


FTR, this is what the sendchange was looking like:
buildbot sendchange --master buildbot-master36.build.mozilla.org:9301 \
 --username sendchange-unittest \
 --branch cedar-ics_armv7a_gecko-debug-unittest \
 --revision 120d8a0ad729 \
 --comments 'Merge m-c -_ cedar' \
 --property buildid:20121005085518 \
 --property pgo_build:False \
 --property builduid:a503a445365945d4bc295eaa932ece8f \
 http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko-debug/1349452518/b2g-18.0a1.en-US.eabi-arm.tar.gz \
 http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cedar-ics_armv7a_gecko-debug/1349452518/b2g-18.0a1.en-US.eabi-arm.tests.zip

Comment 15

[Armen [:armenzg]]

I had to back all out because we were causing upload failures on try.
I will try this again on Tuesday.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15849819&tree=Try&full=1

Comment 16

[Rail Aliiev [:rail]]

backout in production

Comment 17

[Armen [:armenzg]]

Attachment: Use the trybld_dsa key

dump_master.py says that this does the job.

Comment 18

[Armen [:armenzg]]

Landed all patches.
Waiting for the right time to put it live in production (waiting on release work).

Comment 19

[Armen [:armenzg]]

Live in production.

Comment 20

[Chris AtLee [:catlee]]

Pushed a bustage fix: http://hg.mozilla.org/build/buildbot-configs/rev/a052f74d1854

Comment 21

[Armen [:armenzg]]

Thanks catlee!

These builds now show up on ftp:
http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-ics_armv7a_gecko/1350063026/

Comment 22

[Jonathan Griffin (:jgriffin)]

Awesome, nice job!

Comment 23

[Armen [:armenzg]]

FTR two more bustage fixes landed after this:
http://hg.mozilla.org/build/buildbot-configs/rev/a052f74d1854
http://hg.mozilla.org/build/buildbot-configs/rev/723ff04f7a76