557572 - enable building/uploading breakpad symbols on Maemo Linux/ARM builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[(not currently active) Ted Mielczarek]

+++ This bug was initially created as a clone of Bug #554138 +++

Once we enable building the Breakpad code in bug 555674, we should enable building and uploading symbols on the Maemo Linux/ARM builds.

Comment 1

[Mark Finkle (:mfinkle) (use needinfo?)]

Aki, any idea on the time frame for this?

Comment 2

[Aki Sasaki (not active)]

I can work on it.
Rearranging my schedule to get the appropriate stuff in for 1.1rc1.

Comment 3

[Aki Sasaki (not active)]

Attachment: massive mozconfig rearrange + edits

Rearrange/edit mobile mozconfigs. I did the good-practice 'hg mv' and 'hg cp' steps and then my repo got corrupted, and then it was all 'hg rm' and 'hg add' while muttering.

So I:

1) created a mobile/ dir tree for mobile mozconfigs. This should hopefully have a more intuitive layout, and it prevented me from having to create a mobile-desktop-1.1 directory.
2) added the following to mobile-1.1 mozconfigs for crash reporting (this bug and bug 562969):

  export CFLAGS="-gdwarf-2"
  export CXXFLAGS="-gdwarf-2"
  export MOZ_DEBUG_SYMBOLS=1
  export MOZILLA_OFFICIAL=1

3) added the following to mobile-browser mozconfigs, also for crash reporting:

  export MOZ_DEBUG_SYMBOLS=1
  ac_add_options --enable-debug-symbols="-gdwarf-2"
  export MOZILLA_OFFICIAL=1

4) added --enable-official-branding to release mozconfigs if not there.

5) added --disable-ipc to desktop mozconfigs (ipc seems to be e11s only).

6) pulled the android mozconfig from here: https://wiki.mozilla.org/Mobile/Fennec/Android#Full_Build

7) pointed the mobile_configs to these

8) pointed the release configs to these. I renamed maemo to maemo4 there, in preparation for adding maemo5-gtk.  I will be running a test release when working on bug 538511, so should weed out release mozconfig issues then.

Comment 4

[Aki Sasaki (not active)]

Mark, Ted: do the above changes look correct, at least in summary?  If you want to look at the actual mozconfigs created in mozilla2*/mobile/ in the above patch, cool; otherwise confirming that the logic in comment 3 is correct that should hopefully be sufficient.

Comment 5

[Aki Sasaki (not active)]

Attachment: same thing, but with checkconfig

only changed one line, in release_mobile_master.py (s,branchConfig,mobileBranchConfig, in the l10n repack factory); now passes checkconfig =P

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to comment #3)
> 3) added the following to mobile-browser mozconfigs, also for crash reporting:
> 
>   export MOZ_DEBUG_SYMBOLS=1
>   ac_add_options --enable-debug-symbols="-gdwarf-2"
>   export MOZILLA_OFFICIAL=1

If this is only for builds against mozilla-central, then drop the "export MOZ_DEBUG_SYMBOLS=1" line.

Comment 7

[(not currently active) Ted Mielczarek]

Oh, and those options aren't right for Windows desktop builds. For those, just use --enable-debug-symbols for mozilla-central, and "export
MOZ_DEBUG_SYMBOLS=1" by itself for branch.

Comment 8

[Aki Sasaki (not active)]

Attachment: with ted's corrections

updated the mozconfigs:

- removed export C*FLAGS from win32-i686 mozconfigs
- removed export MOZ_DEBUG_SYMBOLS=1 from mobile-browser/ mozconfigs
- removed ='-gdwarf-2' from win32-i686 mobile-browser/ --enable-debug-symbols

which also ended up removing a typo :)

Comment 9

[Aki Sasaki (not active)]

Attachment: [wip][untested] buildsymbols, uploadsymbols, make upload (factories)

I might be feature-creeping this bug too much.
This patch started fixing bug 524519 (make upload for mobile) and I ran smack-dab into the middle of two other bugs: bug 525327 (split maemo multi-locale) and bug 527928 (allow uploading directories in post_upload.py).

I also thought we might be able to bypass both by setting a post_upload.py --subdir option, which would upload all the binaries to tinderbox-builds/123456790/subdir/, which we could use for the en-US portion, then upload the multilocale build without --subdir.

Though maybe I should *only* add buildsymbols + uploadsymbols and leave the upload portion til those are fixed?

Comment 10

[Aki Sasaki (not active)]

Attachment: [wip][untested] configs for same

this patch assumes attachment 444513 [details] [diff] [review] has been applied.
also renames linux-gnueabi-arm to maemo4 which makes for less ugliness.

... will probably be rewriting both of these to only touch the buildsymbols portion, depending on armen's feedback.

Comment 11

[Aki Sasaki (not active)]

Attachment: just the hg cp's + hg mv's

At jhford's request, splitting the first patch up into 3.

This one is just the hg mv's and cp's for the mozconfigs (plus removing the unused linux64/mobile-*).

The next two will have the config updates to use the new mozconfigs, and the changes to the mozconfigs.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 444717 [details] [diff] [review]
just the hg cp's + hg mv's

thanks for the split out patch :)

Comment 13

[Armen [:armenzg]]

Comment on attachment 444598 [details] [diff] [review]
[wip][untested] buildsymbols, uploadsymbols, make upload (factories)

Looks about right.
We just have to make sure that "make upload" is available to more than just mobile-trunk (I have not followed up where the target landed and where not).

Does the upload work properly for the mobile desktop builds?

/me wishes I had been able to finish up all these pieces

Good job aki and best wishes with this huge refactoring :S

Comment 15

[Aki Sasaki (not active)]

Attachment: point configs at moved mozconfigs

* changed mozconfigs to point to the new location
* renamed linux-gnueabi-arm to maemo4 to match the directory name, simplify logic
* updated release configs to rename maemo to maemo4 for consistency
** (still going with 'maemo' in l10n-changesets*.json until I understand the implications of changing that)
* desktop release builders point to desktop release mozconfigs

Phew!

These two patches will have to land together.  I'll attach patches to update the mozconfigs and add buildsymbols/uploadsymbols afterwards, then roll right into adding maemo5-gtk release builders & test all my release build changes.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 444756 [details] [diff] [review]
point configs at moved mozconfigs

looks good.

Comment 17

[Aki Sasaki (not active)]

Attachment: update moved mozconfigs for debug symbols etc.

so, *now*:

1) update android mozconfigs with https://wiki.mozilla.org/Mobile/Fennec/Android#Full_Build + MOZILLA_OFFICIAL

2) added the following to mobile-1.1 mozconfigs:

  export CFLAGS="-gdwarf-2"
  export CXXFLAGS="-gdwarf-2"
  export MOZ_DEBUG_SYMBOLS=1
  export MOZILLA_OFFICIAL=1

(no C*FLAGS for win32)

3) added the following to mobile-browser mozconfigs:

  ac_add_options --enable-debug-symbols="-gdwarf-2"
  export MOZILLA_OFFICIAL=1

(no ="-gdwarf-2" for win32)

4) added --enable-official-branding where missing in release mozconfigs

5) --disable-ipc in non-e10s (it's e11s!) mozconfigs

6) removed --disable-debug + --enable-optimize (default)

7) removed some commented out options

This patch assumes the above 2 r+'ed patches have been applied.

Comment 18

[(not currently active) Ted Mielczarek]

FYI:
e[lectrolysi]s
  1234567890
e10s

Comment 19

[Aki Sasaki (not active)]

(In reply to comment #18)
> FYI:
> e[lectrolysi]s
>   1234567890
> e10s

Yeah, wc -c betrayed me :)

Comment 20

[Aki Sasaki (not active)]

eval eval /builds/scratchbox/sbin/chroot-uid /builds/scratchbox/users/cltbld /scratchbox/tools/bin/env - $SBENV /scratchbox/tools/bin/bash -l /scratchbox/tools/bin/exec.sh build/maemo-1.9.2-nightly/mozilla-1.9.2/objdir make buildsymbols
make: Nothing to be done for `buildsymbols'.

Does 'make buildsymbols" work for mobile-1.1 + mozilla-1.9.2 ?

Comment 21

[(not currently active) Ted Mielczarek]

buildsymbols won't do anything on mozilla-1.9.2 until bug 555674 lands there.

Comment 22

[Aki Sasaki (not active)]

Cool, so it's expected. Testing a maemo m-c build then I'll post the patch.

Comment 23

[Aki Sasaki (not active)]

Attachment: factory changes for make buildsymbols/make uploadsymbols

Comment 24

[Aki Sasaki (not active)]

Attachment: configs for make buildsymbols/uploadsymbols

This patch:

- assumes the above 3 r+'ed configs patches are already applied.
- creates default envs for each platform.
- adds an upload_symbols config flag (true everywhere but maemo5 qt)
- fixes the e10s mozconfig =P
- adds the env and uploadSymbols flags to each builder

Comment 25

[Aki Sasaki (not active)]

Comment on attachment 445292 [details] [diff] [review]
factory changes for make buildsymbols/make uploadsymbols

http://hg.mozilla.org/build/buildbotcustom/rev/dd17c706eee8

Comment 26

[Aki Sasaki (not active)]

Comment on attachment 444717 [details] [diff] [review]
just the hg cp's + hg mv's

http://hg.mozilla.org/build/buildbot-configs/rev/639d7aa64e36

Comment 27

[Aki Sasaki (not active)]

Comment on attachment 444781 [details] [diff] [review]
update moved mozconfigs for debug symbols etc.

http://hg.mozilla.org/build/buildbot-configs/rev/3acc4f66e66e

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 445297 [details] [diff] [review]
configs for make buildsymbols/uploadsymbols

http://hg.mozilla.org/build/buildbot-configs/rev/654cdd19981b

Comment 29

[Aki Sasaki (not active)]

mozilla-central maemo4 builds only:

/home/cltbld/build/maemo-trunk/mozilla-central/objdir/config/nsinstall -D ./dist/crashreporter-symbols
/usr/bin/python2.5 /home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py \
	  -c --vcs-info                                          \
	  -s /home/cltbld/build/maemo-trunk/mozilla-central               \
	  ./dist/host/bin/dump_syms                                                \
	  ./dist/crashreporter-symbols                                   \
	  ./dist/bin >                                        \
	  ./dist/crashreporter-symbols/fennec-2.0a1pre-Linux-20100514233107-symbols.txt
Processing file: ./dist/bin/TestDNS
Unexpected error:  <type 'exceptions.IOError'>
Traceback (most recent call last):
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 780, in <module>
    main()
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 776, in main
    dumper.Process(arg)
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 464, in Process
    return self.ProcessDir(file_or_dir)
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 481, in ProcessDir
    if not self.ProcessFile(fullpath):
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 529, in ProcessFile
    (filename, rootname) = GetVCSFilename(filename, self.srcdirs)
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 363, in GetVCSFilename
    fileInfo = HGFileInfo(file, srcdir)
  File "/home/cltbld/build/maemo-trunk/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 276, in __init__
    rev = os.popen('hg identify -i "%s"' % srcdir, "r").readlines()[0].rstrip()
IOError: [Errno 4] Interrupted system call
make: *** [buildsymbols] Error 1

Should I back out for those? :~(

Comment 30

[Aki Sasaki (not active)]

Turned off Maemo m-c buildsymbols until this is resolved.

make buildsymbols is still on for m-1.1 (though it does nothing) and m-c desktop fennec builds.

http://hg.mozilla.org/build/buildbot-configs/rev/fb635fd85c89

Comment 31

[(not currently active) Ted Mielczarek]

Ick, I guess scratchbox's hg is causing problems there? That's...really weird.

Comment 32

[Aki Sasaki (not active)]

Turned back on: http://hg.mozilla.org/build/buildbot-configs/rev/bdad5886fa71

Comment 33

[Aki Sasaki (not active)]

just passed the buildsymbols/uploadsymbols steps successfully in the currently running maemo4 m-c nightly.

Comment 34

[Aki Sasaki (not active)]

The errors are still there, just showing up much less often now that buildsymbols are off for depend builds.

/home/cltbld/build/maemo-trunk-nightly/mozilla-central/objdir/config/nsinstall -D ./dist/crashreporter-symbols
/usr/bin/python2.5 /home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py \
	  -c --vcs-info                                          \
	  -s /home/cltbld/build/maemo-trunk-nightly/mozilla-central               \
	  ./dist/host/bin/dump_syms                                                \
	  ./dist/crashreporter-symbols                                   \
	  ./dist/bin >                                        \
	  ./dist/crashreporter-symbols/fennec-2.0a1pre-Linux-20100523014056-symbols.txt
Processing file: ./dist/bin/TestDNS
Unexpected error:  <type 'exceptions.OSError'>
Traceback (most recent call last):
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 784, in <module>
    main()
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 780, in main
    dumper.Process(arg)
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 466, in Process
    return self.ProcessDir(file_or_dir)
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 483, in ProcessDir
    if not self.ProcessFile(fullpath):
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 531, in ProcessFile
    (filename, rootname) = GetVCSFilename(filename, self.srcdirs)
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 365, in GetVCSFilename
    fileInfo = HGFileInfo(file, srcdir)
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 281, in __init__
    'parent', '--template={node|short}')
  File "/home/cltbld/build/maemo-trunk-nightly/mozilla-central/toolkit/crashreporter/tools/symbolstore.py", line 263, in read_output
    (stdout, _) = Popen(args=args, stdout=PIPE).communicate()
  File "/usr/lib/python2.5/subprocess.py", line 594, in __init__
    errread, errwrite)
  File "/usr/lib/python2.5/subprocess.py", line 1086, in _execute_child
    data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
OSError: [Errno 4] Interrupted system call
make: *** [buildsymbols] Error 1
qemu: uncaught target signal 1 (Hangup) - exiting

Exceptions limited to 1MB ??

Comment 35

[(not currently active) Ted Mielczarek]

I don't know what to tell you at this point. Clearly there's something goofy with scratchbox Python.

Comment 36

[Aki Sasaki (not active)]

Filed bug 567827.