401202 - upload Talkback symbols from 2.0.0.x release builds to symbol server

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

taken from emails:

We been doing this manually since we setup the symbol server.  We put FF200x symbols up there just because it was helpful to people, and it wasn't a huge effort.  Apparently that's gone from "nice to have" to "necessary" for some people. [snip] On trunk this happens automatically as part of the Breakpad symbol upload.  We could easily automate this if we wanted, [snip] but we need to ensure that it doesn't duplicate the work on trunk builds.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

It's definitely something we should look into automating, given
there's likely 9 months of support for 200x ahead of us. Whether
that's part of Bootstrap or not is interesting - there are
requirements below that aren't part of our current setup. Ted also has
a bug open on getting access to the symbols on the Talkback server, so
presumably he has another plan in mind.

The steps go something like this (substitute the BuildID for your build):

First copy the symbols from the tbox that made the build to stage

1, Connect to production-pacifica-vm
2, cd /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8-release/WINNT_5.2_Depend
3, cd 2007100816/symbols
4, scp 2007100816.tar.bz2
[e-mail]:/tmp/firefox-2008-symbols.tar.bz2

For the next bit you need a windows box running MSYS (dump_syms didn't
want to run on Cygwin). I used fx-win32-tbox as I don't have a MSYS
setup yet, plus it's at the colo and has helpful ssh keys.

5, connect to your MSYS box
6, make a working dir with no spaces in the path, and cd into it (I
used /c/tmp/)
7, scp [e-mail]:/tmp/firefox-2008-symbols.tar.bz2 .
   (substitute your account on stage if necessary)
8, tar xfjv firefox-2008-symbols.tar.bz2

9, cvs -d [e-mail]:/cvsroot co -d code
mozilla/toolkit/crashreporter/tools/
   (substitute your cvs account as req'd)
10, mkdir symbols
11, find 2007100816/ -name "*.pdb" | xargs code/symbolstore.py -c
./code/win32/dump_syms.exe ./symbols > ./
symbols/firefox-2.0.0.8-symbols.txt
    (this will think for a minute or three, with no output)

You should get a symbols dir that looks something like:

AccessibleMarshal.pdb/
AccessibleMarshal.pdb/470AC9B02/
AccessibleMarshal.pdb/470AC9B02/AccessibleMarshal.pdb
AccessibleMarshal.pdb/470AC9B02/AccessibleMarshal.sym
GetShortPathName.pdb/
GetShortPathName.pdb/470ACA892/
GetShortPathName.pdb/470ACA892/GetShortPathName.pdb
GetShortPathName.pdb/470ACA892/GetShortPathName.sym
ds32.pdb/
ds32.pdb/470ACA882/
ds32.pdb/470ACA882/ds32.pdb
ds32.pdb/470ACA882/ds32.sym
firefox-2.0.0.8-symbols.txt
firefox.pdb/
firefox.pdb/470ACC241/
firefox.pdb/470ACC241/firefox.pdb
firefox.pdb/470ACC241/firefox.sym
...

The second level directory names will change with each compilation;
total size is about 120MB. Now we get these files back onto stage:

12, cd symbols; tar cfjv ../firefox-2008-symbols-processed.tar.bz2 .
13, scp ../firefox-2008-symbols-processed.tar.bz2 [e-mail]:/tmp/

And publish them ..

14, connect to stage with your own account, sudo su - ffxbld
    (symbols should be owned by ffxbld)
15, cd /tmp; mkdir symbols; cd symbols
16, tar xfjv ../firefox-2008-symbols-processed.tar.bz2

[ I also switched to root to change the group on the files to ffxbld too,
  dunno if this is impt but it matches exsting files ]

17, rsync -nav . /mnt/netapp/breakpad/symbols_ffx/
18, verify output of 17
19, rsync -av . /mnt/netapp/breakpad/symbols_ffx/

20, tidy up
* Remove firefox-2008-symbols* from stage.m.o:/tmp
* Remove working dir from fx-win32-tbox

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Fx2009 was done manually.

11, should be:
find 2007100816/ -name "*.pdb" | xargs code/symbolstore.py -c
./code/win32/dump_syms.exe ./symbols > ./
symbols/firefox-2.0.0.8-WINNT-2007100816-symbols.txt

to match what Ted did originally. This file would be used to clean up old symbols.

Comment 3

[(not currently active) Ted Mielczarek]

Just realized today that we left out one extra step here, run as ffxbld in the symbols_ffx dir:
cat firefox-2.0.0.9-symbols.txt | perl -F'/' -ane 'system "ln -s $F[1] $F[0]/".lc($F[1])'

This is needed to make this work in Visual Studio, since it and WinDBG disagree on what case the identifier should be for VC6 produced PDB files.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I found the txt file had two lines for each binary, one each for a sym and pdb, and ln subsequently made links like
 # good
 create symbolic link `AccessibleMarshal.pdb/4721168b2' to `4721168B2'
 # not good
 create symbolic link `AccessibleMarshal.pdb/4721168b2/4721168B2' to `4721168B2'

So a slightly different command was needed:

grep sym firefox-2.0.0.9-WINNT-2007102514-symbols.txt | perl -F'/' -ane 'system "ln -vs $F[1] $F[0]/".lc($F[1])'

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I'll get the FF2.0.0.11 symbols, and work on an automation patch for this.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> I'll get the FF2.0.0.11 symbols

This is now done. Patch in progress. 

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: static dump_syms.exe [checked in]

The version of dump_syms.exe checked in at mozilla/toolkit/crashreporter/tools/win32/ depends on the VC8 runtime dll's, so it doesn't run on older tinderboxes that make Talkback symbols. Ted made this static version for me on Dec 11, and IIRC, said it should be checked in as a replacement. 

Should I go ahead and do that ? Any testing req'd first ? What about approval ?

Comment 9

[(not currently active) Ted Mielczarek]

r=me to check it in, but you'll probably need approval, since it will affect the Firefox build.  It shouldn't change anything though.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 299762 [details]
static dump_syms.exe [checked in]

Requesting approval to land this build tool. Any bustage would affect Win32 breakpad symbols/crash reports for nightlies only. It's not urgent, and could land after b3 is done.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #10)
It would of course hit release builds too, but we'd notice before then.


In other news, 2.0.0.12rc1 symbols are uploaded and automation patch is well in hand. Just need to figure out how to handle the symlinking in comment #4, given that win32 doesn't care about case and only some commands are available in the new staging setup.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Can we use something like mod_speling to make the d/l requests be handled in a case insensitive way ? Would that impact on Breakpad symbols at all ?

Comment 13

[(not currently active) Ted Mielczarek]

Breakpad just hits the disk directly, HTTP access is purely for the symbol server, so making them accessible case-insensitively via HTTP should work fine.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

testcase: 
 http://symbols.mozilla.org/firefox/xpt_link.pdb/470ac3db2/xpt_link.sym

This from an older Firefox release and doesn't have a symlink already set up, so it's currently a 404. 

Comment 15

[Damon Sicore (:damons)]

Due to the fact that the freeze is tonight for beta 3, I'd prefer holding off on this till after the beta.  I'll approve this, but please don't check in till after.

a1.9+=damons

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Process and push Talkback symbols after the build

* we're pulling the symbols tools from the trunk, rather than a tag. Worth creating another tag for this ?
* attachment 299762 [details] should be checked in before this patch will work, waiting on the tree to reopen for that
* I've enabled this for staging too, pushing to /data/symbols. Looks like we're pushing Talkback symbols there too (but the directories are empty!). Need to follow up with cleanup changes for 1.8, and we need to do the same for trunk's Breakpad symbols
* this has been tested on production-prometheus-vm

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Prevent build up of symbols on the staging server

1.9 staging pushes Breakpad symbols to /data/symbols on its master, and with attachment 301008 [details] [diff] [review] we'll be pushing them for 1.8 too. This should prevent them building up.

Comment 18

[(not currently active) Ted Mielczarek]

Comment on attachment 299762 [details]
static dump_syms.exe [checked in]

Checked in on trunk.

Comment 19

[Robert Helmer [:rhelmer]]

Comment on attachment 301689 [details] [diff] [review]
[checked in] Prevent build up of symbols on the staging server

Hmm.. I think the reason I didn't do it this way (at least on 1.8) is that it's a "symbols" user and cltbld can't remove this dir.

on staging-1.8-master there is a cronjob for the symbols user:
5 0 * * *       find /data/symbols/ | xargs rm

Will this work in that case? I guess we could dispense with the "symbols" and use cltbld for this on staging..

Comment 20

[Robert Helmer [:rhelmer]]

Comment on attachment 301689 [details] [diff] [review]
[checked in] Prevent build up of symbols on the staging server

Disregard previous comment; I looked at these patches in reverse order, didn't see what this was for :)

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 301008 [details] [diff] [review]
[checked in] Process and push Talkback symbols after the build

Checking in Bootstrap/Step/Build.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Build.pm,v  <--  Build.pm
new revision: 1.25; previous revision: 1.24
done
Checking in configs/fx-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-bootstrap.cfg,v  <--  fx-moz18-bootstrap.cfg
new revision: 1.34; previous revision: 1.33
done
Checking in configs/fx-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-staging-bootstrap.cfg,v  <--  fx-moz18-staging-bootstrap.cfg
new revision: 1.15; previous revision: 1.14
done

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 301689 [details] [diff] [review]
[checked in] Prevent build up of symbols on the staging server

Checking in Makefile;
/cvsroot/mozilla/tools/release/Makefile,v  <--  Makefile
new revision: 1.25; previous revision: 1.24
done

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Fixed, bar any shake out required for the staging setup.

Comment 24

[Robert Helmer [:rhelmer]]

Attachment: fix up nightly config too

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 302894 [details] [diff] [review]
fix up nightly config too

the more the merrier, but don't forget to add 
  symbolDir       = /builds/symbols
on checkin.

Comment 26

[Robert Helmer [:rhelmer]]

Attachment: [checked in] fix up nightly config [as landed]

Checking in fx-moz18-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v  <--  fx-moz18-nightly-bootstrap.cfg
new revision: 1.5; previous revision: 1.4
done

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

This patch doesn't handle nighlties off the automation very well, complaining that "/builds/symbols/firefox-nightly/rc1 already exists". 

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Test for a symbol archive before really getting started

Something like this should work, but needs testing.

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 303583 [details] [diff] [review]
Test for a symbol archive before really getting started

Rob pointed out that this will still fail, because GetBuildID looks for a string that only appears when generating a complete update. There are plenty of other places in the build log we could get the buildid from, lets just disable pushing talkback symbols to the symbol server for hourlies & nightlies.

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Disable Talkback symbol push for nightlies

I'm going to go ahead an commit this simple change, to unbreak the staging version of 1.8 win32 nightlies. Looking for retro-spective review.

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 304021 [details] [diff] [review]
[checked in] Disable Talkback symbol push for nightlies

Checking in Bootstrap/Step/Build.pm;
/cvsroot/mozilla/tools/release/Bootstrap/Step/Build.pm,v  <--  Build.pm
new revision: 1.26; previous revision: 1.25
done

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: including bustage fix

Could have sworn I ran perl -c on this, but apparently not. This is the diff between rev 1.25 (before the original checkin), and rev 1.27 (after the fix).

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Clean up the symbols on the slaves

The prestage factories for each slave call |make clean_stage|, and that's also used for the master. The new rm on /builds/symbols/* is for the slave, the existing one on /data/symbols for the master. Sorry I missed this.

Comment 34

[Robert Helmer [:rhelmer]]

Comment on attachment 306554 [details] [diff] [review]
Clean up the symbols on the slaves

>Index: Makefile
>===================================================================
>RCS file: /cvsroot/mozilla/tools/release/Makefile,v
>retrieving revision 1.25
>diff -U10 -r1.25 Makefile
>--- Makefile	12 Feb 2008 12:11:31 -0000	1.25
>+++ Makefile	29 Feb 2008 19:20:30 -0000
>@@ -65,20 +65,21 @@
> 	chgrp -R cvs /builds/cvsmirror.clean/mofo
> 	chmod g+rwx /builds/cvsmirror.clean/mofo
> 	chmod -R g+rw /builds/cvsmirror.clean/mofo
> 
> clean_stage: clean_logs
> 	rm -rf /builds/config/*
> 	rm -rf /builds/tags/*
> 	rm -rf /builds/release/logs/*
> 	rm -rf /builds/updates/*
> 	rm -rf /builds/verify/*
>+        rm -rf /builds/symbols/*
> 	rm -rf /data/cltbld/${PRODUCT}-*/
> 	rm -rf /data/symbols/*
> 	rm -rf /home/ftp/pub/${PRODUCT}/*

Make needs tab not spaces for the new instruction. r=me with that fixed.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Landed with a tab,
Checking in Makefile;
/cvsroot/mozilla/tools/release/Makefile,v  <--  Makefile
new revision: 1.26; previous revision: 1.25
done