sort out breakpad symbol upload for stage migration


Status Graveyard
Server Operations
11 years ago
3 years ago


(Reporter: ted, Assigned: justdave)





11 years ago
Currently tinderboxen (both MoCo & community) upload the symbols for breakpad so stage:/mnt/netapp/breakpad/.  justdave says this will break with the new stage architecture, so we'll need to sort this out.  The tinderboxen are using the role keys (ffxbld etc) to do this.  I also have been occasionally uploading OS symbols to stage:/mnt/netapp/breakpad/symbols_os using my LDAP auth, so that process needs to be enabled as well.


11 years ago
Blocks: 394069
Assignee: server-ops → justdave
Accounts are chrootable individually -- for example we can set it up so a given account is chrooted somewhere else.  This is probably what we need to do, and set up an account specific for symbol uploads (or a couple perhaps if there's different people running different ones), and set those accounts to chroot to the directories where the symbols go instead of the ftp staging area.

Comment 2

10 years ago
Probably cf and coop can comment on this specifically.  Right now we're using the tinderbox role keys, but that's all configurable on the tinderbox.
What accounts are being used for this currently, and are they also being used to upload stuff to ftp staging as well?
(In reply to comment #3)
> What accounts are being used for this currently, ...

ffxbld, tbirdbld, calbld, seabld

> and are they also being used to upload stuff to ftp staging as well?

we moving to these keys (and caminobld, xrbld) for ftp staging (bug 399848).

To give you an idea of the functionality already in tinderbox/breakpad, the config variables for symbol push are like this:

$ENV{'SYMBOL_SERVER_USER'}   = 'ffxbld';
$ENV{'SYMBOL_SERVER_PATH'}   = '/mnt/netapp/breakpad/symbols_ffx/';
$ENV{'SYMBOL_SERVER_SSH_KEY'}   = "$ENV{HOME}/.ssh/ffxbld_dsa";

OK, so we need to do one of two things here...  either we have these users upload the symbols to a different server, or we set up additional users to use for symbol upload so they can be chrooted to a different place.  Which one would you prefer?
Priority: -- → P2
A different server is a lot less work on our side, since we don't need to distribute keys to a bunch of tinderboxes. How about for you ?
Actually, that's not quite true - we'll have to accept the sig of a new machine on each tinderbox. 

I guess I'd prefer to not have a bunch more keys to deal with, but that's not a very compelling reason to set up another box. Perhaps you had an existing box in mind ?
I was thinking dm-stage01 (kinda like the original plan before we simplified it)
If you're okay with me going forward with this, I'll see what I can do to get this set up on dm-stage01.
Hmm, I thought dm-stage01 was to be "locked down" since it has r/w access to the netapp. Aren't we sacrificing that if we setup symbol upload there ? Were you thinking of a chroot jail again, which would mostly mitigate that risk ?
Yeah, it would get chrooted, just to a different place.  If I recall correctly, the symbols partition is a separate partition on the netapp, and surf was just a convenient place to put it.  It's probably conceivable that it could be mounted from any other arbitrary host (perhaps a VM in the build network?) and uploaded to there.  Do outside tinderboxes ever upload symbols?

Comment 12

10 years ago
We have symbols uploaded from community boxes, but yeah, this was on surf just because it was a place to mount it.
(In reply to comment #11)
> [...] Do outside tinderboxes ever upload symbols?
KaiRo compiles some Sm versions on his own tinderboxen (in Austria IIUC). I'm not sure if trunk or branch, but I'm adding him to the Cc list so he can answer your question.

Comment 14

10 years ago
Only the Mozilla community tinderboxen upload symbols though, my private ones don't.
What's the target for usage of these symbols?  Does it just have to go somewhere that socorro/talkback can get to them? If so it sounds like we just need any old server that's accessible from both the build and community build vlans and can mount the netapp share.  We could set up a VM for this instead of using stage.

Comment 16

10 years ago
That's pretty much it, the machine doing the crash report processing mounts this same share and reads the files, and we also serve them via HTTP for the symbol server, but it's all just files.

Comment 17

10 years ago
Dave: whatever box we hang this off of, can you get me a shell account there? I occasionally wind up manually verifying some symbols' existence or contents, and it's easier with shell access.
Dave, are you setting up a VM for this ? Do you have a hostname ready or should I send symbols to stage-old for the moment ?

Need to work up a patch ahead of the switch.
Should have it up within the next couple hours.  Hostname is probably going to be dm-symbolpush01, since I noticed we have a bm-symbolfetch01 and it seems to go well with it.  I was also considering dm-symbolshove01, but that sounds a little more abrasive. :)
Ok, I'll assume it's going to be dm-symbolpush01.m.o
If you could use the same mountpoint that'd be a great help. (eg /mnt/netapp/breakpad, so that /mnt/netapp/breakpad/symbols_ffx is the push dir for Firefox t'boxes)
Tinderboxes time out trying to reach Is there a firewall restriction ? Both community and build tinderboxes will need to use this machine.

This isn't a blocker, since the DNS switch will be after the Friday nightlies, but we'd need it fixed during Friday.
It also needs to be in the public DNS so the community boxes can resolve it.
Hmm, this was done. :)

It's in the DMZ just like surf was, and in the public DNS.

I'm pretty sure all the relevant firewall rules between the vlans got cloned as well.

Any issues, open new bugs.
Last Resolved: 10 years ago
Resolution: --- → FIXED
Product: → Graveyard
You need to log in before you can comment on or make changes to this bug.