973901 - Add mapping for crashreporter-symbols.zip for nightly builds on ftp.mozilla.org

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For tinderbox builds we have the crashreporter-symbols.zip file available. Sadly we miss that one for the nightly builds across branches. Would it be possible to also copy/link this file from the appropriate tinderbox build which is the current Nightly?

QA would need this for crash investigation while running our Mozmill tests. Lately we discovered a lot of crashes and having that automated would be a big help for us.

Here an example:

The tinderbox build for todays nightly:
https://tbpl.mozilla.org/?rev=6e3ec93efe1d

It's folder on FTP for the Linux build:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1392728388/

The appropriate folder on FTP for the Linux Nightly build:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/02/2014-02-18-03-02-02-mozilla-central/

Most likely not only Mozmill could benefit from but also the rest of automation. It makes it way easier to get the symbols for investigation.

Right now I'm blocked on bug 619204, so I'm going to mark this bug as qa-automtaion-blocked.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

You're picking up the nightly via pulse, right ? If so, the properties packageUrl and symbolsUrl are already pointing at the copy in tinderbox-builds.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We are only using Pulse build notifications for the Mozmill CI systems, but also only for Nightly and NOT tinderbox builds. So no, there is also no crashreporter-symbols package listed. And I think Pulse is another level which should not be covered via this bug.

Compared to buildbot we have a lot of users who run our Mozmill tests on their own systems. I would like to make it easier for them to actually analyze crashes when those happen via a Mozmill run. The crash reporter is disabled, so they will have to get the stack locally via the symbols for the build they are  testing.

Comment 3

[cmtalbert]

- A drive by comment -
I think solving this for the automation case should take priority over solving for the case of individual users. Individual users that are really trying to debug or investigate crashes should be running debug builds in the first place.

Comment 4

[(not currently active) Ted Mielczarek]

Agreed. For individual users running nightly builds you can just let them submit crashes to crash-stats as normal and view them that way.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Can we get an update here please? We would highly appreciate if we can get this into place. A couple of projects would benefit from. Personally I would like to have full crash analysis support included in Mozmill 2.1.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Asking needinfo more specifically from people.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Could you please clarify what automation we need to fix this for. It seems to me we are either OK already [1], or need to change our upload process [2].

[1] Pulse on existing builds
AIUI some/all of the automation is Mozmill, which is listening to pulse. I'm looking at the latest mozilla-central nightly for Windows, and it's setting the following properties:

packageUrl	http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1398938522/firefox-32.0a1.en-US.win32.zip
symbolsUrl	http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1398938522/firefox-32.0a1.en-US.win32.crashreporter-symbols.zip

and those should be published through pulse. (ie nightly builds are published in two places on ftp.m.o, the tinderbox-builds location has the symbols). The retention on those files is currently 30 days (but subject to change, as we migrate to another archive with longer retention).

[2] If this is for long-term archaeology then we should retain crash symbols in nightly/YYYY/MM/<dated-dir>/. That's controlled by http://hg.mozilla.org/build/tools/file/1fb7364360a1/stage/post_upload.py#l154

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Nick Thomas [:nthomas] from comment #7)
> Could you please clarify what automation we need to fix this for. It seems
> to me we are either OK already [1], or need to change our upload process [2].
> 
> [1] Pulse on existing builds
> AIUI some/all of the automation is Mozmill, which is listening to pulse. I'm
> looking at the latest mozilla-central nightly for Windows, and it's setting
> the following properties:
> 
> packageUrl
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> central-win32/1398938522/firefox-32.0a1.en-US.win32.zip
> symbolsUrl
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> central-win32/1398938522/firefox-32.0a1.en-US.win32.crashreporter-symbols.zip

This will only help for tests directly triggered via Pulse, but we (in case of Mozmill) are also doing a lot of regression and manual testing, so Pulse will not be of any help here. 

> [2] If this is for long-term archaeology then we should retain crash symbols
> in nightly/YYYY/MM/<dated-dir>/. That's controlled by
> http://hg.mozilla.org/build/tools/file/1fb7364360a1/stage/post_upload.py#l154

Yes, that would be totally preferred by us.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Allow crashreporter-symbols.zip files in nightly/YYYY/MM/<dated-dir>

This will add about 130MB per nightly per branch, which is non-trivial but not the end of the world compared to the current nighty load. We'll continue to exclude crash symbols from nightly/latest-<branch> dirs.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8416631 [details] [diff] [review]
[tools] Allow crashreporter-symbols.zip files in nightly/YYYY/MM/<dated-dir>

Review of attachment 8416631 [details] [diff] [review]:
-----------------------------------------------------------------

dooo ittttt

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8416631 [details] [diff] [review]
[tools] Allow crashreporter-symbols.zip files in nightly/YYYY/MM/<dated-dir>

https://hg.mozilla.org/build/tools/rev/ce616c283028

Sending        post_upload.py
Transmitting file data .
Committed revision 86853.

Will be in production within 30 minutes.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Deployment complete, please reopen if there are any problems after the next nightly.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks great. Thanks a ton, Nick!

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Are we keeping these crash symbols forever ? I was looking at a related issue today and discovered we have 435G of Firefox crashreporter-symbols.zip files from the last 4.5 months. That's about 8% of all the storage allocated to Firefox nightly builds.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

If we do so, I have nothing against it to get older archives clobbered. It might be worth keeping them for the last month or two?

Comment 16

[(not currently active) Ted Mielczarek]

(In reply to Nick Thomas [:nthomas] from comment #14)
> Are we keeping these crash symbols forever ? I was looking at a related
> issue today and discovered we have 435G of Firefox crashreporter-symbols.zip
> files from the last 4.5 months. That's about 8% of all the storage allocated
> to Firefox nightly builds.

They should probably last as long as the builds do.