The default bug view has changed. See this FAQ.

Add mapping for crashreporter-symbols.zip for nightly builds on ftp.mozilla.org

VERIFIED FIXED

Status

Release Engineering
General Automation
P2
normal
VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: whimboo, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa-automation-blocked])

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
For tinderbox builds we have the crashreporter-symbols.zip file available. Sadly we miss that one for the nightly builds across branches. Would it be possible to also copy/link this file from the appropriate tinderbox build which is the current Nightly?

QA would need this for crash investigation while running our Mozmill tests. Lately we discovered a lot of crashes and having that automated would be a big help for us.

Here an example:

The tinderbox build for todays nightly:
https://tbpl.mozilla.org/?rev=6e3ec93efe1d

It's folder on FTP for the Linux build:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1392728388/

The appropriate folder on FTP for the Linux Nightly build:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/02/2014-02-18-03-02-02-mozilla-central/

Most likely not only Mozmill could benefit from but also the rest of automation. It makes it way easier to get the symbols for investigation.

Right now I'm blocked on bug 619204, so I'm going to mark this bug as qa-automtaion-blocked.
(Reporter)

Updated

3 years ago
Blocks: 973978
No longer blocks: 619204
(Assignee)

Comment 1

3 years ago
You're picking up the nightly via pulse, right ? If so, the properties packageUrl and symbolsUrl are already pointing at the copy in tinderbox-builds.
(Reporter)

Comment 2

3 years ago
We are only using Pulse build notifications for the Mozmill CI systems, but also only for Nightly and NOT tinderbox builds. So no, there is also no crashreporter-symbols package listed. And I think Pulse is another level which should not be covered via this bug.

Compared to buildbot we have a lot of users who run our Mozmill tests on their own systems. I would like to make it easier for them to actually analyze crashes when those happen via a Mozmill run. The crash reporter is disabled, so they will have to get the stack locally via the symbols for the build they are  testing.

Comment 3

3 years ago
- A drive by comment -
I think solving this for the automation case should take priority over solving for the case of individual users. Individual users that are really trying to debug or investigate crashes should be running debug builds in the first place.
Agreed. For individual users running nightly builds you can just let them submit crashes to crash-stats as normal and view them that way.
(Reporter)

Comment 5

3 years ago
Can we get an update here please? We would highly appreciate if we can get this into place. A couple of projects would benefit from. Personally I would like to have full crash analysis support included in Mozmill 2.1.
Flags: needinfo?(general-automation)
(Reporter)

Comment 6

3 years ago
Asking needinfo more specifically from people.
Flags: needinfo?(nthomas)
Flags: needinfo?(general-automation)
Flags: needinfo?(catlee)
Flags: needinfo?(bhearsum)
(Assignee)

Comment 7

3 years ago
Could you please clarify what automation we need to fix this for. It seems to me we are either OK already [1], or need to change our upload process [2].

[1] Pulse on existing builds
AIUI some/all of the automation is Mozmill, which is listening to pulse. I'm looking at the latest mozilla-central nightly for Windows, and it's setting the following properties:

packageUrl	http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1398938522/firefox-32.0a1.en-US.win32.zip
symbolsUrl	http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1398938522/firefox-32.0a1.en-US.win32.crashreporter-symbols.zip

and those should be published through pulse. (ie nightly builds are published in two places on ftp.m.o, the tinderbox-builds location has the symbols). The retention on those files is currently 30 days (but subject to change, as we migrate to another archive with longer retention).

[2] If this is for long-term archaeology then we should retain crash symbols in nightly/YYYY/MM/<dated-dir>/. That's controlled by http://hg.mozilla.org/build/tools/file/1fb7364360a1/stage/post_upload.py#l154
Flags: needinfo?(nthomas)
Flags: needinfo?(catlee)
Flags: needinfo?(bhearsum)
(Reporter)

Comment 8

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #7)
> Could you please clarify what automation we need to fix this for. It seems
> to me we are either OK already [1], or need to change our upload process [2].
> 
> [1] Pulse on existing builds
> AIUI some/all of the automation is Mozmill, which is listening to pulse. I'm
> looking at the latest mozilla-central nightly for Windows, and it's setting
> the following properties:
> 
> packageUrl
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> central-win32/1398938522/firefox-32.0a1.en-US.win32.zip
> symbolsUrl
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-
> central-win32/1398938522/firefox-32.0a1.en-US.win32.crashreporter-symbols.zip

This will only help for tests directly triggered via Pulse, but we (in case of Mozmill) are also doing a lot of regression and manual testing, so Pulse will not be of any help here. 

> [2] If this is for long-term archaeology then we should retain crash symbols
> in nightly/YYYY/MM/<dated-dir>/. That's controlled by
> http://hg.mozilla.org/build/tools/file/1fb7364360a1/stage/post_upload.py#l154

Yes, that would be totally preferred by us.
(Assignee)

Comment 9

3 years ago
Created attachment 8416631 [details] [diff] [review]
[tools] Allow crashreporter-symbols.zip files in nightly/YYYY/MM/<dated-dir>

This will add about 130MB per nightly per branch, which is non-trivial but not the end of the world compared to the current nighty load. We'll continue to exclude crash symbols from nightly/latest-<branch> dirs.
Attachment #8416631 - Flags: review?(bhearsum)
(Assignee)

Updated

3 years ago
Assignee: nobody → nthomas
Priority: -- → P2
Comment on attachment 8416631 [details] [diff] [review]
[tools] Allow crashreporter-symbols.zip files in nightly/YYYY/MM/<dated-dir>

Review of attachment 8416631 [details] [diff] [review]:
-----------------------------------------------------------------

dooo ittttt
Attachment #8416631 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 11

3 years ago
Comment on attachment 8416631 [details] [diff] [review]
[tools] Allow crashreporter-symbols.zip files in nightly/YYYY/MM/<dated-dir>

https://hg.mozilla.org/build/tools/rev/ce616c283028

Sending        post_upload.py
Transmitting file data .
Committed revision 86853.

Will be in production within 30 minutes.
Attachment #8416631 - Flags: checked-in+
(Assignee)

Comment 12

3 years ago
Deployment complete, please reopen if there are any problems after the next nightly.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 13

3 years ago
Looks great. Thanks a ton, Nick!
Status: RESOLVED → VERIFIED
(Assignee)

Comment 14

3 years ago
Are we keeping these crash symbols forever ? I was looking at a related issue today and discovered we have 435G of Firefox crashreporter-symbols.zip files from the last 4.5 months. That's about 8% of all the storage allocated to Firefox nightly builds.
(Reporter)

Comment 15

3 years ago
If we do so, I have nothing against it to get older archives clobbered. It might be worth keeping them for the last month or two?
(In reply to Nick Thomas [:nthomas] from comment #14)
> Are we keeping these crash symbols forever ? I was looking at a related
> issue today and discovered we have 435G of Firefox crashreporter-symbols.zip
> files from the last 4.5 months. That's about 8% of all the storage allocated
> to Firefox nightly builds.

They should probably last as long as the builds do.
You need to log in before you can comment on or make changes to this bug.