458903 - Have Talos boxes submit crash reports when they crash

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Shawn Wilsher :sdwilsh]

It'd be really useful to see why something crashed on talos, especially when the issue is intermittent or difficult to reproduce locally.  A backtrace can often point a developer in the right direction about what's wrong with a piece of code.

Comment 1

[alice nodelman [:alice] [:anode]]

To make this happen the dependent bugs would have to fixed along with:

- some method of managing the glut of symbols that we would build up
- changes to the talos code to handle crashes differently and ensure that the autosubmit worked correctly

Comment 2

[(not currently active) Ted Mielczarek]

I filed bug 465759 about improving the cleanup script. (It's blocking bug 385785 which is blocking this bug.)

Comment 3

[Benjamin Smedberg]

Attachment: Blind patch to buildbot configs for enabling symbol fetching for talos-staging, rev. 1

Alice, this is my blind attempt at updating the buildbot configs to make symbols/crash reporting work. Let me know the best way to get this tested, because I don't think setting up a local buildbot config will work very well.

Comment 4

[alice nodelman [:alice] [:anode]]

I'm working through the issues in the patch on talos staging - so far I've got symbols downloading and unzipping.  Once I'm convinced that things are working correctly I'll submit a patch that covers the changes that I've made.

Comment 5

[Benjamin Smedberg]

Awesome. I've committed the Talos code changes, so I think that the buildbot bits are all that's left!

Comment 6

[alice nodelman [:alice] [:anode]]

Attachment: configs for enabling symbold fetching for talos-staging, rev2

Did some small revisions to make things run on stage.  I'm seeing symbols be downloaded and unpacked, unfortunately I haven't had any crashes come through yet to actually see if they get picked up.

Comment 7

[Benjamin Smedberg]

Looks good. I have a repo which will crash for testing: http://hg.mozilla.org/users/bsmedberg_mozilla.com/crashme if that helps. Also, if we got this deployed on tryserver it would be easy to test, except that tryserver doesn't currently upload symbol packages.

Comment 8

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]configs for enabling symbold fetching for talos-staging, rev3

Issues with growing arrays of commands - due to shallow copy issue.  Added copy.copy to ensure that we are only altering a local copy of an array instead of accidentally updating the original array.

Comment 9

[alice nodelman [:alice] [:anode]]

Comment on attachment 366392 [details] [diff] [review]
[Checked in]configs for enabling symbold fetching for talos-staging, rev3

999:469a5b5c452f

Comment 10

[alice nodelman [:alice] [:anode]]

Looks to me like we are only creating crashreporter-symbols.zip files for tracemonkey and moz-central (not 1.9.1 or 1.9.0) - is this true?

Comment 11

[Benjamin Smedberg]

That's correct, there is no automation for creating those on the branches. For the moment this is trunk-only.

Comment 12

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]configs for enabling symbold fetching for production talos, rev1

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 366943 [details] [diff] [review]
[Checked in]configs for enabling symbold fetching for production talos, rev1

The __init__ methods for MozillaWgetSymbols and MozillaUnpackSymbols don't look like they're doing anything, so they could probably be left out.

Comment 14

[alice nodelman [:alice] [:anode]]

Comment on attachment 366943 [details] [diff] [review]
[Checked in]configs for enabling symbold fetching for production talos, rev1

8f213be2b8ee

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #10)
> Looks to me like we are only creating crashreporter-symbols.zip files for
> tracemonkey and moz-central (not 1.9.1 or 1.9.0) - is this true?

(In reply to comment #11)
> That's correct, there is no automation for creating those on the branches. For
> the moment this is trunk-only.

Err... can we get this working on mozilla-1.9.1? 

I can maybe understand ignoring mozilla-1.9.0, but mozilla-1.9.1 is in active development for a while still, and these crash reports would likely be really useful in the leadup to the release of Firefox 3.5.

Reopening, or is this already being tracked in a different bug?

Comment 16

[(not currently active) Ted Mielczarek]

John, bugs are fixed when they're fixed on trunk. We have keywords for tracking branch fixed-ness. It's quite possible we can land the necessary bits on 1.9.1, it's just build config stuff. We'll track as necessary with keywords.

Comment 17

[(not currently active) Ted Mielczarek]

I landed bug 478221 on 1.9.1, so the 1.9.1 build machines are uploading symbols now. alice says we just need to modify the Talos config for the 1.9.1 machines to make them download symbols.

Comment 18

[alice nodelman [:alice] [:anode]]

Attachment: [Checked in]have talos 1.9.1 boxes download symbols

Comment 19

[alice nodelman [:alice] [:anode]]

Comment on attachment 369302 [details] [diff] [review]
[Checked in]have talos 1.9.1 boxes download symbols

changeset:   1041:c3922bdde04b