461020 - Need a stack trace to aid debugging

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Blake Kaplan (:mrbkap) (inactive)]

When speculative script parsing (bug 364315) landed, we started seeing intermittent orange on the Talos boxes due to crashes during Tp. Eventually, I disabled speculative parsing so that the trees would be green for beta1. However, it would be nice to re-enable speculative parsing.

The problem is that it isn't at all clear why the Talos boxes are crashing. As far as I know, nobody reported any speculative parsing related crashes or was able to reproduce the crashiness. I did try to install standalone Talos; however, I was unable to reproduce the crash with the regular pageset and installing the "full" pageset into my standalone talos caused it to not work at all (as far as I could tell, the browser refused to start at all, becoming a defunct process immediately).

I was able to hack something up to run through the pageset, but after ~40 cycles across 4 machines, I crashed only once in seemingly unrelated code.

It would be really nice if I could get some sort of crash stack for the crashes that Talos is seeing so I could direct my debugging efforts directly at the problems that Talos is running into (instead of staring at the code, trying to come up with any possible problem).

If you need any other information, please ask.

Comment 1

[Blake Kaplan (:mrbkap) (inactive)]

I really need this to move forward on speculative parsing...

Comment 2

[alice nodelman [:alice] [:anode]]

What operating systems were the crashes observed on?

We don't really have any system in place for being able to pull stack traces from builds and there is no crash reporting/symbols in tinderbox builds.

Would having access to a fully configured talos box that you could do your own testing on get you moving forward?  I suggest this because it is doubtful that we could pull together a full automated system in any sort of timely fashion.

Comment 3

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to comment #2)
> What operating systems were the crashes observed on?

We were seeing crashes on all 3 platforms.

> Would having access to a fully configured talos box that you could do your own
> testing on get you moving forward?  I suggest this because it is doubtful that
> we could pull together a full automated system in any sort of timely fashion.

That would be brilliant. All I really need here is a one-off for a couple of hours (possibly) to get a stack trace. Having a full-on automated way of doing this is far more work than I'm asking for.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Stuart: Is this on mozilla-central or on the tracemonkey branch? We have talos machines on each, 

Alice: Is it ok to take one of the existing slaves out of production, for manual experiment? Or would it be safer instead to we setup a new mac mini, as a clone of an existing production slave? Or...any other ideas?

Comment 5

[Blake Kaplan (:mrbkap) (inactive)]

John, this is mozilla-central.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> Alice: Is it ok to take one of the existing slaves out of production, for
> manual experiment? Or would it be safer instead to we setup a new mac mini, as
> a clone of an existing production slave? Or...any other ideas?

After talking to Alice, we do not want to take an existing talos slave out of production, or risk destabilizing the talos cluster with any manual experiments being carried out in the colo. 

Instead, do you have a mac mini on your desk that we can put a talos slave image onto? This would still need some manual setup afterwards (apache config, talos install, pageloader install spring to mind right now), but its a lot safer then risking polluting the production talos cluster.

Comment 7

[Blake Kaplan (:mrbkap) (inactive)]

I don't have a mac mini.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

in meeting with bkaplan: this crash can be reproduced on the try server talos.

Comment 9

[alice nodelman [:alice] [:anode]]

It would be good if we could reproduce on try server talos - but we can't get stack traces/core dumps off of those machines.

Comment 10

[(not currently active) Ted Mielczarek]

If you can crash on the win32 try server talos, and we can configure it to save a minidump (which shouldn't be hard), we do in fact upload symbols for the Windows builds there.

Comment 11

[Blake Kaplan (:mrbkap) (inactive)]

There was supposed to be a new followup bug filed as a result of a meeting yesterday, but I don't think I've seen it go by. John, did I miss a bugmail?

IIRC the conclusion was that a mac mini configured with Talos would end up with my ldap credentials on it, so I could make the changes necessary to get a minidump and reproduce the crash.

Comment 12

[(not currently active) Ted Mielczarek]

If you're going to do this on Try talos, or on some other Talos machine, either way, you should set
MOZ_CRASHREPORTER_NO_REPORT=1
and
MOZ_CRASHREPORTER=1
in the environment of the Windows box.

If we did this for Try talos, mrbkap could just submit his patch a bunch of times and when it crashes, someone could grab the minidump for him, and he could debug it using the symbol server.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #10)
> If you can crash on the win32 try server talos, and we can configure it to save
> a minidump (which shouldn't be hard), we do in fact upload symbols for the
> Windows builds there.

mrbkap: did you see the symbols from windows runs? Are they enough?


(In reply to comment #11)
> There was supposed to be a new followup bug filed as a result of a meeting
> yesterday, but I don't think I've seen it go by. John, did I miss a bugmail?
> 
> IIRC the conclusion was that a mac mini configured with Talos would end up with
> my ldap credentials on it, so I could make the changes necessary to get a
> minidump and reproduce the crash.
Ah, ok. Sorry, I thought that was being filed by you, but I can do that if you
still need it after looking at windows (above)??

Comment 14

[alice nodelman [:alice] [:anode]]

Talos auto-kills upon noticing a browser crash.  The talos code is checked out per run.  To solve this we'd need to:

1.  patch the talos code not to kill a crashed browser and to wait for manual intervention
2.  patch try talos buildbot to use the patched talos code

Comment 15

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to comment #13)
> mrbkap: did you see the symbols from windows runs? Are they enough?

Which symbols were those? Any symbols from any talos enabled build would be fine.

> Ah, ok. Sorry, I thought that was being filed by you, but I can do that if you
> still need it after looking at windows (above)??

I asked if I should, but was told that since RelEng needs to give IT the name of the image to flash (or whatever the proper verb is there) that I wasn't the right person to file.

Comment 16

[Damon Sicore (:damons)]

(In reply to comment #14)
> Talos auto-kills upon noticing a browser crash.  The talos code is checked out
> per run.  To solve this we'd need to:
> 
> 1.  patch the talos code not to kill a crashed browser and to wait for manual
> intervention
> 2.  patch try talos buildbot to use the patched talos code

Alice, are you saying we need to do this in addition to what Ted says in Comment 12 ?

Comment 17

[Justin Fitzhugh]

Could we just get blake on the phone with alice, ben or ted to get this solved?  This is a high priority to get blake what he needs to debug - let's debug real time, not over bugmail.  Who's point on this from build and can you call blake?

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Summary of phone call:

- win32 symbol is a redherring ; dropped.
- cloning another mac mini would take too long (and possible colo trip); dropped.
- we're pulling an OSX mac mini from try-talo and changing the keys on it, so Blake can access without risk of contaminating graphserver or leaking passwords. This should be done before blake heads out for dinner.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #18)
> Summary of phone call:
> 
> - win32 symbol is a redherring ; dropped.
> - cloning another mac mini would take too long (and possible colo trip);
> dropped.
> - we're pulling an OSX mac mini from try-talo and changing the keys on it, so
> Blake can access without risk of contaminating graphserver or leaking
> passwords. This should be done before blake heads out for dinner.

Note: This means that try server has no talos coverage on OSX for 'n' days. Also, once mrbkap is finished experimenting, we'll have to reimage this machine before putting it back into use on try server.

Comment 20

[alice nodelman [:alice] [:anode]]

mrbkap was given access to qm-ptiger-try01.  It will be re-imaged once he's done with it.