1144865 - debug id for the symbol is missing from the text file uploaded w/ the symbol to Soccoro

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

rhelmer investigated why https://crash-stats.mozilla.com/report/index/9ced99ad-cfba-480b-923c-3eee92150318 wasn't showing the crash report associated with the symbols to a build.

He found that the debug id for the symbol is missing from the text file uploaded w/ the symbol to Soccoro

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Note: after he looked into it , he resolved the situation for this particular crash ( and any crash with from this build ).

We ask that the text file be updated correctly for future builds.

Comment 2

[Robert Helmer [:rhelmer]]

The reason this is important is because we're syncing from the old NFS store to the new S3 bucket (until the releng systems start uploading directly via crash-stats symbol upload), I have since sync'd and reprocessed the crash in comment 0.

The txt index file uploaded to the symbol store with this build doesn't seem to include the right libxul.so symbol:

$ grep AAE3B4B03234E504A7535FB53B4DE2410 b2g-39.0a1-Android-20150318055750-arm-symbols.txt

It does include libxul.so for a different debug id, which is odd since this crash doesn't seem to use this (according to "Modules" tab in crash-stats at least):

$ grep libxul b2g-39.0a1-Android-20150318055750-arm-symbols.txt
libxul.so/5679187F7D8D71161F8AC02FAFAD82B70/libxul.so.sym

Comment 3

[Robert Helmer [:rhelmer]]

We just hit this again (bug 1145823).

Will bug 1085557 make B2G also upload to the new crash-stats symbol upload instead of the old symbol server? If so, we can probably WONTFIX this one since we won't depend on the text files anymore (the symbol upload service tracks contents as it unzips)

Comment 4

[Robert Helmer [:rhelmer]]

So far this only seems to be happening with libxul... I can have the sync script keep this whole directory up to date, that's just too expensive to run for all products and symbols (which is why the sync script depends on the txt index files being right)

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> So far this only seems to be happening with libxul... I can have the sync
> script keep this whole directory up to date, that's just too expensive to
> run for all products and symbols (which is why the sync script depends on
> the txt index files being right)

I put this workaround in place for the time being:
https://github.com/rhelmer/sync-symbols-s3/commit/fac893d5ca9219364fb7eea02909072c337a5998

Comment 6

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Would the work around help with the eng builds as well as the production builds or is this only affecting the production builds?

We (QA) would like to request that the nightly flame-kk eng builds (MC/2.2) also is associated symbols.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #6)
> Would the work around help with the eng builds as well as the production
> builds or is this only affecting the production builds?
> 
> We (QA) would like to request that the nightly flame-kk eng builds (MC/2.2)
> also is associated symbols.

Only production build symbols are being uploaded by the build automation at this time.

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I guess then that's a separate bug.  Bug 1146466 filed.  Thanks rhelmer.

Comment 9

[bhavana bajaj [:bajaj]]

thanks for helping with this naoki.

Comment 10

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Hey James and Cat, I'm not sure who to contact about socorro issues any more or the build infrastructure.
In any cases, we aren't showing the appropriate crash stacks, could we get someone to look into this please?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

I talked at length with rhelmer on IRC, and we diagnosed the problem that there are 8 builds which upload with the same manifest name (4 devices - dolphin, flame, nexus-4, nexus-5-l; and two types of builds - opt & eng). So relying on the manifest is not safe.

rhelmer tells me that once we are using S3 as the file store that this problem goes away (and of course it's mitigated by the syncing at the moment). Bug 1085557 is enabling upload to S3 in production and is ready to land.

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I just purposely crashed FMD and got a crash id and the symbols were applied.  Closing this bug out as fixed.