ceph processor failing ("this crash cannot be found in raw crash storage")

RESOLVED FIXED in 95

Status

RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: rhelmer, Unassigned)

Tracking

unspecified

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
Ceph crash storage is failing in processor with: 

2014-07-17 17:50:04,020 WARNING  - Thread-1 - 35deb152-04af-4b57-ba4f-8233a2140717 rejected: this crash cannot be found in raw crash storage

If I dig into this, I see the problem is when get_raw_crash()->do_get_raw_crash()->boto_s3_store._fetch_from_boto_s3(), if I put debugging in https://github.com/mozilla/socorro/blob/master/socorro/external/ceph/crashstorage.py#L360 like:

        self.logger.debug('%s %s %s' % (the_day_bucket_name, bucket, key))

The result is:

2014-07-17 17:54:51,418 DEBUG  - Thread-2 - 140717 <Bucket: 140717> e391429c-9be0-4927-a6f3-d25002140717.raw_crash
2014-07-17 17:54:51,933 DEBUG  - Thread-2 - 140717 <Bucket: 140717> e391429c-9be0-4927-a6f3-d25002140717.dump_names
2014-07-17 17:54:51,955 DEBUG  - Thread-2 - 140717 <Bucket: 140717> e391429c-9be0-4927-a6f3-d25002140717.upload_file_minidump

The last one is where it is failing. When I look in S3, I see that collector has stored using these filenames:

e391429c-9be0-4927-a6f3-d25002140717.raw_crash
e391429c-9be0-4927-a6f3-d25002140717.dump_names
e391429c-9be0-4927-a6f3-d25002140717.dump

If I look in the .dump_names file, I see:

["upload_file_minidump"]

So - there's a disagreement somewhere about what the dump file should be named.
(Reporter)

Updated

5 years ago
Blocks: 1008246
this is one of those the-name-changes-as-it-moves-through-the-system because of historical glitches.  see https://github.com/mozilla/socorro/blob/master/socorro/external/hb/crashstorage.py#L167 and https://github.com/mozilla/socorro/blob/master/socorro/external/hb/crashstorage.py#L367

BotoS3 needs the same translation logic that HBase is using...  PR on the way...

Comment 2

5 years ago
Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/8be62766433e85b34fbc0a3b32abfed2c5907689
Merge pull request #2222 from twobraids/dump-name

Fixes Bug 1040365 - added dump name mangling & test

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED

Updated

5 years ago
Target Milestone: --- → 95
You need to log in before you can comment on or make changes to this bug.