1040365 - ceph processor failing ("this crash cannot be found in raw crash storage")

Status: RESOLVED FIXED

Description

[Robert Helmer [:rhelmer]]

Ceph crash storage is failing in processor with: 

2014-07-17 17:50:04,020 WARNING  - Thread-1 - 35deb152-04af-4b57-ba4f-8233a2140717 rejected: this crash cannot be found in raw crash storage

If I dig into this, I see the problem is when get_raw_crash()->do_get_raw_crash()->boto_s3_store._fetch_from_boto_s3(), if I put debugging in https://github.com/mozilla/socorro/blob/master/socorro/external/ceph/crashstorage.py#L360 like:

        self.logger.debug('%s %s %s' % (the_day_bucket_name, bucket, key))

The result is:

2014-07-17 17:54:51,418 DEBUG  - Thread-2 - 140717 <Bucket: 140717> e391429c-9be0-4927-a6f3-d25002140717.raw_crash
2014-07-17 17:54:51,933 DEBUG  - Thread-2 - 140717 <Bucket: 140717> e391429c-9be0-4927-a6f3-d25002140717.dump_names
2014-07-17 17:54:51,955 DEBUG  - Thread-2 - 140717 <Bucket: 140717> e391429c-9be0-4927-a6f3-d25002140717.upload_file_minidump

The last one is where it is failing. When I look in S3, I see that collector has stored using these filenames:

e391429c-9be0-4927-a6f3-d25002140717.raw_crash
e391429c-9be0-4927-a6f3-d25002140717.dump_names
e391429c-9be0-4927-a6f3-d25002140717.dump

If I look in the .dump_names file, I see:

["upload_file_minidump"]

So - there's a disagreement somewhere about what the dump file should be named.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

this is one of those the-name-changes-as-it-moves-through-the-system because of historical glitches.  see https://github.com/mozilla/socorro/blob/master/socorro/external/hb/crashstorage.py#L167 and https://github.com/mozilla/socorro/blob/master/socorro/external/hb/crashstorage.py#L367

BotoS3 needs the same translation logic that HBase is using...  PR on the way...

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/8be62766433e85b34fbc0a3b32abfed2c5907689
Merge pull request #2222 from twobraids/dump-name

Fixes Bug 1040365 - added dump name mangling & test