We recently switched to a new CrashStorage class for Elasticsearch, which allows only a subset of the json_dump to be stored. That CrashStorage is destructive of the processed_crash, altering the json_dump, with the effect that all following stores will receive the modified version. On production, it works fine, because Elasticsearch is the last of the stores in our list, but that is not the case in stage: S3 comes right after ES. So right now, in S3, we store a very limited json_dump. The solution to this is simple: changing the configuration so S3 comes before ES in the list of stores.
Proposed change: # Cosmetic change to the list of classes (only the number of classes matters). consulate kv set socorro/processor/destination.storage_classes 'socorro.external.postgresql.crashstorage.PostgreSQLCrashStorage, socorro.external.boto.crashstorage.BotoS3CrashStorage, socorro.external.es.crashstorage.ESCrashStorageRedactedJsonDump, socorro.external.statsd.crashstorage.StatsdCrashStorage' # Remove a bunch of useless / legacy keys. consulate kv rm socorro/processor/destination.storage1.elasticsearch.elasticsearch_urls consulate kv rm socorro/processor/destination.storage1.use_mapping_file consulate kv rm socorro/processor/destination.storage0.wrapped_crashstore consulate kv rm socorro/processor/destination.storage1.wrapped_crashstore consulate kv rm socorro/processor/destination.storage2.wrapped_crashstore # Switch the crash storage classes. consulate kv set socorro/processor/destination.storage1.wrapped_object_class 'socorro.external.boto.crashstorage.BotoS3CrashStorage' consulate kv set socorro/processor/destination.storage2.wrapped_object_class 'socorro.external.es.crashstorage.ESCrashStorageRedactedJsonDump' # Switch the statsd prefixes. consulate kv set socorro/processor/destination.storage1.statsd_prefix 'processor.s3' consulate kv set socorro/processor/destination.storage2.statsd_prefix 'processor.es' # Move the benchmark tag. consulate kv set socorro/processor/destination.storage1.benchmark_tag 'BotoBenchmarkWrite' consulate kv rm socorro/processor/destination.storage2.benchmark_tag
So these are ALL on stage, ONLY? Clearly we need this. So go ahead.
Applied to stage, seems to be working fine!
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.