1749727 - [traceback] RuntimeErrorsocorro.external.es.super_search_fields in has_key_val

Status: RESOLVED FIXED

(Socorro :: Processor, defect, P3)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Sentry: https://sentry.prod.mozaws.net/operations/socorro-new-prod/issues/16921239/

Traceback:

RuntimeError: dictionary changed size during iteration
  File "socorro/external/crashstorage_base.py", line 478, in save_processed_crash
    a_store.save_processed_crash(raw_crash, processed_crash)
  File "socorro/external/crashstorage_base.py", line 639, in benchmarker
    return wrapped_attr(*args, **kwargs)
  File "socorro/external/es/crashstorage.py", line 401, in save_processed_crash
    self.prepare_crash_data(raw_crash, processed_crash)
  File "socorro/external/es/crashstorage.py", line 640, in prepare_crash_data
    super().prepare_crash_data(raw_crash, processed_crash)
  File "socorro/external/es/crashstorage.py", line 594, in prepare_crash_data
    super().prepare_crash_data(raw_crash, processed_crash)
  File "socorro/external/es/crashstorage.py", line 356, in prepare_crash_data
    truncate_keyword_field_values(
  File "socorro/external/es/crashstorage.py", line 127, in truncate_keyword_field_values
    keyword_fields = get_fields_by_item(fields, "analyzer", "keyword")
  File "socorro/external/es/super_search_fields.py", line 431, in get_fields_by_item
    fields = [field for field in fields.values() if has_key_val(key, val, field)]
  File "socorro/external/es/super_search_fields.py", line 431, in <listcomp>
    fields = [field for field in fields.values() if has_key_val(key, val, field)]
  File "socorro/external/es/super_search_fields.py", line 423, in has_key_val
    ret = has_key_val(key, val, this_val)
  File "socorro/external/es/super_search_fields.py", line 423, in has_key_val
    ret = has_key_val(key, val, this_val)
  File "socorro/external/es/super_search_fields.py", line 423, in has_key_val
    ret = has_key_val(key, val, this_val)
  File "socorro/external/es/super_search_fields.py", line 421, in has_key_val
    for this_key, this_val in data.items():

This should get fixed. It's getting kicked up after me adding DumperError, so maybe it's only a problem after adding a new field that is in crash reports, but isn't in an index, yet.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5974: bug 1749727: rename variables with same name but different types

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5974: "bug 1749727: rename variables with same name but different types" in e7e410a.

This error is weird. The Sentry error report is missing information in the frames. When Sentry truncates something, it shows a "...", but in this case, it's not doing that--the information is just missing. That's weird.

I tried to reproduce the error locally and in prod with the crash report that was in the Sentry error report, but couldn't. I read through the code and I'm not sure why the dict would be changing--this particular method traverses a structure and returns a smaller version of it, but it doesn't edit the structure.

However, the variable names kind of sucked, so I fixed one of them so that we don't have a variable name that's bound to two different things at different points with different types.

I'm going to make this a P3 and keep it around for a bit and if it recurs, we can look at things then.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I haven't seen any new instances of this in Sentry, so I'm going to mark it FIXED.