1706076 - TestESCrashStorage.test_crash_size_capture fails

Status: RESOLVED FIXED

(Socorro :: Processor, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

A couple of weeks ago, this test started failing out of the blue. The raw crash size had changed from 53 to 2. I looked into it for a bit and ended up "fixing" it with this commit:

https://github.com/mozilla-services/socorro/commit/e99ce3d79bc77502d8f5a5164c93b23ab1984bab

I wasn't wildly excited about the theory and couldn't prove it one way or the other.

The test started failing out of the blue again today. This time, it's up to 55 bytes:

__________________ TestESCrashStorage.test_crash_size_capture __________________
Traceback (most recent call last):
  File "/app/socorro/unittest/external/es/test_crashstorage.py", line 774, in test_crash_size_capture
    mm.assert_histogram("processor.es.raw_crash_size", value=2)
  File "/usr/local/lib/python3.9/site-packages/markus/testing.py", line 19, in _print_on_failure
    return fun(metricsmock, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/markus/testing.py", line 195, in assert_histogram
    assert (
AssertionError: assert 0 >= 1
 +  where 0 = len([])
 +    where [] = <bound method MetricsMock.filter_records of <markus.testing.MetricsMock object at 0x7f70b4ef2370>>('histogram', stat='processor.es.raw_crash_size', value=2, tags=None)
 +      where <bound method MetricsMock.filter_records of <markus.testing.MetricsMock object at 0x7f70b4ef2370>> = <markus.testing.MetricsMock object at 0x7f70b4ef2370>.filter_records
---------------------------- Captured stdout setup -----------------------------
setup: creating index: testsocorro_16
setup: creating index: testsocorro_15
----------------------------- Captured stdout call -----------------------------
<MetricsRecord type=histogram key=processor.es.raw_crash_size value=55 tags=[]>
<MetricsRecord type=histogram key=processor.es.processed_crash_size value=96 tags=[]>
<MetricsRecord type=histogram key=processor.es.crash_document_size value=239 tags=[]>
------------------------------ Captured log call -------------------------------
WARNING  elasticsearch:base.py:80 GET http://elasticsearch:9200/testsocorro_14/_mapping [status:404 request:0.018s]
--------------------------- Captured stdout teardown ---------------------------
teardown: delete test index: testsocorro_16
=================== 1 failed, 927 passed in 64.79s (0:01:04) ===================

Weird.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to look into today.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm fairly sure this is an order-of-operations problem. There's some ordering of happenstances wherein something is cached or something is indexed such that Elasticsearch has a mapping with ProductName and ReleaseChannel in it or doesn't have a mapping that doesn't have those in it. if that happens before this test, then this test will treat those as valid fields and the raw_crash_size value ends up as 55. If that doesn't happen before this test, then this test won't treat those as valid fields and the raw_crash_size will end up as 2.

I'm not positive because I haven't figured out the exact circumstances such that I can reproduce it. There's a lot of Elasticsearch test code and some of it is mocked, so it's tricky to track down.

Anyhow, the proper value should be 2 because those aren't valid fields. That's confused by the fact that they are in super_search_fields.py, but they're not used. There are a bunch of fields like this. I'll remove them.

Looking into this caused me to open up bug #1706171 because I found a problem that occurs because the PHC fields don't have storage_mapping set. Then looking into that helped me figure out another set of problems where the set of valid fields has multiple levels, but it's comparing against the top level of the raw/processed crash dicts. So some json_dump fields and some other fields are getting removed.

Thus in this bug:

1. I'll remove a bunch of dead fields in super_search_fields.py -- this includes ProductName and Releasechannel
2. I'll fix the problem where the raw/processed crash dicts are getting nested fields incorrectly removed

I think fixing both of those things should fix the underlying problem in this bug.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5752: bug 1706076, 1706171: fix indexing problems

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5752: "bug 1706076, 1706171: fix indexing problems" in d8fae15.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I pushed this to prod today in bug #1708188. Marking as FIXED.