TestESCrashStorage.test_crash_size_capture fails
Categories
(Socorro :: Processor, defect, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
Details
Attachments
(1 file)
A couple of weeks ago, this test started failing out of the blue. The raw crash size had changed from 53 to 2. I looked into it for a bit and ended up "fixing" it with this commit:
https://github.com/mozilla-services/socorro/commit/e99ce3d79bc77502d8f5a5164c93b23ab1984bab
I wasn't wildly excited about the theory and couldn't prove it one way or the other.
The test started failing out of the blue again today. This time, it's up to 55 bytes:
__________________ TestESCrashStorage.test_crash_size_capture __________________
Traceback (most recent call last):
File "/app/socorro/unittest/external/es/test_crashstorage.py", line 774, in test_crash_size_capture
mm.assert_histogram("processor.es.raw_crash_size", value=2)
File "/usr/local/lib/python3.9/site-packages/markus/testing.py", line 19, in _print_on_failure
return fun(metricsmock, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/markus/testing.py", line 195, in assert_histogram
assert (
AssertionError: assert 0 >= 1
+ where 0 = len([])
+ where [] = <bound method MetricsMock.filter_records of <markus.testing.MetricsMock object at 0x7f70b4ef2370>>('histogram', stat='processor.es.raw_crash_size', value=2, tags=None)
+ where <bound method MetricsMock.filter_records of <markus.testing.MetricsMock object at 0x7f70b4ef2370>> = <markus.testing.MetricsMock object at 0x7f70b4ef2370>.filter_records
---------------------------- Captured stdout setup -----------------------------
setup: creating index: testsocorro_16
setup: creating index: testsocorro_15
----------------------------- Captured stdout call -----------------------------
<MetricsRecord type=histogram key=processor.es.raw_crash_size value=55 tags=[]>
<MetricsRecord type=histogram key=processor.es.processed_crash_size value=96 tags=[]>
<MetricsRecord type=histogram key=processor.es.crash_document_size value=239 tags=[]>
------------------------------ Captured log call -------------------------------
WARNING elasticsearch:base.py:80 GET http://elasticsearch:9200/testsocorro_14/_mapping [status:404 request:0.018s]
--------------------------- Captured stdout teardown ---------------------------
teardown: delete test index: testsocorro_16
=================== 1 failed, 927 passed in 64.79s (0:01:04) ===================
Weird.
Assignee | ||
Comment 1•3 years ago
|
||
Grabbing this to look into today.
Assignee | ||
Comment 2•3 years ago
|
||
I'm fairly sure this is an order-of-operations problem. There's some ordering of happenstances wherein something is cached or something is indexed such that Elasticsearch has a mapping with ProductName
and ReleaseChannel
in it or doesn't have a mapping that doesn't have those in it. if that happens before this test, then this test will treat those as valid fields and the raw_crash_size
value ends up as 55. If that doesn't happen before this test, then this test won't treat those as valid fields and the raw_crash_size
will end up as 2.
I'm not positive because I haven't figured out the exact circumstances such that I can reproduce it. There's a lot of Elasticsearch test code and some of it is mocked, so it's tricky to track down.
Anyhow, the proper value should be 2 because those aren't valid fields. That's confused by the fact that they are in super_search_fields.py
, but they're not used. There are a bunch of fields like this. I'll remove them.
Looking into this caused me to open up bug #1706171 because I found a problem that occurs because the PHC fields don't have storage_mapping
set. Then looking into that helped me figure out another set of problems where the set of valid fields has multiple levels, but it's comparing against the top level of the raw/processed crash dicts. So some json_dump
fields and some other fields are getting removed.
Thus in this bug:
- I'll remove a bunch of dead fields in
super_search_fields.py
-- this includesProductName
andReleasechannel
- I'll fix the problem where the raw/processed crash dicts are getting nested fields incorrectly removed
I think fixing both of those things should fix the underlying problem in this bug.
Assignee | ||
Comment 3•3 years ago
|
||
Assignee | ||
Comment 4•3 years ago
|
||
Assignee | ||
Comment 5•3 years ago
|
||
I pushed this to prod today in bug #1708188. Marking as FIXED.
Description
•