1972307 - intermittent failures Bugzilla commenter hangs or fails with ERROR: 137

Status: RESOLVED FIXED

(Tree Management :: Bugherder, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The calculation of the total runs of a test causes the intermittent failures Bugzilla commenter script to hang in GCP (no comments, worker keeps running) or failure locally with error code 137 (docker container failure, often caused by high memory consumption).

Without a fix today, this would have to be rolled back so the script for the weekly update can run.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

trying to figure out the problem here, I am going under the assumption that we are taking too long to run.

There are really a few parts:

1. collect the artifact and load into memory (expensive, but appears to be done once)
2. find the manifest and related label in the data from #1
3. repeat for #1 and #2 for each bug

I think we can optimize the data in get_task_labels_and_count, specifically we pass in the manifest and task label to match on and instead of building a structure with all the data for the manifest, just collect the data for the manifest manifest AND label- less looping.

another area to optimize is that we only call the script with either 1 day or 7 days, so we can always assume the same number of days and can do the summarizing of data by day ONCE and not have to loop through each day to get data. for example, if this is 1 day, we find the data, build the data structure and store it in self.summary_groups, then we can reference it faster, otherwise we squash the data from 7 days -> 1 data structure.

another thing I want to make sure is happening is that the runcount variable is data for the specific platform and variant, not just the first item.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

new changes have been landed- currently the weekly mode is still taking too long and potentially not commenting on all bugs. Still more work to make this faster and accurate.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I believe this is working properly now.