Correlate tests and bug components to Orange Factor frequent intermittent test failures

NEW
Unassigned

Status

Testing
General
9 months ago
9 months ago

People

(Reporter: armenzg, Unassigned)

Tracking

(Depends on: 1 bug)

Version 3
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

9 months ago
On bug 1352495 I can generate a data structure that associates test files to bug components.

In this bug I would like to take that data and correlate it to OF data.
(Reporter)

Comment 1

9 months ago
I will be looking to match relative test paths to bug summaries.
and we should ensure that all bug summaries that do not match have a represented bucket- ideally we can fix up the bug summaries to have proper names- that seems reasonable to expect.
(Reporter)

Comment 3

9 months ago
Created attachment 8857193 [details] [diff] [review]
Mach subcommand to show summary of bug components with intermittent test failures

NOTE: The subcommand from bug 1352495 cannot handle all test files in the tree. There's an issue on the build system that makes it to take too long once we start passing 15k test files.

jmaher: Could you please have a look at the output below to see if I'm grasping what we want? We can jump on Vidyo to quickly go over your feedback
        Would you like this subcommand to not be separate from the other one?

gbrown: Would you mind looking at the code I have? It is WIP. I'm mainly interested if I'm interacting with OF correctly.


armenzg-loaner:firefox armenzg$ ./mach file-info intermittent-orange-per-component
The following components have intermittent oranges (excluding resolved bugs) in the last 21 days with more than 100 occurrences

	Firefox::Tours -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1352043
	Core::WebRTC -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1351590
	Core::DOM -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1352923&bug_id=1313972&bug_id=1350875&bug_id=1332040
	Firefox::Session Restore -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1348583

The following bugs don't match to any test file:
	https://bugzilla.mozilla.org/buglist.cgi?bug_id=1303834&bug_id=1329836&bug_id=1310638&bug_id=1354681&bug_id=1276813&bug_id=1348390&bug_id=1348241&bug_id=1308343&bug_id=1345724&bug_id=1345723&bug_id=1345735&bug_id=1052467&bug_id=1337578&bug_id=1261598&bug_id=1336760&bug_id=1351409&bug_id=1338638&bug_id=1334336&bug_id=1336775

Comment 4

9 months ago
Comment on attachment 8857193 [details] [diff] [review]
Mach subcommand to show summary of bug components with intermittent test failures

I'd really recommend avoiding urllib2 and using requests instead (believe it's already in the tree).

Also, don't forget to set a User Agent and to check for status==200 :-)

(Side node: perhaps mozilla-central/mach needs a common fetch_json() helper? Like: https://github.com/mozilla/treeherder/blob/39e5e7c4ea3adae6aec3a39973113d35aef218f0/treeherder/etl/common.py#L31-L57)
overall this looks in the right direction.  I like the list of bugs at the bottom that don't match anything :)

I looked at a few things which should match test files, edited some titles, this bug should match though:
https://bugzilla.mozilla.org/show_bug.cgi?id=1348241

a few other misses are related to web-platform-tests, or tests which have a secondary manifest and run in a different context:
https://bugzilla.mozilla.org/show_bug.cgi?id=1354681

issues like:
https://bugzilla.mozilla.org/show_bug.cgi?id=1338638
https://bugzilla.mozilla.org/show_bug.cgi?id=1303834

these have a long summary and trying to put the full path in fails- we could figure out now to make the summary have the full path, or maybe there are other methods for matching?  maybe a partial path?

Overall, I think outputting a related .json file as we do for the test mappings would be ideal.  Then we have json data for per component of # tests, and high priority intermittent failures.

Given these two data points, we should be able to build a simple dashboard to display the current data- if we put this in a database, then we could see trends over time and show relative deltas if we wish, but for now and getting something useful- what we have is a great start and proper data.


as a note, I was not able to add this patch locally to my system, I assume this isn't a proper diff?
(Reporter)

Comment 6

9 months ago
The patch is based on the one from bug 1352495.
(Reporter)

Comment 7

9 months ago
With regards to bugs not matching properly remember that due to performance issues (bug 1355630) I was working based on a small subset of the whole tree. Also note that I'm ignoring web platform tests (bug 1334525).

Let's think of other matching ideas after the performance issue is fixed.

emorley: what is the User Agent for? I'm not interacting with TH.
(Reporter)

Comment 8

9 months ago
I rebased my changes over gps'. This is the latest output [1]

I believe the test matching can be better for the following (ignoring web platform):
* bug 1338638 - Intermittent test_safe_browsing_initial_download.py
* bug 1303834 - [tier-3] Intermittent test_fallback_update.py
* bug 1354681 - Intermittent browser/components/extensions/test/browser/test-oop-extensions/browser_ext_omnibox.js

###############
[1]
Fetching https://brasstacks.mozilla.com/orangefactor/api/bybug?startday=2017-03-22&endday=2017-04-12&tree=trunk
The following components have intermittent oranges (excluding resolved bugs) in the last 21 days with more than 100 occurrences

	Core::WebRTC -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1351531&bug_id=1351590
	UNKNOWN -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1308343
	Core::DOM -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1352923&bug_id=1313972&bug_id=1350875&bug_id=1282207&bug_id=1284733&bug_id=1332040
	Firefox::Tours -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1352043
	Toolkit::Password Manager -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1334336
	Toolkit::WebExtensions: General -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1348241
	Firefox::Session Restore -> https://bugzilla.mozilla.org/buglist.cgi?bug_id=1348583

The following bugs don't match to any test file:
	https://bugzilla.mozilla.org/buglist.cgi?bug_id=1303834&bug_id=1310638&bug_id=1354681&bug_id=1276813&bug_id=1348390&bug_id=1345724&bug_id=1345723&bug_id=1345735&bug_id=1052467&bug_id=1204281&bug_id=1337578&bug_id=1261598&bug_id=1336760&bug_id=1351409&bug_id=1338638&bug_id=1336775

Comment 9

9 months ago
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #3)
> gbrown: Would you mind looking at the code I have? It is WIP. I'm mainly
> interested if I'm interacting with OF correctly.

Sorry, I almost forgot about this. Don't hesitate to ni me!

Your OF interaction looks fine to me. Listen to :emorley -- he has more experience with it than I do.


I applied your patch on top of 1352495 but couldn't get anything useful out of it:

$ ./mach file-info intermittent-orange-per-component
The following components have intermittent oranges (excluding resolved bugs) in the last 21 days with more than 100 occurrences


I don't know why. :(


I know this is a wip and I'm not sure of your plans, but I'll just mention, I would prefer a different command interface:

$ ./mach test-bugs-for-component <component> [--threshold <threshold>] [--days <days>]

eg.

./mach test-bugs-for-component Core::DOM --days 21 --threshold 100

https://bugzilla.mozilla.org/buglist.cgi?bug_id=1352923&bug_id=1313972&bug_id=1350875&bug_id=1332040
(Reporter)

Comment 10

9 months ago
I'm going to refocus on some Quantum related work. I won't be able to look at this for a while.
Assignee: armenzg → nobody
You need to log in before you can comment on or make changes to this bug.