1090289 - Handle jobs where the revision is invalid, causing redundant fetch-missing-pushlog tasks

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

Once the analysis in bug 1085682 has been performed, we should blacklist the job types that consistently have bad revisions, so they do not continually hit the "oops we are missing the revision, let's try fetching it again" fallback added in bug 1077136.

This should reduce the load on the pushlog ingestion workers.

Comment 1

[Ed Morley [:emorley]]

Waiting on bug 1085682 and deps. Though from looking at the logs I couldn't see any of these in the short timeframe I searched through, so perhaps not as bad as thought.

Comment 2

[Ed Morley [:emorley]]

Some l10n jobs lie about their branch/revision combination - ie: they list a revision associated with a l10n repo, but in the branch property they put a gecko branch. This is tracked against buildbotcustom in bug 1125433.

That bug might not be fixed for a while, so in the meantime we should blacklist ingestion of those job types to avoid fruitless fetch-missing-pushlogs tasks.

Comment 3

[Ed Morley [:emorley]]

The only instance I'm seeing so far is the l10n jobs.

The real fix is to make builds-4hr actually contain the correct revision (bug 1125433), but that's waiting for mozharness stuff to land first, in case it fixes the issue. So for now let's just blacklist this case.

Comment 4

[Ed Morley [:emorley]]

Attachment: Skip jobs in builds-4hr that specify the wrong revision

I've run this through my analysis script locally on builds-2015-05-04.js and the only hits it found were the expected bad l10n jobs. 

l10n_revision is either not specified for a job, or if it is, then it's always referring to another repository, so should never match what we find for the main revision.

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/b7f1fc1ccdc5629af3107dd8758270cb3b0eeb70
Bug 1090289 - Skip jobs in builds-4hr that specify the wrong revision

Some l10n jobs specify the l10n repo revision under 'revision', rather
than the gecko revision. This results in fetch_missing_resultsets
requests that are guaranteed to 404. As such, these should be skipped
until this is fixed in builds-4hr by bug 1125433.

Comment 6

[Ed Morley [:emorley]]

Thank you for the reviews :-)

Comment 7

[Ed Morley [:emorley]]

This is done for builds-4hr, but they also use the wrong revision in builds-pending too, it would appear:

[2015-05-08 01:03:00,206: WARNING/Worker-302] Found pending jobs with missing resultsets.  Scheduling re-fetch: defaultdict(<type 'set'>, {'try': ['bfc7ed277b7b', 'Ubuntu VM 12', 'e666c7984fb4', '9fecaf59ae7c', 'd0cd1fd2cc02', '3f69978c27f5', '2dd8becba31e', 'b22c90e9bef2', '91352807d3de', 'c177153860a0', '4d9b888ffa28', '660b4103588f', 'fe2804d9f434', 'df80590de928', 'db0c55dec2a5', '8eb2da6af654', 'b3f8088e500c', 'de5da93b23b4', '02acc40f456c', 'f473e2edd616', '54d8d2af8e11', '4a7b1736f8bb', '75e7c3d0b279', 'e04b06dcfc1c', '9e14ebf35104']})

My favourite being the revision of 'Ubuntu VM 12', sigh.

Comment 8

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #7)
> My favourite being the revision of 'Ubuntu VM 12', sigh.

I'm going to file a releng bug for this, but let's make this bug handle protecting against this case too.

Comment 10

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/bc15ddb0dddf38ea229a7b4e2b106a2661cf9839
Bug 1090289 - Fix the skipping of l10n jobs with invalid revisions

There exist jobs that have invalid revision-branch combinations, where
the value given in `revision` is actually a SHA from an l10n repository,
rather than the gecko repository referenced by `branch`. A release
engineering bug (bug 1125433) has been filed but may never be fixed.

These invalid revisions are problematic, since they cause us to trigger
many spammy fetch-missing-pushlogs tasks that will never succeed.

Until now, we identified these jobs by comparing the value of
`l10n_revision` and `revision`, however `l10n_revision` is not available
in builds-{pending,running}.js, so a more complete fix is to blacklist
by buildername instead.

Running local analysis on builds-4hr found the invalid jobs had these
buildernames:

Firefox ash linux l10n dep
Firefox ash linux64 l10n dep
Firefox ash macosx64 l10n dep
Firefox ash win32 l10n dep
Firefox ash win64 l10n dep
Firefox mozilla-aurora linux l10n dep
Firefox mozilla-aurora linux64 l10n dep
Firefox mozilla-aurora macosx64 l10n dep
Firefox mozilla-aurora win32 l10n dep
Firefox mozilla-aurora win64 l10n dep
Firefox mozilla-central linux l10n dep
Firefox mozilla-central linux64 l10n dep
Firefox mozilla-central macosx64 l10n dep
Firefox mozilla-central win32 l10n dep
Firefox mozilla-central win64 l10n dep
Thunderbird comm-aurora linux l10n dep
Thunderbird comm-aurora linux64 l10n dep
Thunderbird comm-aurora macosx64 l10n dep
Thunderbird comm-aurora win32 l10n dep
Thunderbird comm-central linux l10n dep
Thunderbird comm-central linux64 l10n dep
Thunderbird comm-central macosx64 l10n dep
Thunderbird comm-central win32 l10n dep

I also confirmed that the `.endswith(' l10n dep')` did not result in
any false positives.