1074851 - Hidden jobs are briefly visible on initial pageload & after loading additional pushes

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

Jobs that are hidden by the exclusion profile are visible for several seconds on initial pageload - before they correctly disappear. In some cases they even seem to stay visible until you scroll down to them / mouse over them.

For example, open https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-central and look for the orange "JP" jobs on initial pageload.

Comment 1

[Ed Morley [:emorley]]

This also occurs when using the "load N more result sets" buttons.

Also, this affects the unclassified failure count as well as the display of jobs.

Comment 2

[Ed Morley [:emorley]]

Also another frustrating workflow:
1) Open treeherder
2) Press 'u'
3) Press the button at the bottom of the page to load additional result sets

-> The unclassified failure count top right of the page is wrong, since the hidden jobs haven't been excluded for the newly loaded result sets.

Comment 4

[Ed Morley [:emorley]]

Raising priority based on sheriffs top-picks of P2s.

Comment 5

[Cameron Dawson [:camd]]

Chatted with mdoglio today about this.  We think the best approach to speeding this up would be to do the filtering on the server prior to fetching the jobs.  This has many pros, and one con:

pros
1. reduce payload size of fetching resultset jobs (we don't transfer down hidden/excluded jobs)
2. we don't have to run filtering through the local exclusion profile for each job
3. we don't have to wait to download the flattened exclusion profile before we can filter (this is where the delay is coming from)

cons
1. switching between show/hide excluded jobs will be a bit slower (we'd have to re-fetch from the server)

The pros seem to GREATLY outweigh the cons because everybody does all of the things in the pros section, and the cons section is done less frequently.

Also: only other way would be to increase the speed of the exclusion profile fetch and that would only give us an incremental improvement, not really a fix.

Comment 6

[Ed Morley [:emorley]]

I like that approach (TBPL used it in fact, though in a more clunky manor and wasn't able to switch to fetching all jobs without a page reload, which was the main drawback).

Comment 7

[Cameron Dawson [:camd]]

Attachment: service PR

Comment 8

[Cameron Dawson [:camd]]

Attachment: ui PR

Comment 9

[Cameron Dawson [:camd]]

For this fix, I actually went with the "reload the page" approach.  This reduced the UI code complexity quite a bit and will fix a few other bugs with "unclassified count -1" and other areas where the counts got out of sync due to trying to handle the hidden ones.

This increases performance in several ways delineated in the service PR.

Now we have the url param of ``exclusion_state`` which can be one of:
included - don't show hidden jobs (default when not present)
excluded - ONLY show hidden jobs
all - show all jobs hidden and not.

clicking the toggle button will go between no param (included only) and "all", reloading the page on change.  The page will automatically reload if the user changes the query param.

Note: when this is deployed, we will need to go into the sheriff panel and re-save the exclusion profile so that the saved JSON will be in the correct format.

Comment 10

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/1e20c0be16f10c67b516e00489cbd79b32820e92
Bug 1074851 - save job exclusions as a flatlist of signatures

 - saves exclusions as a list of refdata signatures
 - the get_resultset_jobs endpoint honors the default exclusion profile via the signatures
 - group exclusion signatures by repo for performance

Also added handling for a profile with no job exclusions

https://github.com/mozilla/treeherder-service/commit/5f152bf3ee8812260d2b09ea812560ea6e11900d
Merge pull request #292 from mozilla/server-hidden-jobs

Bug 1074851 - Implement server side job filtering

Comment 11

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-ui

https://github.com/mozilla/treeherder-ui/commit/5383f0f4c3e7f8c0ba40ae20dfcad0080ba35ed9
Bug 1074851 - ``exclusion_state`` url param support

* controls whether to show ``included`` ``excluded`` or ``all`` jobs
based on the default exclusion profile.
* delay loading exclusions profiles to the UI unless you open the
sheriff panel
* no handling needed for counts wrt exclusions
* one-time binding of class for show/hide exclusion button
* show/hide exclusion changes url param, which reload page

https://github.com/mozilla/treeherder-ui/commit/287e182f62f0bb4aa135dd195cda80fbc47d00c7
Merge pull request #280 from mozilla/server-hidden-jobs

Bug 1074851 - Implement server side job filtering

Comment 12

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/9c277efe9560c49717fd55b5b29a4c9b765be4ba
Bug 1074851 - ``exclusion_state`` url param support

* controls whether to show ``included`` ``excluded`` or ``all`` jobs
based on the default exclusion profile.
* delay loading exclusions profiles to the UI unless you open the
sheriff panel
* no handling needed for counts wrt exclusions
* one-time binding of class for show/hide exclusion button
* show/hide exclusion changes url param, which reload page

https://github.com/mozilla/treeherder/commit/2262be1490f9a17ccb01e44ecc33b67f95d7b7a6
Merge pull request #280 from mozilla/server-hidden-jobs

Bug 1074851 - Implement server side job filtering