1259494 - Non-sheriffs can't backfill jobs, with unhelpful error "Please login to Treeherder using Persona to complete this action"

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[David Burns :automatedtester]

STR:

1) Be logged into persona and buildapi
2) Visit https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab2d28ea54f9&selectedJob=18554682 (which is what I was using)
3) Click a the first green Mn job
4) Click the button to backfill
5) get error saying I am not logged in.

screenshot
https://www.evernote.com/shard/s63/sh/4b4042b2-7770-4bf7-ac8b-f295e8b9a1be/9b854a22b046346f

Comment 1

[Ed Morley [:emorley]]

Just double-checking this was a backfill request rather than a retrigger?
Also, was the email address of the Persona account a Moco email, or other domain? :-)

Comment 2

[David Burns :automatedtester]

I was on my moco email and I blicked backfill by accident and saw the error. I was meaning to do retriggers to track an intermittent and stumbled across this

Comment 3

[Ed Morley [:emorley]]

I'm not really sure what's going on here. Backfilling that job works for me at the moment. If you try again now, does it work?

Failing that, I'm not sure what else is going on here. Both Persona and backfilling are black boxes about which the Treeherder team know very little (we didn't write the backfilling code, even the Treeherder parts). Given Persona is going to be replaced soon, I think waiting for that, or else speaking to Armen about backfilling may be the next best step.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Any non-standard cookie settings? Maybe something about third party cookies not getting saved or passed along or something?

Comment 8

[Daniel Holbert [:dholbert]]

I'm running into this as well; nearly filed a dupe.

Comment 9

[Daniel Holbert [:dholbert]]

(In reply to Wes Kocher (:KWierso) from comment #4)
> Any non-standard cookie settings? Maybe something about third party cookies
> not getting saved or passed along or something?

Nope, that's not it -- I can reproduce this in a 100% fresh profile (with current Nightly).

I've logged into treeherder using the Persona login button in upper-right corner (with my @mozilla.com address).

"Retrigger job" works fine (and prompts me with an additional modal HTTP Auth dialog). But "Backfill job" spams this error message that incorrectly suggests that I'm not logged in.

Comment 10

[Daniel Holbert [:dholbert]]

(Seems to me this might be an account-permissions issue, given that this works for some folks but not others?  Maybe backfill requires a different set of permissions than retrigger, whether intentionally or unintentionally?)

Comment 11

[Daniel Holbert [:dholbert]]

I tried current Firefox release (version 46), as well (to be sure this wasn't simply failing due to some Nightly-only experimental setting). I'm getting this same bug there.

Comment 12

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Can you look in the Network Monitor im devtools and see what the various network requests end up looking like for retriggers and backfills?

Comment 13

[Ed Morley [:emorley]]

(In reply to Daniel Holbert [:dholbert] from comment #10)
> (Seems to me this might be an account-permissions issue, given that this
> works for some folks but not others?  Maybe backfill requires a different
> set of permissions than retrigger, whether intentionally or unintentionally?)

Ah you are correct - both backfill and trigger missing jobs are set to 'is_staff' users only (ie sheriff; though we should use a separate group in the future, likely as part of bug 1273092). Looks like this was intentional - see bug 1183923 comment 0 and bug 1183923 comment 1.

Sorry I wasn't aware of this - both the backfill and trigger missing jobs features were not written by the treeherder team, so we're not as familiar with them. I thought the handling of permissions was purely in pulse_actions (a tool external to Treeherder) via a whitelisting of @mozilla.com email addresses (with some contributors added).

Armen, can this restriction now be lifted, now that pulse_actions handles whitelisting?

https://github.com/mozilla/treeherder/blame/c4c781221269567bc921523ef5c809c1c347ce71/treeherder/webapp/api/jobs.py#L180
https://github.com/mozilla/treeherder/blame/c4c781221269567bc921523ef5c809c1c347ce71/treeherder/webapp/api/resultset.py#L151

What makes this more confusing is that for some reason we use the same error string for both 401s and 403s:
https://github.com/mozilla/treeherder/blob/2a9dbefa49d26d8d0ddc7c402c571691b1679770/ui/js/models/error.js#L28-L29

However that said, the API returns 403 even when not logged in at all, eg:

[~/src/treeherder]$ curl -Is -X POST "https://treeherder.mozilla.org/api/project/try/resultset/106632/trigger_missing_job s/" | grep HTTP
HTTP/1.1 403 FORBIDDEN

This is due to django-browserid not setting a WWW-Authenticate, which the spec says in mandatory if HTTP 401 is to be used:
http://www.django-rest-framework.org/api-guide/authentication/#unauthorized-and-forbidden-responses

Regardless there are a few things we can/should do to make that error clearer, as well as adjusting the required permissions for these endpoints.

Comment 14

[Armen [:armenzg]]

Note that I've actually removed the whitelisting from pulse_actions and I get alerted for non-mozilla addresses which I review. So far, no one suspicious has been using the feature. All emails were recognizable to someone at Mozilla.

Which makes me wonder how am I seeing those requests come in if there are restrictions on Treeherder.
Are there non-moco email addresses marked as "isStaff" on Treeherder? (I assume this could be the answer to my confusion).

Comment 15

[Ed Morley [:emorley]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #14)
> Which makes me wonder how am I seeing those requests come in if there are
> restrictions on Treeherder.
> Are there non-moco email addresses marked as "isStaff" on Treeherder? (I
> assume this could be the answer to my confusion).

Yes, any sheriff (or person that has asked for access to the sheriffing panel) that's either not an employee or is an employee using a non-moco email.

Comment 16

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:remove-backfill-restrictions > mozilla:master

Comment 17

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/17eb4bc74cd4851b52bf12e087ef8d624dbb028c
Bug 1259494 - Allow non-sheriffs to backfill & retrigger missing jobs

For now permission decisions should be made in pulse_actions. In the
future Treeherder will use Taskcluster scopes (bug 1273092) or
accessTokens (bug 1273096).