1227780 - MainThread ERROR 401 Client Error: Unauthorized for url when using mozregression tool

Status: VERIFIED FIXED

(Testing :: mozregression, defect)

Description

[KTucker [:KTucker][Inactive 3/4/2016]]

When using the mozregression tool, the user keeps being prompted ERROR 401 Client Error: Unauthorized for url

1. Input this command into the terminal --app b2g-flame --bad-rev c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4 --good-rev b93dd434b3cdc248c114f9289c37b2fb92fb6229 and observe. 

Using revs: b93dd434b3cdc248c114f9289c37b2fb92fb6229 - c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4
 0:00.22 LOG: MainThread main INFO Getting inbound builds between b93dd434b3cdc248c114f9289c37b2fb92fb6229 and c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4
 0:03.70 LOG: MainThread Bisector INFO Pushlog:
https://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=b93dd434b3cdc248c114f9289c37b2fb92fb6229&tochange=c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4

 0:03.71 LOG: MainThread download INFO Downloading build from: https://queue.taskcluster.net/v1/task/Pbu4e9JOQkSF84JPNfQPBA/runs/0/artifacts/private%2Fbuild%2Fflame-kk.zip?bewit=dGMtbG9naW5cMTQ0ODQxMTg0MVxFekJGUzljbDRNcmdKMWNMOGgrcXhzM3duVXVuU1JCdXcyS2lQdzJ1M3lFPVxlMzA9
===== Downloaded 100% =====
 0:04.15 LOG: MainThread ERROR 401 Client Error: Unauthorized for url: https://queue.taskcluster.net/v1/task/Pbu4e9JOQkSF84JPNfQPBA/runs/0/artifacts/private%2Fbuild%2Fflame-kk.zip?bewit=dGMtbG9naW5cMTQ0ODQxMTg0MVxFekJGUzljbDRNcmdKMWNMOGgrcXhzM3duVXVuU1JCdXcyS2lQdzJ1M3lFPVxlMzA9
 0:04.15 LOG: Dummy-5 Bisector INFO Last good revision: b93dd434b3cdc248c114f9289c37b2fb92fb6229
 0:04.15 LOG: Dummy-5 Bisector INFO First bad revision: c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4
 0:04.15 LOG: Dummy-5 Bisector INFO Pushlog:
https://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=b93dd434b3cdc248c114f9289c37b2fb92fb6229&tochange=c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4

 0:04.15 LOG: Dummy-5 main INFO To resume, run:
 0:04.15 LOG: Dummy-5 main INFO /usr/local/bin/mozregression --app b2g-flame --good-rev=b93dd434b3cdc248c114f9289c37b2fb92fb6229 --bad-rev=c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4
flash@FlashStation53:~$

Comment 1

[KTucker [:KTucker][Inactive 3/4/2016]]

Can you take a look at this Julien?

Comment 2

[Julien Pagès (:parkouss)]

Well, I believe you defined "taskcluster-clientid" and "taskcluster-accesstoken" in ~/.mozilla/mozregression/mozregression.cfg with good and double checked values.

I remember :mshal tried the download of private artifacts for me as I don't have valid credentials (being a contributor). :mshal, could you try with the command above please ?


I am not sure I will be able to help much given that I won't be able to test, but I'll do my best.

Comment 3

[Michael Shal [:mshal]]

I ran it and it starts to download https://queue.taskcluster.net/v1/task/Pbu4e9JOQkSF84JPNfQPBA/runs/0/artifacts/private/build/flame-kk.zip for me, so it looks like it's working. :KTucker, are you able to download private artifacts manually with your credentials? Eg: go to https://tools.taskcluster.net/task-inspector/#Pbu4e9JOQkSF84JPNfQPBA/0 and click on private/build/flame-kk.zip

You might need to poke someone in #taskcluster for further help here if it's just a credentials issue.

Comment 4

[KTucker [:KTucker][Inactive 3/4/2016]]

Thanks for the reply. Yes, i can download the build just fine from taskcluster when going here https://tools.taskcluster.net/task-inspector/#Pbu4e9JOQkSF84JPNfQPBA/0  but some reason hit an issue when going to https://queue.taskcluster.net/v1/task/Pbu4e9JOQkSF84JPNfQPBA/runs/0/artifacts/private/build/flame-kk.zip 

  "message": "Authorization Failed",
  "error": {
    "info": "None of the scope-sets was satisfied",
    "scopesets": [
      [
        "queue:get-artifact:private/build/flame-kk.zip"
      ]
    ],
    "scopes": []
  }
}

Comment 5

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Micheal, Julien, I'll check with Kevin tomorrow in regards to the configuration file.  The build seems to have downloaded fine for me as well.  

Side note : the part that didn't happen was the flashing, though I'm not sure if that's in scope yet or not for this current release.  (ie unzip the zip file downloaded and run flash.sh script that's inside the b2g-distro folder once unzipped )  We might want to have that as a option of autoflashing or something?  That's a different bug/request than what's covered in this bug.

Comment 6

[Julien Pagès (:parkouss)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #5)
> Side note : the part that didn't happen was the flashing, though I'm not
> sure if that's in scope yet or not for this current release.  (ie unzip the
> zip file downloaded and run flash.sh script that's inside the b2g-distro
> folder once unzipped )  We might want to have that as a option of
> autoflashing or something?  That's a different bug/request than what's
> covered in this bug.

Yes, that's not implemented yet - and that's something needed yes! I will look at it in a few days, since it sound easy (unzipping/running a shell file), and I would be happy to meet you in Orlando and talk about that. :)

Comment 7

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Side note : OH SWEET!  you're going to orlando!  We're giving a regression session and we will be show casing our regression range process in finding the right builds currently and then the use of the tool.  We would love to have you there.

Wednesday, December 9 : 1:15  PM - Finding Regression Ranges - Moz Regression Tool    Northern A2

Comment 8

[Julien Pagès (:parkouss)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #7)
> Side note : OH SWEET!  you're going to orlando!  We're giving a regression
> session and we will be show casing our regression range process in finding
> the right builds currently and then the use of the tool.  We would love to
> have you there.
> 
> Wednesday, December 9 : 1:15  PM - Finding Regression Ranges - Moz
> Regression Tool    Northern A2

Yes, of course, that sounds great! Just added it to sched. Thanks!

Comment 9

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I spoke with ktucker and we seemed to have the mozregression config working and the commandline he used works for me.

Which means that it's probably the scope that's too limiting for his account.  I asked him to speak with Jonasfj in regards to this issue so that we can get him in a wider scope to download the build from this tool.

Comment 10

[Jonas Finnemann Jensen (:jonasfj)]

ah, he isn't moco.

moco has: queue:get-artifact:private/*
but https://tools.taskcluster.net/auth/roles/#mozillians-group:b2g-qa-internal
only has: queue:get-artifact:private/build/*

I'll grant queue:get-artifact:private/*

@ktucker, please confirm that it works.

Comment 11

[KTucker [:KTucker][Inactive 3/4/2016]]

Thanks Jonas but unfortunately I am still getting the same error. I logged out of Taskcluster and logged back in and changed the credentials in the config file but to no avail.

Comment 12

[KTucker [:KTucker][Inactive 3/4/2016]]

So Dustin thinks we are getting the error because the certificate is not in our config file. What is the correct syntax for that?

Comment 13

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

nhirata_: dustin: fyi, I don’t have a certificate in my config either and mine works.  is it because I’m in a different scope?
[2:38pm] nhirata_: there’s a few other people that also don’t have the cert in their config.
[2:38pm] hwine|mtg is now known as hwine.
[2:39pm] dustin: you and those people are probably using old permacreds
[2:39pm] dustin: when you get creds from mozillians, they're temporary, and thus require a cert

Talking to dustin we need to make sure that certificate pushing is possible on taskcluster.

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

https://github.com/mozilla/mozregression/search?utf8=%E2%9C%93&q=taskcluster

Looks like it's not in the tool yet.  Julien, fyi.  It turns out that we need to be able to upload certs as well.

Comment 15

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

jonasfj: just use the login flow for nozregression
[3:08pm] jonasfj: ie. see taskcluster-cli login
[3:08pm] jonasfj: nhirata_: https://github.com/taskcluster/taskcluster-cli/blob/master/bin/taskcluster-login.js
[3:10pm] jonasfj: that way a browser open... you login, click grant access.. browser window closes... and you're done... in nozregression. you need to run a server on localhost but that's easy one is already built into python... all you are looking for is querystring

Comment 16

[Julien Pagès (:parkouss)]

Hum, I don't understand what is needed here.

mozregression uses buildSignedUrl from taskcluster-client.py to get a url of a private resource, then it just download this url.

https://github.com/taskcluster/taskcluster-client.py/blob/master/taskcluster/client.py#L153

I don't understand what we need to add in mozregression to make it work (certificates, local server ??) and I have the feeling that most of whatever code is required for that should go into https://github.com/taskcluster/taskcluster-client.py.

:jonasfj, can we use taskcluster-python.py (and if so, how) to let :ktucker download private artifacts ? Or how should we do ?

Comment 17

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

As far as I understand, we need to pass a certificate.  Jonas had irced me the following : 

jonasfj: the login flow discussion from yesterday: https://github.com/taskcluster/taskcluster-client.py/pull/33 it's a bit crazy, not sure if it's a viable option for mozregression, perhaps a good fallback if no creds are given... (anyways, have a look when you get up)


Personally I think the temp credentials is causing more of an issue in work flow/work flow performance for the end user.  I think we really need to have UX improve the UX on taskcluster in general for login and UI design...

Comment 18

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Based on AuthCallBackRequestHandler, looks like we need certificate?

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Stepping back to looking at what Julien is saying : 
https://github.com/taskcluster/taskcluster-client.py/blob/master/taskcluster/client.py#L174

Don't we need to account for cert in the buildSignedUrl from taskcluster-client.py ?

I think we need a fix in the API as well as to account for certs within mozregression.  Maybe I'm looking at this wrong...

Comment 20

[Julien Pagès (:parkouss)]

Yes, being able to use buildSignedUrl with cert authentication would be a huge win for mozregression - because then we would only require to add the right parameter. I believe Jonas will tell us if that is possible.

Comment 21

[Jonas Finnemann Jensen (:jonasfj)]

> buildSignedUrl with cert authentication would be a huge win for mozregression
This is supported by out of the box.

Temp credentials are creds with the only limitation that they can't issue new temporary credentials.
Otherwise, they work as normal credentials would work.

As for the auth flow, I hope we can improve the UI to make it less confusing, because the flow has a
lot of good techincal aspects: namely anyone can integration it, users only gets temp creds so leakage
is a minor concern.

(Granted maybe we should build in a way that clients can do long-polling rather having a webserver on localhost, which doesn't work when ssh'ing into another machine)

Comment 22

[Julien Pagès (:parkouss)]

Attachment: use certificate in credentials

So, I have something working here. :) I can download aries builds, finally. ;

Thanks Jonas for the taskcluster release!

The support in the configuration file of the taskcluster client/token is still active - so you have to remove the credentials in your configuration file if any to use the certificate based authentication.

The second commit replace '%2F' in taskcluster artifacts urls to '/'. I don't know when/why it is like this, but this is breaking mozregression persist option. Urls looks like:

https://queue.taskcluster.net/v1/task/OXKBfifVROaLvcL_SOgdNw/runs/0/artifacts/private%2Fbuild%2Faries.zip?bewit=d...

And I am pretty sure we previously had

https://queue.taskcluster.net/v1/task/OXKBfifVROaLvcL_SOgdNw/runs/0/artifacts/private/build/aries.zip?bewit=d...

(I noticed the same thing on some public urls)

So Will, I'm asking you for a review -

Comment 23

[Julien Pagès (:parkouss)]

(In reply to Julien Pagès (:parkouss) from comment #22)

> So Will, I'm asking you for a review -

Err sorry I'm a bit tired I guess, this sentence is not finished. :) So I'm asking you for a review, but I don't know if you can test it. Anyway this is working fine for me, and comment or test from others is welcome!

Comment 24

[William Lachance (:wlach)]

Comment on attachment 8700859 [details] [review]
use certificate in credentials

All the code looks reasonable!

Comment 25

[Julien Pagès (:parkouss)]

Great. :)

Landed in:
https://github.com/mozilla/mozregression/commit/590d79c20dc9e63370574ec80a9a119f5ddeb9fd
https://github.com/mozilla/mozregression/commit/d9c9a816caecc8cfcbfede1964953fbb08f79038

Will try to release mozregression soon.

Comment 26

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I tested and verified the patch : 
1) took in the pull via fetch, cherry-pick
2) modified ~/.mozilla/mozregression/mozregression.cfg and commented out the taskcluster-accesstoken and the taskcluster-clientid by putting a # in front of it
3) then went to my mozregression git pull again and ran : 
python main.py --bad 2015-09-10 --good 2015-09-07  --repo mozilla-central --app b2g-aries -B debug

The work flow is that it will pop a browser window which you have to click on the grant access and it will run through with a temp credential.  It's a bit of a pain for the first time, but a consecutive regression will not show the browser.

I'm not sure what will happen if the credit expires.  We might have to delete the taskcluster-credentials.json  in the ~/.mozilla/mozregression folder, at that point?  Or maybe we'll be fine and it will auto request.  I'm not sure how I can speed up my expiration on my temp cred.

Comment 27

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Please push this to mozregression soon because it would be easier for our testers to just do : 
pip install --upgrade mozregression  

;)  Thanks for the fix!  Worked well!

Comment 28

[Julien Pagès (:parkouss)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #26)
> I tested and verified the patch : 
> 1) took in the pull via fetch, cherry-pick
> 2) modified ~/.mozilla/mozregression/mozregression.cfg and commented out the
> taskcluster-accesstoken and the taskcluster-clientid by putting a # in front
> of it
> 3) then went to my mozregression git pull again and ran : 
> python main.py --bad 2015-09-10 --good 2015-09-07  --repo mozilla-central
> --app b2g-aries -B debug

Thanks for testing!

> The work flow is that it will pop a browser window which you have to click
> on the grant access and it will run through with a temp credential.  It's a
> bit of a pain for the first time, but a consecutive regression will not show
> the browser.
> 
> I'm not sure what will happen if the credit expires.  We might have to
> delete the taskcluster-credentials.json  in the ~/.mozilla/mozregression
> folder, at that point?  Or maybe we'll be fine and it will auto request. 
> I'm not sure how I can speed up my expiration on my temp cred.

It should be all automatic. mozregression should ask again for a new cert, and overwrite the old one.

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #27)
> Please push this to mozregression soon because it would be easier for our
> testers to just do : 
> pip install --upgrade mozregression  
> 
> ;)  Thanks for the fix!  Worked well!

You're welcome! And there is a new version, 2.1.0 with this fixed:

http://mozilla.github.io/mozregression/news.html#2.1.0-release-and-gui-0.7.0

> pip install --upgrade mozregression

Should be enough (just be sure your testers remove the taskcluster related options in the config file if any).