1388578 - Request for secure PGP-encrypted/emailed API key for wpt.stage.mozaws.net instance

Status: VERIFIED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Is it possible to get the api_key for https://wpt.stage.mozaws.net/ emailed to me via PGP, to [e-mail]?

Context: My Q3 goal is automated performance-regression testing for AMO, and right now I'm using the public instance of http://www.webpagetest.org/ with a 200-calls-per-day API limit.

For now, it'd be invoked primarily through https://fx-test-jenkins-dev.stage.mozaws.net:8443/ or https://fx-test-jenkins.stage.mozaws.net/

Thanks!

(I just realized I'll also likely have to work out how/if it's possible to get through the Auth0 layer, hrm...)

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

(The following is for posterity, but let me know if it should be in a separate bug, if at all.)

I should clarify that I'm actually using a NodeJS wrapper for the WPT RESTful API, via https://github.com/marcelduran/webpagetest-api

In chatting with Greg a bit, he mentioned it might be possible to whitelist the API endpoints to bypass Auth0 and letting the app handle the API keys (paraphrasing).

Comment 2

[Greg G]

Assuming we go with the whitelist approach from https://bugzilla.mozilla.org/show_bug.cgi?id=1376465 we'd whitelist and possibly rate limit the endpoints:

/runtest.php
/testStatus.php 
/getLocations.php
/xmlResult/091111_2XFH/?r=12345
/cancelTest.php?test=<testId>
/getLocations.php?f=xml

per https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis

and rely on the wpt's API key access control.

currently the nginx config is in this private repo and proxies everything through auth0: https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/wpt/puppet/modules/wpt/templates/http_wpt.conf.erb

Comment 3

[Greg G]

+relud: since he's listed as the service owner

Comment 4

[Daniel Thorn [:relud]]

not everything is through auth0. if it's possible to submit tests while behind the vpn, just send to http://wpt-vpn.stage.mozaws.net

if vpn isn't an option, then configuring the host submitting to have a static public ip, and I will open a firewall rule for it, like we did for :digitarald's thinkpad client.

> and rely on the wpt's API key access control.

:g-k that won't work because wpt uses a query parameter to specify the api key when polling for work, but uses a POST form-field to specify the key when submitting test results, and I couldn't get lua nginx to reliably detect the presence of the api key in a POST body. instead we should stick to the access methods already in place for wpt for now, and use vpn/ip controls.

Comment 5

[Greg G]

(In reply to Daniel Thorn [:relud] from comment #4)

Good to know! If wpt-vpn works, lets do that and keep the nginx config simple.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Daniel Thorn [:relud] from comment #4)
> not everything is through auth0. if it's possible to submit tests while
> behind the vpn, just send to http://wpt-vpn.stage.mozaws.net
> 
> if vpn isn't an option, then configuring the host submitting to have a
> static public ip, and I will open a firewall rule for it, like we did for
> :digitarald's thinkpad client.

I tried submitting to http://wpt-vpn.stage.mozaws.net from our current dev Jenkins instance[0], but it failed to connect to port 80, from there (I'm not familiar with AWS and its config, much, so it could be a matter of tweaking the AWS IAM role(s) and/or ports/flows.

Here's that quick test run of the above: https://fx-test-jenkins-dev.stage.mozaws.net:8443/job/wpt-amo-homepage/58/console

Ops is provisioning us (through CloudBees) Jenkins Enterprise instances, which will be replacing our current dev/prod ones, so it's likely that whatever we put in place here might have to be revisited for that setup, just FYI (:oremj and :ckolos are leading that charge).

Happy to work through this over IRC or Vidyo, tomorrow, too, if that's faster/easier.

[0] https://mana.mozilla.org/wiki/display/TestEngineering/fx-test-jenkins-dev.stage.mozaws.net

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Know you're (always) busy, and thanks for soldiering through bug 1379044, Daniel!  Would you have some time next week to try working together to look at this, again?

Comment 8

[Daniel Thorn [:relud]]

Okay, I think this is worked out. :stephend has been sent creds for using https://wpt-api.stage.mozaws.net/ and I've confirmed that will work with his tooling.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Thanks; verified this is fixed, using the same jobs, with just the tweaks for server URL + authentication!