795236 - add oauth support to mozharness talos

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect, P3)

Description

[Aki Sasaki (not active)]

This probably means we need an optional FileDownload step(s) in ScriptFactory, or a different solution.

Without this, we'll regress bug 776612 when we push mozharness talos out to m-c and project branches.

Comment 1

[[e-mail] (:bhearsum)]

I realize it's a bit more work, but I think having some sort of secret server/service would be better here because it leaves us less tied to Buildbot.

Comment 2

[Jeff Hammel]

Yeah, not sure I have the knowledge of how all the deployment works to comment on the best solution.  Talos currently have --authfile.  This could be altered to take a URL if desired.  Or if you plan on continue to puppet this to the slaves then perhaps just a mozharness.mozilla.testing.talos --authfile option could be added and passed on to talos?

Comment 3

[[e-mail] (:bhearsum)]

I was more thinking that Mozharness would deal with the service, actually. I don't think we need to change the Talos interface here.

Comment 4

[Jeff Hammel]

So looking at bug 776612 and TBPL logs, we have the credentials file deployed to all existing machines. Inferring what is desired to here, it sounds like:

- mozharness's talos runner should have a configuration variable, say 'authfile'
- if 'authfile' is specified, it should be DLed from its URL (I think? should it be put somewhere temporary? permanant?
- the talos subprocess should be invoked with --authfile `config['authfile']`

Is this about right?  If I'm to work on this, I'll also need to know the URL of the authfile we want to hit.

I would also tend to support the use case where 'authfile' is a local file; that is, if '://' in config['authfile'], interpret it as a URL; otherwise, it is a local path.  Is this a use case we care about?  Would people object if I coded it this way?

This has been a blocker for jetperf for months now.  It seems like about 30 minutes of coding time to get this unblocked.  I'll gladly do the work if I can understand the problem and have all the missing pieces.

Comment 5

[Aki Sasaki (not active)]

(In reply to Jeff Hammel [:jhammel] from comment #4)
> So looking at bug 776612 and TBPL logs, we have the credentials file
> deployed to all existing machines.

Actually, not completely accurate.
The credentials are on the master, and bug 776612 uses a buildbot FileDownload step to download a file from the master onto the slave.
Mozharness (run through a ShellCommand step) doesn't have the rights to files on the master that a buildbot FileDownload step has, so we'd have to put the credentials elsewhere.

> - mozharness's talos runner should have a configuration variable, say
> 'authfile'
> - if 'authfile' is specified, it should be DLed from its URL (I think?
> should it be put somewhere temporary? permanant?
> - the talos subprocess should be invoked with --authfile `config['authfile']`

I think that works.
Since the authfile has sensitive info, I'd prefer if we removed the authfile afterwards.

> Is this about right?  If I'm to work on this, I'll also need to know the URL
> of the authfile we want to hit.

I'd be fine if we had the capability of downloading an authfile from any url and using it.  We could deal with putting the production authfile in a secure location and specifying the real url later.

That would mean you could put a test authfile on, say, people.m.o; if that worked, we could point production at the real url when landing.

> I would also tend to support the use case where 'authfile' is a local file;
> that is, if '://' in config['authfile'], interpret it as a URL; otherwise,
> it is a local path.  Is this a use case we care about?  Would people object
> if I coded it this way?

I think it's fine to allow a local file, as long as it supports the production url.

> This has been a blocker for jetperf for months now.  It seems like about 30
> minutes of coding time to get this unblocked.  I'll gladly do the work if I
> can understand the problem and have all the missing pieces.

Agreed.  I don't think you actually need the real url or real authfile to be able to write and test this, so go for it.

Comment 6

[Jeff Hammel]

Come to think of it....is there any reason that talos shouldn't real the remote file itself?  Is it http:// accessible (or planned to be)?

Comment 7

[Aki Sasaki (not active)]

Sure it could, talos --oauth-url http://url/to/auth ...

http will be available for internal hosts, not the internet, but I think that should be sufficient.

Comment 8

[Jeff Hammel]

with 822478 fixed, pending a talos deployment, all that should be necessary is updating the production mozharness config files

Comment 9

[Jason Yeo[:jyeo]]

This is fixed in Bug 879973. We added oauth to mozharness talos so that it can connect to datazilla.