983116 - staging release runner instructions don't work with new dev-master1/ssh key management

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

08:58 <@nthomas|away> [05:31:25] So now we have a nice puppeteized dev-master, it comes along and wipes out the ~/.ssh hacks we need to run release-runner
08:58 <@nthomas|away> [05:32:04] as described at https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Staging_Specific_Notes#Release_runner

We probably need to do one of the following:
- somehow not clobber keys in our homedirs
- use our own keys instead of stage-ffxbld
- look for keys somewhere else, that won't get clobbered

Comment 1

[[e-mail] (:bhearsum)]

It was pretty easy to make things work with my own key: https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Staging_Specific_Notes#Release_runner

But I've realized that our own keys have push access to production repos, so that might not be a great strategy after all...what do you think?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That's a good point. Can we teach fabric to update local masters without resorting to ssh ?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #2)
> That's a good point. Can we teach fabric to update local masters without
> resorting to ssh ?

This seems worthwhile, and probably not too difficult. I'll give it a shot.

Comment 4

[[e-mail] (:bhearsum)]

Apparently this is a highly wanted feature Fabric: https://github.com/fabric/fabric/issues/98

It'd be easier with that having been fixed, but even without it we should be able to make it happen pretty easily by using local() instead of run()...

Comment 5

[[e-mail] (:bhearsum)]

Attachment: make local actions possible

I only did basic testing on this so far...what do you think of the approach?

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8392313 [details] [diff] [review]
make local actions possible

Review of attachment 8392313 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/util/fabric/actions.py
@@ +18,5 @@
>      os.path.dirname(__file__),
>      "../../../../buildfarm/maintenance/buildbot-wrangler.py"))
>  
>  
> +def run(cmd, workdir=None):

I have a little concern here. This fuction kind of mimics facric's run function buf the signature is different. I wonder if there is a better way to do this. Maybe using *args, **kwargs and pass them to the wrappped functions?

@@ +19,5 @@
>      "../../../../buildfarm/maintenance/buildbot-wrangler.py"))
>  
>  
> +def run(cmd, workdir=None):
> +    if env.host_string in ('127.0.0.1', 'localhost'):

Can you move this condition to a function so you can reuse it below too?

@@ +21,5 @@
>  
> +def run(cmd, workdir=None):
> +    if env.host_string in ('127.0.0.1', 'localhost'):
> +        if workdir:
> +            cmd = "cd %s && %s" % (workdir, cmd)

maybe use lcd() to avoid mungling cmd?

@@ +34,5 @@
> +def put(src, dst):
> +    if env.host_string in ('127.0.0.1', 'localhost'):
> +        return shutil.copyfile(src, dst)
> +    else:
> +        return fabric_put(src, dst)

Similar here, you may want to pass some other params, like mode or mirror_local_mode. Passing *args, **kwargs would be better.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #6)
> Comment on attachment 8392313 [details] [diff] [review]
> make local actions possible
> 
> Review of attachment 8392313 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: lib/python/util/fabric/actions.py
> @@ +18,5 @@
> >      os.path.dirname(__file__),
> >      "../../../../buildfarm/maintenance/buildbot-wrangler.py"))
> >  
> >  
> > +def run(cmd, workdir=None):
> 
> I have a little concern here. This fuction kind of mimics facric's run
> function buf the signature is different. I wonder if there is a better way
> to do this. Maybe using *args, **kwargs and pass them to the wrappped
> functions?

run() and local() take different arguments, I'm not sure how this would work...any ideas?

Comment 8

[Rail Aliiev [:rail]]

It's ok then. We can change the signature in the future if we need.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: address review comments

I'm not awfully happy about the silly "doit" function, but I don't know of a better way to factor out a block that might or might not have a context manager...

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8393732 [details] [diff] [review]
address review comments

Landed this. But I realized that I may have partly missed the point :(. I don't think we can do staging releases without forwarding our keys just with this...this only solves the fabric part, not the need to push to hg.m.o in the first place. Might be able to address the latter part by tweaking where release runner clones from in the first place.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: don't use remote paths for local repos

With this patch + the new instructions/configs at https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Staging_Specific_Notes#Ship_It, I got things working without any ssh key forwarding \o/

Comment 12

[[e-mail] (:bhearsum)]

Should be okay here now.

Comment 13

[[e-mail] (:bhearsum)]

I think this still isn't fixed. I started doing a staging release for 30.0b1 today and realized that release runner only pushes to my master dir's clones...which means that my user repo isn't tagged, and anything that tries to update to things like FIREFOX_30_0b1_RELEASE fails.

Comment 14

[[e-mail] (:bhearsum)]

I think I've fixed things up now. I was able to kick off releases using my forwarded ssh key when running release runner. I'm going to give it another run through after my current staging release is done, though.

If anyone else is doing one soon, I'd appreciate it if they'd give the latest version of https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Staging_Specific_Notes a try, too.

Comment 15

[[e-mail] (:bhearsum)]

I tried these instructions out again and they worked. I'm calling this fixed (again).

The tl;dr version is that you need to forward your ssh key when running release runner.