809174 - Create a script to copy hosted partner repacks to the appropriate dir on stage

Status: RESOLVED DUPLICATE of bug 884914

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

Most partners host their own repacks, but Mozilla does host the repacks for a limited subset of partners (e.g. Bing). These repacks are hosted outside of the regular release hierarchy on releases.m.o, and currently Tomcat needs to move these files in place by hand. Often the files also need to be renamed. As with any predictable, repeated process, this cries out to be automated.

Tomcat: I'm not sure how your python skills are, but I'm thinking this is a good opportunity to pick them up or brush them off. The alternative would be a shell script, but I think we're going to want some configurable options per partner which is why I lean towards python.

I would recommend adding the script to the scripts/ dir under https://hg.mozilla.org/build/partner-repacks/. Since there may be partners involved that can't be mentioned in a publicly accessible config file, the script should probably include the ability to merge config info from two sources, unless we just want to keep a locally-modified copy (danger!) on the stage machine.

Down the road, we may want to have these files end up in the correct final location by default (possibly by adding some vars to the appropriate repack.cfg), but this will at least remove some manual steps for now.

Comment 1

[Carsten Book [:Tomcat]]

yeah i agree on the python stuff, will work on it, its a good learning thing i guess :)

Comment 2

[Carsten Book [:Tomcat]]

ok here are the steps (only talking about bing for now, since it has specific requirements for the name of the files) that i do manual and need to be automated.

For Mac (Bing has only Mac and Windows Builds)

cd /mnt/netapp/stage/releases.mozilla.com/bundles/bing/mac/en-US # the location on stage for the bing builds

cp /mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates/16.0.2-candidates/build1/partner-repacks/bing/mac/en-US/Firefox\ 16.0.2.dmg . # copies the files from the candidate directory to the destination folder

mv Firefox\ 16.0.2.dmg Firefox-Bing.dmg # because no Version Number can be in the Files and the requirement has Firefox-Bing.dmg we rename it

Similar for Windows - but we use a complete different final Name for the Installer with Firefox-Bing Setup.exe

cd /mnt/netapp/stage/releases.mozilla.com/bundles/bing/win32/en-US
cp /mnt/netapp/stage/archive.mozilla.org/pub/firefox/candidates/16.0.2-candidates/build1/partner-repacks/bing/win32/en-US/Firefox\ Setup\ 16.0.2.exe .
mv Firefox\ Setup\ 16.0.2.exe Firefox-Bing\ Setup.exe

Comment 3

[[e-mail] (:bhearsum)]

Will not strictly part of this bug, we should probably integrate this into the release automation as well. I can provide some guidance on how to do that after the script is ready.

Comment 4

[Carsten Book [:Tomcat]]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Will not strictly part of this bug, we should probably integrate this into
> the release automation as well. I can provide some guidance on how to do
> that after the script is ready.

yeah that would be great and would avoid a single point of failure with human acting :) so yeah that sounds great

Comment 5

[Carsten Book [:Tomcat]]

Attachment: wip still working on this

still working on this to figure all stuff out, but argparse seems a great way

Comment 6

[Carsten Book [:Tomcat]]

(In reply to Chris Cooper [:coop] from comment #0)
> Down the road, we may want to have these files end up in the correct final
> location by default (possibly by adding some vars to the appropriate
> repack.cfg), but this will at least remove some manual steps for now.

hm the vars to the repack script are something that need to be carefully done, since we broke mac builds on some partner with a value in their, see bug 805788 (this comment is mostly for myself to check that when its done :)))

Comment 7

[Carsten Book [:Tomcat]]

Attachment: created a shell script that does the job - v1

since i found out i really need to start into python from scratch, but better in shell script i have created this first v1

Comment 8

[Carsten Book [:Tomcat]]

Comment on attachment 683088 [details]
created a shell script that does the job - v1

hey coop can you take a look, a example command with the options is in the first line

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 683088 [details]
created a shell script that does the job - v1

This is fine as a first pass, but I have some nits:

* add as a first line: "#!/bin/bash" <- this will let you run the script directly assuming the executable bit is also set
* make sure to fix up the indentation on the conditionals before landing

This can probably live in the partner-repacks/scripts dir for now. 

Assuming this script lives in a defined place on (dev-)stage, I wonder if we could already run this as the POST_UPLOAD_CMD? Idle speculation at this point, but we should check it in staging at some point.

r+ with the nits fixed.

Comment 10

[Carsten Book [:Tomcat]]

cool thanks, working on the changes and also will check the post_upload stuff 

(In reply to Chris Cooper [:coop] from comment #9)

> Assuming this script lives in a defined place on (dev-)stage, I wonder if we
> could already run this as the POST_UPLOAD_CMD? Idle speculation at this
> point, but we should check it in staging at some point.
> 
> r+ with the nits fixed.

Comment 11

[Carsten Book [:Tomcat]]

Attachment: nits adressed

Comment 12

[[e-mail] (:bhearsum)]

I think this is done in stage-tasks.py now...