Closed Bug 1178292 Opened 4 years ago Closed 4 years ago

Make partner repacks upload to S3

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: coop)

References

Details

Attachments

(1 file)

Partner repacks currently upload with post_upload.py. In order for them to hook into release promotion properly they'll need to upload to Taskcluster instead.
We talked about one possible implementation of this being a Mozharness script that wraps the existing partner repacks script, and then uploads to TC. This would probably be simpler than making the current script upload to TC directly, and it would be one step towards moving partner repacks to Mozharness.
No longer necessary because partner repacks are getting overhauled in bug 1181183.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
Blocks: 1181183
We still need to change where the partner repacks are uploaded to since ftp is going away. It's possible we can still use the S3 functionality of TC, but it may be easier to just use boto directly, tinys3, or a similar module.

I also don't think this blocks release promotion per se. The repack script can be plugged into the process to grab & repack release builds after promotion happens.
Group: mozilla-employee-confidential
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Summary: make partner repacks upload to Taskcluster → Make partner repacks upload to S3
:jlund received credentials from :mostlygeek Aug 19. They are in releng/private/passwords

Remaining to do: verify both:
 a) we have s3 access from the partner box, and
 b) the credentials work
I need to get some builds for yandex done out of github, is there any estimate on completing this?  Seems like everything necessary is in place.
Flags: needinfo?(coop)
 (In reply to Shane Caraveo (:mixedpuppy) from comment #5)
> I need to get some builds for yandex done out of github, is there any
> estimate on completing this?  Seems like everything necessary is in place.

I'll have time on Friday to look at this.
Flags: needinfo?(coop)
Sorry, I've had a million buildduty fires to put out between last week and now. I finally had an uninterrupted hour today to sit down and look at s3cmd and try an upload from partner-repack1.

I used the same s3cmd command that Shane has tentatively added on the s3 branch in github:

https://github.com/mozilla-partners/repack-scripts/blob/s3/partner-repacks.py#L1019

The only change was to use s3://fxpartners-distros/coop-test/ as the bucket. I used the existing repacked_builds dir from the recent vi funnelcake bug.

Here is the output:

partner-repack1:scripts cltbld$ s3cmd -c ~/.s3cfg sync --skip-existing -r ./repacked_builds/ s3://fxpartners-distros/coop-test/
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe  [part 1 of 3, 15MB]
 15728640 of 15728640   100% in    4s     3.20 MB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe  [part 2 of 3, 15MB]
 15728640 of 15728640   100% in    4s     3.42 MB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe  [part 3 of 3, 10MB]
 10602184 of 10602184   100% in    3s     3.05 MB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe.asc -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake45/win32/vi/Firefox Setup 40.0.3.exe.asc  [2 of 4]
 836 of 836   100% in    0s     8.12 kB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe  [part 1 of 3, 15MB]
 15728640 of 15728640   100% in    3s     4.76 MB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe  [part 2 of 3, 15MB]
 15728640 of 15728640   100% in    3s     4.26 MB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe  [part 3 of 3, 10MB]
 10532136 of 10532136   100% in    3s     3.15 MB/s  done
./repacked_builds/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe.asc -> s3://fxpartners-distros/coop-test/signed/funnelcake/40.0.3-1/funnelcake46/win32/vi/Firefox Setup 40.0.3.exe.asc  [4 of 4]
 836 of 836   100% in    0s     8.17 kB/s  done
Done. Uploaded 84050552 bytes in 22.7 seconds, 3.53 MB/s. Copied 0 files saving 0 bytes transfer.

So...seems to work.
Shane: assuming we use the proper s3://fxpartners-distros bucket for future, real partner uploads, are you happy with the rest of the resulting directory structure? 

I'm specifically wondering if we should recurse from the "signed" dir, ending up with a command similar to the following:

s3cmd -c ~/.s3cfg sync --skip-existing -r ./repacked_builds/signed/ s3://fxpartners-distros/
Flags: needinfo?(mixedpuppy)
that would be the correct command.  I thought that was in the script already....yes...

partner-repacks.py --s3sync ....

that will produce the command

s3cmd -c ~/.s3cfg sync --skip-existing -r "script_directory"/repacked_builds/signed/ s3://fxpartners-distro

You can change the location of the s3cfg file using --s3cfg and you can change the bucket using --bucket

if MOZ_SIGN_CMD is not set, the unsigned directory will be used (./repacked_builds/unsigned/)
Flags: needinfo?(mixedpuppy)
btw, I'm unsure what we want to do with funnelcake.  It could just go into the partner s3 bucket, and whoever wants to download will need to login on the fxpartners website to get the binaries.  Or, you can use the existing upload functionality in partner-repacks.py to upload to the public downloads site.
(In reply to Shane Caraveo (:mixedpuppy) from comment #10)
> btw, I'm unsure what we want to do with funnelcake.  It could just go into
> the partner s3 bucket, and whoever wants to download will need to login on
> the fxpartners website to get the binaries.  Or, you can use the existing
> upload functionality in partner-repacks.py to upload to the public downloads
> site.

There's a meeting this afternoon to discuss Funnelcake automation. It's a manual process right now, so we can continue moving them by hand in the interim.
(In reply to Shane Caraveo (:mixedpuppy) from comment #9)
> that would be the correct command.  I thought that was in the script
> already....yes...
> 
> partner-repacks.py --s3sync ....
> 
> that will produce the command
> 
> s3cmd -c ~/.s3cfg sync --skip-existing -r
> "script_directory"/repacked_builds/signed/ s3://fxpartners-distro
> 
> You can change the location of the s3cfg file using --s3cfg and you can
> change the bucket using --bucket
> 
> if MOZ_SIGN_CMD is not set, the unsigned directory will be used
> (./repacked_builds/unsigned/)

Planning to try an end-to-end run from the github sources + upload to S3 in a production-like environment this afternoon. If I can make that process work manually from there, I can modify the existing automation to pull from git and get s3cmd and its configs into puppet/hiera/tooltool.
I don't see anything sensitive here, making this public so our partners can follow when builds will be available.
Group: mozilla-employee-confidential
We disabled the partner repacks as part of the release process for Firefox 41 with the idea that we would generate the repacks for this cycle by hand using the new git+s3 process and then hook that into automation. 

Yesterday I generated the repacks by hand for Firefox 41. Here are my notes from that process:

* default signing token validity duration is too short at 2 hours. To get through *just* the Mac repacks, the duration needs to be at least 3 hours. To get through *all* the various platforms in one go, the duration would need to be increased to at least 6 hours. You can affect the duration by changing the -d option passed to get_token.py in the signing-server-setup.sh script in the repack-scripts repo.

* to cope with the token duration problem, I increased the duration to 3 hours and repacked each platform individually (similar to how we had it setup in automation), generating a new token between each one.

* only 3 platforms matter for releases: linux, mac, win32. We don't ship win64 on release, so don't bother trying to create them for release builds. Do create them for beta builds though.

* uploading all the repacks to S3 takes a fair amount of time by itself: for the 41.0 repacks, it took 82 minutes just to upload.

* I only ran the partner-repack.py script with the --s3sync for the last platform I repacked (Mac) and it uploaded all the repacks in one go.

Ideas for improvements if we need to ship partner repacks before automation is ready, e.g. if there's a 41.0.1 chemspill or similar:

* the easiest way to get through all the repacks in one go will be to set a large duration on the signing token (say 8 hours) and then run everything in one go. Trying to do one platform at a time for this round  meant that there were delays where I didn't start the next platform in a timely manner because I was e.g. stuck in a meeting.

* consider finding a way to upload each repack individually in the background as it is completed to avoid the extra 80-90 minutes for batch upload at the end.
(In reply to Chris Cooper [:coop] from comment #14)
> * only 3 platforms matter for releases: linux, mac, win32. We don't ship
> win64 on release, so don't bother trying to create them for release builds.
> Do create them for beta builds though.

We should probably flip this to False too:
http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/release-firefox-mozilla-beta.py.template#l155

Or strip everything out of the hg partner-repacks except EME-free, so those are still automated until we build something new for that.

> * the easiest way to get through all the repacks in one go will be to set a
> large duration on the signing token (say 8 hours) and then run everything in
> one go.

The signing server allows up to 7 hours, controlled by
https://dxr.mozilla.org/build-central/source/puppet/modules/signingserver/templates/signing.ini.erb?offset=200#45
You can just create a new manifest repo for other types of builds (e.g. eme-free), that way you can simply pull those builds.  The same command line for partner-repacks.py that you use from the hg repo should still work with the version in github (if not it can be fixed).  e.g. here is a funnelcake only manifest:

https://github.com/mozilla-partners/funnelcake-manifest
(In reply to Nick Thomas [:nthomas] from comment #15)
> We should probably flip this to False too:
> http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/release-
> firefox-mozilla-beta.py.template#l155

Does only the template need to change? I modified the actual copy in-tree too.
Attachment #8667946 - Flags: review?(nthomas)
Assignee: nobody → coop
Comment on attachment 8667946 [details] [diff] [review]
[buildbot-configs] Disable partner-repacks for betas

The template is sufficient but doing both is fine too.
Attachment #8667946 - Flags: review?(nthomas) → review+
Comment on attachment 8667946 [details] [diff] [review]
[buildbot-configs] Disable partner-repacks for betas

Review of attachment 8667946 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/56bcb5e87c33
Attachment #8667946 - Flags: checked-in+
The partner-specific side is done here. 

The two outstanding bits are Mozilla-specific: the funnelcake and EME-free builds. These both have special requirements WRT where they end up in the candidates/releases directory, e.g. http://archive.mozilla.org/pub/firefox/candidates/42.0b6-candidates/build1/win32-EME-free/

Rail pointed me at the product-delivery repo this morning: https://github.com/mozilla-services/product-delivery-tools/tree/master/post_upload

We're currently using s3cmd to upload the other repacks to partner-specific buckets. I'm wondering whether we could use s3cmd for these Mozilla repacks as well, or whether we'll need to rely on a post_upload intervention too. 

Specifically, do we rely on the post_upload step to create the directory hierarchy, or can it be done with metadata on upload? I've started poking at the product-delivery repo to figure this out, but I don't know Go.

Nick: do you know enough to answer the above? Should we loop in oremj?
Flags: needinfo?(nthomas)
(In reply to Chris Cooper (away until Oct 19) [:coop] from comment #20)
> Nick: do you know enough to answer the above? Should we loop in oremj?

Nick pointed me at the creds that I need to upload the funnelcakes and EME-free builds.
Flags: needinfo?(nthomas)
Created https://github.com/mozilla-partners/repack-scripts/pull/8 - "Allow individual partner configs to override s3 upload params
This seems to be working fine now.
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.