Closed Bug 785965 Opened 12 years ago Closed 12 years ago

adjust push to mirrors process for CDN

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

Attachments

(6 files)

Since we started fully using the CDN for releases we don't have the ability to push to internal mirrors the same way as before. Basically, whenever we copy to the releases directory we end up 5,000,000 uptake. The release automation docs still talk about a multi part process for final releases:
* uptake rsyncd files
* create index files
* copy to releases directory (aka push to internal mirrors)
* push to external mirrors

A few pages need updating:
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Tagging_through_Signing
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping
https://wiki.mozilla.org/Releases/RelEngChecklist
Assignee: nobody → bhearsum
To expand a bit more on the process outlined in comment #0, I believe this is now the full process for final releases:
1) Update rsyncd files with a block like the one here at the start of the release: https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Tagging_through_Signing#Edit_rsync_exclude_files
2) Create index files, if necessary
3) Copy to the releases directory, which will put us on the CDN and cause 5,000,000. Note that we do _not_ wait for a go here anymore, as we can pull the files back if necessary.
4) After getting a go from drivers, push to the external mirrors (ie, community mirrors) by removing the earlier block from the rsyncd exclude file
5) Remove index files after shipping.

Lukas, Alex, Nick - does that sound right to you all?
Looks good to me, although I'm unfamiliar with the specifics of step 1.
Nit - s/5,000,000/50,000,000/ in comment #0 and 1.

In the last day or so several mirrors have said 'please disable me, I'm not getting much traffic any more but still have to pull new files in', and the traditional choke point getting files out via rsync is still slow. So people are heading towards dismantling the mirror system. Therefore I'd like to figure out what we would do in a CDN-only world, and then work back to address any issues people might have.

Imagine we don't have any rsync to worry about, the files simply get copied into releases/<version>/ and are available on the CDN a few minutes later. How about this for a final release:
1) Release automation runs as usual, QA checks updates on betatest channel
2) Create index files in candidates directory (if this still has value for driveby downloads)
3) After getting a signoff from QA and go from drivers, trigger automation to copy to the releases directory
4) QA checks releasetest, signs off (this verifies the bouncer config and CDN functional, rather than the updater & mar files are OK)
5) Push to release channel, web properties etc; remove index files, update latest symlink

If we have an issue where we have to 'unship' at 4) then we are relying on the purge mechanisms of the CDNs (IT has the access to those interfaces), or push the new files to a different locations. The latter is unchanged from now, the former should work but we haven't tested how reliable it is or the timescale. When we do 3) depends on how long 4) takes and when we want to ship, but potentially the day before an 8am ship. I don't think we'd end up with the files sitting in releases/ any more than we do already though.

In the case of betas it looks like this:
1) Release automation runs as usual, automatically coping into releases/
2) QA checks releasetest channel
3) After getting a signoff from QA and go from drivers, push to beta channel
(In reply to Nick Thomas [:nthomas] from comment #3)
> Nit - s/5,000,000/50,000,000/ in comment #0 and 1.
> 
> In the last day or so several mirrors have said 'please disable me, I'm not
> getting much traffic any more but still have to pull new files in', and the
> traditional choke point getting files out via rsync is still slow. So people
> are heading towards dismantling the mirror system. Therefore I'd like to
> figure out what we would do in a CDN-only world, and then work back to
> address any issues people might have.
> 
> Imagine we don't have any rsync to worry about, the files simply get copied
> into releases/<version>/ and are available on the CDN a few minutes later.
> How about this for a final release:
> 1) Release automation runs as usual, QA checks updates on betatest channel
> 2) Create index files in candidates directory (if this still has value for
> driveby downloads)
> 3) After getting a signoff from QA and go from drivers, trigger automation
> to copy to the releases directory
> 4) QA checks releasetest, signs off (this verifies the bouncer config and
> CDN functional, rather than the updater & mar files are OK)
> 5) Push to release channel, web properties etc; remove index files, update
> latest symlink

It sounds like step #3 is intended to happen the day before ship based on your next paragraph, so this sounds good to me.

> If we have an issue where we have to 'unship' at 4) then we are relying on
> the purge mechanisms of the CDNs (IT has the access to those interfaces), or
> push the new files to a different locations. The latter is unchanged from
> now, the former should work but we haven't tested how reliable it is or the
> timescale. When we do 3) depends on how long 4) takes and when we want to
> ship, but potentially the day before an 8am ship. I don't think we'd end up
> with the files sitting in releases/ any more than we do already though.

> In the case of betas it looks like this:
> 1) Release automation runs as usual, automatically coping into releases/
> 2) QA checks releasetest channel
> 3) After getting a signoff from QA and go from drivers, push to beta channel

It sounds like the only thing that's different compared to now in the past 3 paragraphs is that QA tests on betatest and we wait for an explicit go to push the files to releases/. Assuming my interpretation is correct, it sounds good to me!
I made a few edits which I hope will address this:
- Move "push to internal mirrors" after "create index files"
- Get rid of "e-mail mirrors", since we don't do that anymore.
- A few clarifications about when to do things and what authorization you need

https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&action=historysubmit&diff=471430&oldid=469402
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3ATagging_through_Signing&action=historysubmit&diff=471422&oldid=452189
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&action=historysubmit&diff=471428&oldid=469563

How does that look to folks?
Haven't seen any complains, I'm going to assume success!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Or we're just slow! Any reason to keep making rsync module changes if we're not pushing into releases/ until we go from relman ?
(In reply to Nick Thomas [:nthomas] from comment #7)
> Or we're just slow!

Sorry, hehe.

> Any reason to keep making rsync module changes if we're
> not pushing into releases/ until we go from relman ?

Hm, I had it in my head that we needed to keep doing them in order to have index files synced in advance of the real files. Thinking about it again, I see that we don't need any rsync changes for that, but we will need to tweak the index file creation so that it works without the files already being in releases/.

Alex, can you confirm whether or not index files are still wanted/necessary in the world where we're pushing to the releases/ directory the night before ship (instead of multiple days before)?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Ben Hearsum [:bhearsum] from comment #8)
> Alex, can you confirm whether or not index files are still wanted/necessary
> in the world where we're pushing to the releases/ directory the night before
> ship (instead of multiple days before)?

It'd be preferable, since I can't really predict the effect on press/PR.
(In reply to Alex Keybl [:akeybl] from comment #9)
> (In reply to Ben Hearsum [:bhearsum] from comment #8)
> > Alex, can you confirm whether or not index files are still wanted/necessary
> > in the world where we're pushing to the releases/ directory the night before
> > ship (instead of multiple days before)?
> 
> It'd be preferable, since I can't really predict the effect on press/PR.

OK. Nick, I made some additional edits that I think will address this. With them, we don't need rsync module edits at all and we still get index files:
https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&action=historysubmit&diff=472388&oldid=471586
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3ATagging_through_Signing&action=historysubmit&diff=472387&oldid=471422
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&action=historysubmit&diff=472396&oldid=471588

The index file creation is slightly hacky (due to ignoring "partner-repacks" and "logs" in it), but it seems to work.
Nick and I chatted about my latest edit yesterday and he pointed out that creating the index files before copying files to releases/ will break the "push to mirrors" builder when we run it, because it doesn't want the releases/x.y directory to exist when it runs. (This is a check we added to make sure we don't accidentally overwrite things.) A couple of ideas about how to work around that:
1) Have the "push to mirrors" builder check for a file or directory inside of the version directory rather than that dir itself. Eg, bail if releases/x.y/win32 exists.
2) Do the rsync exclude in the releases _and_ prereleases modules, run "push to mirrors" (which won't propagate because of the excludes), create the index files (which _will_ propagate), then remove the excludes after N minutes (20? 30?) after the index files have propagated to get the files out there.
3) Create the index files in the candidates directory and live with the fact that there will be a window in which some files will be present but not hidden by index files. (Alex, I'd appreciate your thoughts on this option.)
(In reply to Ben Hearsum [:bhearsum] from comment #11)
> 3) Create the index files in the candidates directory and live with the fact
> that there will be a window in which some files will be present but not
> hidden by index files. (Alex, I'd appreciate your thoughts on this option.)

bhearsum let us know this is on the order of an hour. Given that, I don't see press/PR impact.
(In reply to Alex Keybl [:akeybl] from comment #12)
> (In reply to Ben Hearsum [:bhearsum] from comment #11)
> > 3) Create the index files in the candidates directory and live with the fact
> > that there will be a window in which some files will be present but not
> > hidden by index files. (Alex, I'd appreciate your thoughts on this option.)
> 
> bhearsum let us know this is on the order of an hour. Given that, I don't
> see press/PR impact.

What do you think of this idea, Nick?
Where does the hour or so come from if the index.html are already in the directories when the push_to_mirrors builder runs ? Without trying it, I'd hope rsync would handle all the files in each directory together, rather than working in inode order over the whole sync (but it often surprises me!).

Would we remove the index.html from the candidates afterwards ? I can see them being a bit of a pain for QA and other legitimate users of the staging area.

2) is problematic because of the delay between committing to svn and puppet pushing it live. I'd love to make the modules go away anyway.
(In reply to Nick Thomas [:nthomas] from comment #14)
> Where does the hour or so come from if the index.html are already in the
> directories when the push_to_mirrors builder runs ? Without trying it, I'd
> hope rsync would handle all the files in each directory together, rather
> than working in inode order over the whole sync (but it often surprises me!).

One hour may be an overestimate but IIRC we started precreating the index files because creating them after copying the files in meant that if mirrors began syncing files while we were halfway through copying them in, or anytime prior to creating the index files, they would end up with some/all of the files visible until they sync up again. So the "hour" is my SWAG at long it would take for a mirror to get the index files after initially beginning a sync without them.

> Would we remove the index.html from the candidates afterwards ? I can see
> them being a bit of a pain for QA and other legitimate users of the staging
> area.

Yeah, that's a good point. We could create them during the "push to mirrors" builder instead. Basically, just automate what's currently in https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Dealing_with_index.html_files, or create them after the rsync.

> 2) is problematic because of the delay between committing to svn and puppet
> pushing it live. I'd love to make the modules go away anyway.

Good point. I don't think we want to add artificial delay here.
Nick and I chatted today to hammer this out. We're basically doing 3), except cleaning up the candidates directory afterwards. Specifically, we will do the following during the "push to mirrors" builder:
* Create index files in the candidates directory
* Copy the files (including index files) into the releases directory
* Delete the index files from the candidates directory.

Additionally, since I'll be poking around these scripts anyways, Nick suggested automated the clean-up of the index files from the candidates directory, and the updating of the "latest" symlink.

I'll get to work on this soon and it should be ready for 16.0.
Pretty straightforward, I think. I had to change some of the bash that we used to single-command stuff because I couldn't make backticks work with run_remote_cmd). I also renamed the script to stage-tasks.py so that it's more reflective of what it's doing.
Attachment #664956 - Flags: review?(nthomas)
Comment on attachment 664957 [details] [diff] [review]
adjust script name; add postrelease builder

>+        postrelease_factory = ScriptFactory(
>+            scriptRepo=tools_repo,
>+            script_timeout=3*60*60,

This can probably be much shorter than 3 hours. Otherwise looks good.
Attachment #664957 - Flags: review?(nthomas) → review+
Attachment #664958 - Flags: review?(nthomas) → review+
Comment on attachment 664956 [details] [diff] [review]
update scripts to support index files; symlink changing

r+ with fix needed on landing.

>diff --git a/lib/python/release/paths.py b/lib/python/release/paths.py
>+def makeReleasesDir(product, version=None, protocol=None, server=None,
>                     ftp_root='/pub/mozilla.org/'):
>     if protocol:
>         assert server is not None, "server is required with protocol"
> 
>-    directory = '%s%s/releases/%s/' % (ftp_root, product, version)
>+    directory = '%s%s/releases' % (ftp_root, product)
>+    if version:
>+        directory += '/%s' % version

You're dropping a trailing / here, which would go badly if we ever used it as the source of an rsync. As far as I can see we don't ever do that in tools/, so that's OK. Howerver there is a bare '%s%s' that relies on a trailing / at
  http://hg.mozilla.org/build/tools/file/default/scripts/updates/create-update-verify-configs.py#l131
so I'd expect update verify to fail if this lands as is. Maybe os.path.join() ?

The rstrip in scripts/staging/release_downloader.py is also becomes superfluous.

>diff --git a/scripts/release/push-to-mirrors.py b/scripts/release/stage-tasks.py
>+def deleteIndexFiles(cleanup_dir, stageServer, stageUsername,
>+                     stageSshKey):
>+    run_remote_cmd(['find', cleanup_dir, '-name', 'index.html', '-exec', 'rm', '{}', '\\;'],
>+                   server=stageServer, username=stageUsername, sshKey=stageSshKey)

If the cp in makeIndexFiles is verbose then this rm should be too.

>+        if createIndexFiles:
>+            makeIndexFiles(stageServer=stageServer,
>+                             stageUsername=stageUsername,
>+                             stageSshKey=stageSshKey,
>+                             productName=productName,
>+                             version=version,
>+                             buildNumber=buildNumber)

Nit: indent style doesn't match nearby function calls.

>+    if 'postrelease' in args:
>+        if createIndexFiles:
>+            deleteIndexFiles(stageServer=stageServer,
>+                            stageUsername=stageUsername,
>+                            stageSshKey=stageSshKey,
>+                            cleanup_dir=makeReleasesDir(productName, version))

Nit: indenting

>+        updateSymlink(stageServer=stageServer,
>+                      stageUsername=stageUsername,
>+                      stageSshKey=stageSshKey,
>+                      productName=productName,
>+                      version=version)
Attachment #664956 - Flags: review?(nthomas) → review+
(In reply to Nick Thomas [:nthomas] from comment #25)
> Comment on attachment 664956 [details] [diff] [review]
> update scripts to support index files; symlink changing
> 
> r+ with fix needed on landing.
> 
> >diff --git a/lib/python/release/paths.py b/lib/python/release/paths.py
> >+def makeReleasesDir(product, version=None, protocol=None, server=None,
> >                     ftp_root='/pub/mozilla.org/'):
> >     if protocol:
> >         assert server is not None, "server is required with protocol"
> > 
> >-    directory = '%s%s/releases/%s/' % (ftp_root, product, version)
> >+    directory = '%s%s/releases' % (ftp_root, product)
> >+    if version:
> >+        directory += '/%s' % version
> 
> You're dropping a trailing / here, which would go badly if we ever used it
> as the source of an rsync. As far as I can see we don't ever do that in
> tools/, so that's OK. Howerver there is a bare '%s%s' that relies on a
> trailing / at
>  
> http://hg.mozilla.org/build/tools/file/default/scripts/updates/create-update-
> verify-configs.py#l131
> so I'd expect update verify to fail if this lands as is. Maybe
> os.path.join() ?

Actually, we purposely don't use os.path.join because this is a remote path and would be wrong if run on Windows. I can tack on the trailing slash again, though, and fix up the tests to make sure we don't regress this. I did that and addressed your other review comments in this change: https://github.com/bhearsum/tools/commit/638e0c2b98c0a4ed31e7baf3f62beb5025e7227d
Summary: update release automation docs for new CDN → adjust push to mirrors process for CDN
Attachment #664956 - Flags: checked-in+
Comment on attachment 664957 [details] [diff] [review]
adjust script name; add postrelease builder

(In reply to Nick Thomas [:nthomas] from comment #24)
> Comment on attachment 664957 [details] [diff] [review]
> adjust script name; add postrelease builder
> 
> >+        postrelease_factory = ScriptFactory(
> >+            scriptRepo=tools_repo,
> >+            script_timeout=3*60*60,
> 
> This can probably be much shorter than 3 hours. Otherwise looks good.

I took this out altogether so we'll be using the default (20min)
Attachment #664957 - Flags: checked-in+
Attachment #664958 - Flags: checked-in+
Finally, the last doc updates:
https://wiki.mozilla.org/index.php?title=Releases%2FRelEngChecklist&action=historysubmit&diff=475575&oldid=472388
https://wiki.mozilla.org/index.php?title=Release%3ARelease_Automation_on_Mercurial%3AUpdates_through_Shipping&action=historysubmit&diff=475576&oldid=472651

Should be all done here now. I'll send mail to the group summarizing the changes.
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
makeIndexFiles failed during push of FF 16:
CalledProcessError: Command '['scp', '-o', 'BatchMode=yes', '/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/tmp1bITsO', 'ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/16.0-candidates/build1//index.html']' returned non-zero exit status 1
command: END (0.31s elapsed)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Specifically 
 Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Comment on attachment 669377 [details] [diff] [review]
[tools] pass stageSshKey on to scp call

only use of 'scp' in this script, so this should do it! :) r+
Attachment #669377 - Flags: review?(hwine) → review+
Comment on attachment 669377 [details] [diff] [review]
[tools] pass stageSshKey on to scp call

Landed:
 http://hg.mozilla.org/build/tools/rev/b9d8e3cfe2d6
and moved the FIREFOX_16_0_RELEASE tag to that.
Attachment #669377 - Flags: checked-in+
That worked ok, but revealed this

`/pub/mozilla.org/firefox/nightly/16.0-candidates/build1//index.html' -> `/pub/mozilla.org/firefox/nightly/16.0-candidates/build1/contrib/solaris_pkgadd/index.html'
cp: cannot create regular file `/pub/mozilla.org/firefox/nightly/16.0-candidates/build1/contrib/solaris_pkgadd/index.html': Permission denied
`/pub/mozilla.org/firefox/nightly/16.0-candidates/build1//index.html' -> `/pub/mozilla.org/firefox/nightly/16.0-candidates/build1/contrib/solaris_tarball/index.html'
cp: cannot create regular file `/pub/mozilla.org/firefox/nightly/16.0-candidates/build1/contrib/solaris_tarball/index.html': Permission denied
`/pub/mozilla.org/firefox/nightly/16.0-candidates/build1//index.html' -> `/pub/mozilla.org/firefox/nightly/16.0-candidates/build1/contrib-localized/index.html'

Probably ought to exclude the contrib directory from the find call.
The index.html ended up with mode 600 so the rsync daemon couldn't read them.
sigh. Thanks for dealing with this folks, I'll take it from here.
(In reply to Kim Moir [:kmoir] from comment #37)
> http://ftp.mozilla.org/pub/mozilla.org/xulrunner/releases/16.0/ is not
> available either

I removed the index files from that directory as a workaround for now.
So there's a few things here to address, it seems:
1) Stop trying to create index files in contrib directories
2) Fix permissions on index.html file
3) Don't create index files for xulrunner (or at the very least, fix the product name if we do)

I checked ESR logs, and they didn't seem to have any issues with this patch, presumably because makeIndexFiles isn't set. Need to make sure Betas work fine, too.
ESR (at least) has the postrelease builder enabled. Looks like it may be a problem, because updateSymlink() is always called and it updates "latest" symlink, while it should be "latest-esr": http://hg.mozilla.org/build/tools/file/56ee68859dca/scripts/release/stage-tasks.py#l173

Another releaseConfig variable? :)
xulrunner 16.0.1 index.html's removed too.
(In reply to Rail Aliiev [:rail] from comment #40)
> ESR (at least) has the postrelease builder enabled. Looks like it may be a
> problem, because updateSymlink() is always called and it updates "latest"
> symlink, while it should be "latest-esr":
> http://hg.mozilla.org/build/tools/file/56ee68859dca/scripts/release/stage-
> tasks.py#l173
> 
> Another releaseConfig variable? :)

Yeah, may as well given that we have bug 799514.
Attached patch fix all the bugsSplinter Review
I believe this patch fixes all of the issue we found here. I also rolled in Rail's patch from bug 799514 because it was simpler to test them together. Specific fixes are:
* Fix index.html permissions by explicitly chmod'ing the first one to 644.
* Don't try to create index files in anything matches 'contrib'. This means that the root contrib dirs don't get index files, but I don't think that matters.
* Get explicit symlink name from the release config.
* Special case XULRunner to avoid creating index files for it, like we do everywhere else.

For testing, I set up all the candidates directories with the full directory structures, and made sure the contrib dirs had similar permissions to FTP. We don't have a dave.lin on dev-stage01, so I did this:
drwxr-xr-x  4 ffxbld firefox 4096 Oct 17 05:11 contrib
drwxr-xr-x 2 apache firefox 4096 Oct 17 05:11 solaris_pkgadd
drwxr-xr-x 2 apache firefox 4096 Oct 17 05:11 solaris_tarball

...which should be equivalent.

Thunderbird Beta:
* Push to mirrors did not create index files (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-comm-beta-push_to_mirrors/builds/6/steps/run_script/logs/stdio)
* Postrelease updated "latest-beta" symlink (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-comm-beta-postrelease/builds/6/steps/run_script/logs/stdio)

Thunderbird Release:
* Push to mirrors did not create index files (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-comm-release-push_to_mirrors/builds/2/steps/run_script/logs/stdio)
* Postrelease updated "latest" symlink (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-comm-release-postrelease/builds/0/steps/run_script/logs/stdio)

Firefox Beta:
* No index files (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-beta-push_to_mirrors/builds/22/steps/run_script/logs/stdio)
* Postrelease created "latest-beta" symlink (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-beta-postrelease/builds/0/steps/run_script/logs/stdio)

XULRunner Beta:
* No index files (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-beta-xulrunner_push_to_mirrors/builds/3/steps/run_script/logs/stdio)

Firefox Release:
* Created index files, but not in contrib dirs (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-release-push_to_mirrors/builds/34/steps/run_script/logs/stdio)
* Removed index files from releases, updated symlink (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-release-postrelease/builds/3/steps/run_script/logs/stdio)

XULRunner Release:
* No index files (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-release-xulrunner_push_to_mirrors/builds/0/steps/run_script/logs/stdio)

Firefox ESR:
* No index files created (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-esr10-push_to_mirrors/builds/0/steps/run_script/logs/stdio)
* Symlink created (http://dev-master01.build.scl1.mozilla.com:8018/builders/release-mozilla-esr10-postrelease/builds/0/steps/run_script/logs/stdio)
Attachment #673923 - Flags: review?(nthomas)
Comment on attachment 673923 [details] [diff] [review]
fix all the bugs

>diff --git a/scripts/release/stage-tasks.py b/scripts/release/stage-tasks.py
>+    run_remote_cmd(['find', candidates_dir, '-mindepth', '1', '-type', 'd', '-not', '-regex', '.*contrib.*', '-exec', 'cp', '-pv', '%s/index.html' % candidates_dir, '{}', '\\;'],
>                    server=stageServer, username=stageUsername, sshKey=stageSshKey)

Could also exclude logs/, partner-repacks/ for a slight time win, but it's not breaking anything.
Attachment #673923 - Flags: review?(nthomas) → review+
Attachment #673924 - Flags: review?(nthomas) → review+
Attachment #673924 - Flags: checked-in+
Attachment #673923 - Flags: checked-in+
In production
Push to mirrors ran fine for 17.0b3 (modulo unrelated clobberer issues). 99% sure postrelease will too.
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
(In reply to Ben Hearsum [:bhearsum] from comment #48)
> 99% sure postrelease will too.

Add 1% here. Worked fine for Firefox 17.0b3.
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: