699553 - release automation shouldn't require/want a previous build of a release deleted before pushing an updated build

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, enhancement, P2)

Description

[[e-mail] (:bhearsum)]

Since we've started automatically pushing betas to internal mirrors, the situation where we need to overwrite files has come up a couple of times. Both times, the automation complained in some way about the files already existing in releases/, and didn't overwrite them. We need to make it possible to overwrite existing files, because requiring a deletion before re-pushing is unnecessary, and puts extra load on the internal mirrors.

If we're paranoid, perhaps we could require that it be done by "force build" and require a specific property to be present for overwriting.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

There's also a clamp when the internal mirrors rsync from stage, so that only 10 files or dirs are deleted each time. We sync every 10 minutes so deletion is effectively too slow to be useful anyway. The clamp is there to prevent disaster if stage loses the NFS mount with the files on it.

Comment 2

[Rail Aliiev [:rail]]

We could add --force/--override option to the script (which would run rsync with --delete parameter) and use this switch for betas.

Uptake monitoring may not work properly if we push files to the same directory...

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #2)
> We could add --force/--override option to the script (which would run rsync
> with --delete parameter) and use this switch for betas.

Hmmm, I'm not sure we _need_ to use --delete, but it wouldn't hurt. Most importantly, we shouldn't start the rsync if the directory already exists, unless the force flag is present.

> Uptake monitoring may not work properly if we push files to the same
> directory...

Hmm, that would be unfortunate, but not the end of the world if we couldn't make it work.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Having pushed 8.0b6 build2 to the internal mirrors I've kinda changed my tune compared to comment #1. If Sentry was checking hashes we could've known that we had the new files, but since it doesn't I had to fallback to checking file modification times on pv-mirror01/02 (via http). It was a little unnerving not having a strong way to ensure that rsync had replaced properly, in a rush so that running update verify against releasetest wasn't possible.

If we want to make sure bouncer uptake goes to 0 and back up after the new push I suppose we could (hack hack hack!) remove the locale directories that sentry is looking for (plus the other changes discussed here).

An alternate solution is to say no rebuilds after you push anywhere, so we'd have done 8.0b7 like coop expected. Might as well stop creating relbranches for betas too, because they serve no purpose.

Comment 5

[Chris AtLee [:catlee]]

if we don't delete the build beforehand, and build N overwrites build N-1, does rsync work properly?

Comment 6

[[e-mail] (:bhearsum)]

Bulk move of bugs to Release Automation component.

Comment 7

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 8

[[e-mail] (:bhearsum)]

We decided recently that the correct way of overwriting files on mirrors is as documented here: https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Troubleshooting#I_pushed_the_wrong_files_to_releases.2F.2C_help.21

This makes this unnecessary, as we'll be redoing such pushes by hand.