453354 - backupsnip run from bootstrap timing out after major update

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

The 2.0->3.0 MU added about 150,000 snippets to AUS. Most of these are null partner snippets, 3/4 of which are on releasetest-*, beta-*, and betatest-* channels. AUIU, those channels are not used by any builds.

Because of all of these snippets backupsnip is now taking significantly longer to complete. At this point, I've seen it run for over 90 minutes without completing. Shortly after that the ssh process from production-prometheus-vm timed out, causing it to fail.

We can increase that timeout to workaround that but we should consider getting rid of these useless partner snippets.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Definitely. Might also be able to trim locales out from the release-* case too, as many only ship a limited list of languages (much higher time cost to do this though).

Comment 2

[[e-mail] (:bhearsum)]

Attachment: list of null snippets to be removed

Here's a list of the null snippets I intend to remove. If someone (Nick? Kev?) could have a scan of it that would be great. I did the following to verify there are no legitimate null or MU snippets being removed:
[bhearsum@dm-ausstage01 ~]$ egrep '^.*/beta$' partner-remove.log 
[bhearsum@dm-ausstage01 ~]$ egrep '^.*/release$' partner-remove.log 
[bhearsum@dm-ausstage01 ~]$ egrep '^.*/releasetest$' partner-remove.log 
[bhearsum@dm-ausstage01 ~]$ egrep '^.*/betatest$' partner-remove.log 
[bhearsum@dm-ausstage01 ~]$ egrep '^.*/release-.*$' partner-remove.log 
[bhearsum@dm-ausstage01 ~]$ 

But, someone else looking over it would be good too.

This will remove nearly 50,000 snippets - about 1/3 of what we pushed during the MU. Hopefully, backupsnip will work after this.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

imho, we just increase the timeout, and continue to backup the null snippets

We do these backups in case we accidentally push bogus snippets and need to repair the update channels snippets by extracting the backup tar file. In that scenario, we dont want to forget/lose those null updates....otherwise, if we ever restore from backup, we'll accidentally upgrade partner builds to vanilla builds.

Comment 4

[[e-mail] (:bhearsum)]

John, this a list of *test* and *beta* null snippets. We do *not* have partner builds on these channels - the only reason they were generated in the first place is to allow QA to test fallback channels on partner builds. AFAICT, there is no reason to keep them around. We are *not* getting rid of the null partner snippets on the release-* channels. I don't understand how restoring from a backup will MU partner builds.

Comment 5

[Kev Needham [:kev]]

beta, betatest, and releasetest channels are, as far as I know, QA-derived only, and both can (and should) be removed as there are no real users, only QA. only concern I have would be the yahoo japan builds, which may or may not have real beta users on them. cc'ing kohei to ask for comment.

in addition to purging snippets, we should probably purge from aus and metrics.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #5)
> beta, betatest, and releasetest channels are, as far as I know, QA-derived
> only, and both can (and should) be removed as there are no real users, only QA.
> only concern I have would be the yahoo japan builds, which may or may not have
> real beta users on them. cc'ing kohei to ask for comment.

betatest and releasetest channels are for QA only. Safe to purge.
beta is public-facing, not only for QA. We cannot purge from beta channel until we've verified there are no partner builds on beta channel.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> (In reply to comment #5)
> > beta, betatest, and releasetest channels are, as far as I know, QA-derived
> > only, and both can (and should) be removed as there are no real users, only QA.
> > only concern I have would be the yahoo japan builds, which may or may not have
> > real beta users on them. cc'ing kohei to ask for comment.
> 
> betatest and releasetest channels are for QA only. Safe to purge.
> beta is public-facing, not only for QA. We cannot purge from beta channel until
> we've verified there are no partner builds on beta channel.

We are talking about fake beta channels for partner builds. Eg, beta-cck-google, NOT the literal 'beta' channel.

Comment 8

[Kev Needham [:kev]]

sorry, should have clarified better. what bhearsum said. I was speaking more about beta-* vs "beta".

Comment 9

[Kohei Yoshino [:kohei]]

(In reply to comment #5)
> only concern I have would be the yahoo japan builds, which may or may not have
> real beta users on them. cc'ing kohei to ask for comment.

We have never used the beta channels for Yahoo! JAPAN builds.

Comment 10

[[e-mail] (:bhearsum)]

Alright, given kev's and kohei's comments', I'm going to go ahead and wipe these snippets out.

Comment 11

[[e-mail] (:bhearsum)]

I backed up all of these snippets in an outside directory, just in case:
rsync -a --files-from=/home/bhearsum/partner-remove.log /opt/aus2/incoming ~/useless-partner-snippets-backup/
And then I used the originally find statement to remove the snippets from the live directory:
find 3/Firefox/1.5.0.12 3/Firefox/2.0.0.16  -mindepth 4 -maxdepth 4 -type d | egrep -e '^.*/betatest-.+' -e '^.*/releasetest-.+' -e '^.*/beta-.+' | xargs rm -r

I also just remembered that we have null snippets for 2.0.0.14, too. To remove them, I did the following:
# generate the list
find 3/Firefox/2.0.0.14 -mindepth 4 -maxdepth 4 -type d | egrep -e '^.*/betatest-.+' -e '^.*/releasetest-.+' -e '^.*/beta-.+' 2>&1 ~/null-partner-2.0.0.14.log
# backup
rsync -a --files-from=/home/cltbld/null-partner-2.0.0.14.log /opt/aus2/incoming ~/useless-partner-snippets-backup/
# delete
find 3/Firefox/2.0.0.14 -mindepth 4 -maxdepth 4 -type d | egrep -e '^.*/betatest-.+' -e '^.*/releasetest-.+' -e '^.*/beta-.+' | xargs rm -r

After all of this I did a spot check on the live snippet directories and they looked fine. They have snippets for 'release', 'beta', 'betatest', 'releasetest', 'major', 'majortest', and the all of the 'release-*' partner channels.

Interestingly, I could not find null snippets for 'default', 'nightly-cck-google', or 'default-cck-google' *before* or *after* this removal. I could've sworn I had snippets pushed for these, but maybe not. 'default' has approx. 30,000 users on it, so I suspect we would've heard from them if they got a MU. Given that there's no fallback for it (afaict) we're probably okay there.


I also spot checked three builds: ebay 2.0.0.14 en-GB and t-online 2.0.0.14 de and acer 2.0.0.16 en-US - none were not offered a major update.


With this done, we've eliminated nearly 80,000 snippets which I believe will get backupsnip working again.