719465 - Please disable beta updates until we understand why Beta 5 was offered

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Alex Keybl [:akeybl]]

FF10 Beta 5 was just offered to me this morning. Ben and I have confirmed that I'm on the correct beta channel. Until we understand what went wrong, or QA signs off on beta 5, let's disable beta updates.

Comment 1

[Chris Cooper [:coop] (he/him)]

No deep failure here. I ran an accidental pushsnip immediately after the backupsnip this morning, so yes, we did have updates available on the beta channel for about 90 minutes.

bhearsum disabled the updates, so no more update offers should be going out.

Comment 2

[Alex Keybl [:akeybl]]

No problem - just so that we're all on the same page, FF10Beta4 updates should still be offered currently to our beta population (not sure if all of beta was disabled or just beta 5).

Comment 3

[Chris Cooper [:coop] (he/him)]

Going back through the command history, it looks like pushsnip was run *instead* of backupsnip, which makes more sense given the disk artifacts.

I'm running backupsnip now.

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Alex Keybl [:akeybl] from comment #2)
> No problem - just so that we're all on the same page, FF10Beta4 updates
> should still be offered currently to our beta population (not sure if all of
> beta was disabled or just beta 5).

bhearsum's at lunch, but I *think* he only disabled 10.0b5. I'll verify with him when he returns.

Comment 5

[[e-mail] (:bhearsum)]

Yup, I disabled with:
chmod 700 /opt/aus2/incoming/3/Firefox/10.0

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

So that's all the 10.0 betas then.

Comment 8

[[e-mail] (:bhearsum)]

Yeah, I went for the quick fix here.

Coop, since there's no pre-push backup the best way to fix this is to create a null version of the Firefox-10.0b5-build1 snippets, push them, then undo the chmod I did. Something like (WARNING: completely untested):
cd /opt/aus2/snippets/staging
rsync -av Firefox-10.0b5-build1/ Firefox-10.0b5-build1-null/
for i in `find Firefox-10.0b5-build1-null -type f`; do echo -n > $i; done
~/bin/pushsnip Firefox-10.0b5-build1-null
cd /opt/aus2/incoming/3/Firefox
chown 775 10.0

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Ben Hearsum [:bhearsum] from comment #8)
> Yeah, I went for the quick fix here.
> 
> Coop, since there's no pre-push backup the best way to fix this is to create
> a null version of the Firefox-10.0b5-build1 snippets, push them, then undo
> the chmod I did. Something like (WARNING: completely untested):
> cd /opt/aus2/snippets/staging
> rsync -av Firefox-10.0b5-build1/ Firefox-10.0b5-build1-null/
> for i in `find Firefox-10.0b5-build1-null -type f`; do echo -n > $i; done
> ~/bin/pushsnip Firefox-10.0b5-build1-null
> cd /opt/aus2/incoming/3/Firefox
> chown 775 10.0

Could we not accomplish the same thing by running a chmod 700 on just the beta snippet dirs for 10.0b5? Something like:

chmod 700 /opt/aus2/incoming/3/Firefox/10.0/*/20120111092507/*/beta
chmod 775 /opt/aus2/incoming/3/Firefox/10.0

I'm prepping your method JIC, but won't pull the trigger on the pushsnip yet.

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #9) 
> chmod 700 /opt/aus2/incoming/3/Firefox/10.0/*/20120111092507/*/beta

Hrmmm, I guess it would need to include all the build IDs, not just the most recent, but then again I'm not even sure this would work.

Comment 11

[Chris Cooper [:coop] (he/him)]

OK, I've run:

chmod 700 /opt/aus2/incoming/3/Firefox/10.0/*/*/*/beta
chmod 775 /opt/aus2/incoming/3/Firefox/10.0

QA should be able to test on betatest and releasetest again. Just in case it doesn't work, I've prepped bhearsum's method from comment #8 and am ready to pull the trigger on it if necessary.

Comment 12

[Rail Aliiev [:rail]]

Shouldn't it be easier if you push beta4 snippets?

Comment 13

[Chris Cooper [:coop] (he/him)]

(In reply to Rail Aliiev [:rail] from comment #12)
> Shouldn't it be easier if you push beta4 snippets?

Good call. I'm doing that now.

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #13)
> Good call. I'm doing that now.

-bash-3.2$ time ~/bin/pushsnip Firefox-10.0b4-build1

real	75m49.437s
user	0m14.666s
sys	1m5.261s

Comment 15

[u279076]

Thanks guys. Updates on releasetest are all good.

Comment 16

[Chris Cooper [:coop] (he/him)]

Real updates on beta are going out now.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

We ended up with the 10.0b3 build1 dirs still with mode 700, because the pushsnip only overwrote the build2 buildID, and the nightly job to backup and compare with PHX failed. Fixed up with
 chmod 775 /opt/aus2/incoming/3/Firefox/10.0/{Darwin*,Linux*}/20120103192254/*/beta

Comment 18

[Chris Cooper [:coop] (he/him)]

(In reply to Nick Thomas [:nthomas] from comment #17)
> We ended up with the 10.0b3 build1 dirs still with mode 700, because the
> pushsnip only overwrote the build2 buildID, and the nightly job to backup
> and compare with PHX failed. Fixed up with
>  chmod 775
> /opt/aus2/incoming/3/Firefox/10.0/{Darwin*,Linux*}/20120103192254/*/beta

I ran the following this morning just to be sure:

chmod 775 /opt/aus2/incoming/3/Firefox/10.0/*/*/*/beta