530125 - Update generation is failing (no updates on nightly channel on all platforms)

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2b4pre) Gecko/20091119 Namoroka/3.6b4pre ID:20091119033555

Asking releng in #build about that I got the answer that the update generation machine is having some trouble. So no updates will be available for all platforms until this is fixed.

Comment 1

[Armen [:armenzg]]

coop is looking into this.

Added announcement on dev.planning:
> It seems that our update generation machine has got itself into a bad
> state and updates have not been processed since early this morning.
> 
> When the machine gets back into a good state it will take few hours to
> catch up with the work load but en-US updates should be served first
> before other locales.

Comment 2

[Chris Cooper [:coop] (he/him)]

The VM that create partial patches had run out of space. This is a side effect of bug 528457.

I think I have a fix. Testing it now.

Comment 3

[Chris Cooper [:coop] (he/him)]

Attachment: Create the full workdir path if it doesn't exist yet.

Because we force the files early in the process, the full working dir may not exist yet, so we need to create the dir if it's not there.

This bug was causing the partial patch generation script to fail out early. Because of the way we unpack the complete mars to make the partials, this was causing us to skip any cleanup. Repeated attempts (every 5 minutes) quickly filled up the entire disk.

Comment 4

[Chris Cooper [:coop] (he/him)]

Comment on attachment 413659 [details] [diff] [review]
Create the full workdir path if it doesn't exist yet.

Checking in make_incremental_update.sh;
/cvsroot/mozilla/tools/update-packaging/make_incremental_update.sh,v  <--  make_incremental_update.sh
new revision: 1.15; previous revision: 1.14
done

Comment 5

[Chris Cooper [:coop] (he/him)]

Update generation appears to be back on the rails.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Works as expected. Thanks!

Comment 7

[Chris Cooper [:coop] (he/him)]

Added to other repos/branches as well:
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/59125e2f683b
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/a38a293b88ff
http://hg.mozilla.org/mozilla-central/rev/adb526f5ae6d