396899 - Nightly update generation is slow

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[u279076]

I was updated to Minefield 20070919 on Sept 19th,but today I cannot seem to get an update to 20070902.  I checked the ftp and there are files on there for 20070920. However, I cannot get an update.

FYI.  I am using Linux Trunk builds.

Steps to reproduce:
1. Start Minefield
2. Click Help>Check For Updates

Actual Results:
Software Updater appears, scans for an update and does not find one.  A message is displayed: "No Updates Found: There are no new updates available. Minefield may check periodically for updates."

Expected Results:
There should be a partial update available for 20070919 -> 20070920 and a complete update available for <20070919 -> 20070920.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I think it has been slowly getting worse over the last few days - you tend to get no updates at all, then they all come through at once.

By watching the partial-generator log in /tmp, the issue seems to be that the sync from stage to ftp is slow, so that builds aren't available for the patch generator to pull, and it exits before trying to build other updates. We also seem to get builds that were published very recently getting processed earlier than those that are older.

We can
* fix up patcher so that it copes when one build is not available (see also bug 
396314)
* speed up the sync between stage and ftp by reducing the module size, and checking that everything is ok on ftp.m.o

I can probably do some of the first part of point 2; aravind, can you do the second part ?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I found about 145GB of older nightly files that we hadn't been transferring over to archive, so they'd just been accumulating. The dirs were

    calendar/lightning/nightly/200{5,6,7}    
    calendar/sunbird/nightly/200{5,6,7}
    seamonkey/nightly/contrib/

They were safe to remove from ftp.m.o because I've very recently synced them over to netapp, as part of the stage migration project. ftp.m.o is taking it's time picking up the changes though.

Comment 3

[Aravind Gottipati [:aravind]]

Nick:  Which sync are you talking about?  The only thing being synced between stage(surf) and ftp(manna) is stage.mozilla.org::mozilla-all/ and that seems to be fairly up to date (going by the zz/marker.txt file).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Sorry, I missed your reply here (forgot to CC myself). 

I did mean the sync of mozilla-all, which has been having trouble
  https://nagios.mozilla.org/ftplag/10.2.74.11-ftp-weekly.png
(guest/guest).
Obviously it's worse at the moment, but the sync time was up a bit on the 20th when this was first filed - from 15-20 minutes to twice that, IIRC.

It's a problem for update generation because complete mar's are retrieved from ftp.m.o rather than stage.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The slow sync problem is now a non-issue, thanks to changes in the setup of the ftp servers. However, updates are now disabled because we need to cope with the new layout of directories on ftp.m.o - we're tracking a fix for that in bug 394069.