515364 - reduce idle time of prometheus-vm

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Attachment: stop counting "no updates" when counting how many updates have been processed

Every time we process an update we count until we reach a $max_number_processed_locales so we can start again few mins later and catch any new en-US that was not initially included in the list.

I have some results from our staging machine and we have quite some idle times when we count "no updates" in the count. Every "no update" counted is a real update that is not counted and that will be dealt in the next run of patch-packager.pl.
The worst case scenario is when a $max_number_processed_locales of "no updates" is counted and therefore ~5mins of machine time is wasted.
How many "no updates" can we get a day? As many updates as we generated (which is at 443 of them)

Last week we finished processing all of our updates at around this time:
September 2, 2009 7:04:15 PM GMT-07:00
This week we can end up finishing at around this time:
September 8, 2009 11:56:33 PM GMT-07:00

This is the last fix (that I can think of) that can be fixed until we have the updates being generated on the pool of slaves.

Comment 1

[Chris Cooper [:coop] (he/him)]

Comment on attachment 399472 [details] [diff] [review]
stop counting "no updates" when counting how many updates have been processed

Seems silly to have two stanzas back to back where we check "if (($to ne -1)" - can we merge these two together, please?

Comment 2

[Armen [:armenzg]]

Attachment: stop counting "no updates" when counting how many updates have been processed (v1)

Adding poetry to it ;)

Comment 3

[Chris Cooper [:coop] (he/him)]

Comment on attachment 399482 [details] [diff] [review]
stop counting "no updates" when counting how many updates have been processed (v1)

Don't really need two reviews for this.

Checking in patch-packager.pl;
/mofo/release/patcher/patch-packager.pl,v  <--  patch-packager.pl
new revision: 1.28; previous revision: 1.27
done

Comment 4

[Armen [:armenzg]]

Deployed and running well. Thanks Chris!

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

To restate what the fix does - we call create_partial_patch for the newly created builds too, and there's nowhere for them to go yet. This is meant to not count those calls because they're actually null ops.

I don't think it'll work though, because there's this test at the top of create_partial_patch which will short circuit in that case:
    if ($to eq -1) {
        # if the 'from' build has no 'to' component, it gets no update
        return;
    }
If this wasn't there we'd barf trying to download a non-existent to_url.

How long does it normally take to generate 10 updates ? It'll depend on the platform (ie XUL is much larger on mac), but perhaps we're generating 10 updates in 2 minutes, then sitting for 3 until cron fires again. Or worse, generating for 6 mins, and waiting for another 4.

Comment 6

[Armen [:armenzg]]

Attachment: log of one single run with limiting the count to 10 and just counting real updates

In this log it can be seen that 10 updates get generated and tons of not real updates are processed. This run took 12 minutes and 46 seconds. I can give you more numbers by looking at more logs I have in my email.

Comment 7

[Chris Cooper [:coop] (he/him)]

Any harm in having the script run more frequently? We check for the already-running script at the outset.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Ah, I finally see what you're getting at here. You only want to increment the count when we generate a partial update, and hence spend some time doing some work. Tweaked the comment like this:

-        # If we are actually dealing with a real update rather than a "no update"
+        # Only increment the count and notify when we spend time generating a
+        # partial update. It takes almost zero time to handle the latest build case.

/mofo/release/patcher/patch-packager.pl,v  <--  patch-packager.pl
new revision: 1.30; previous revision: 1.29

As long as we don't hammer aus2-staging we could shorten the interval on the cron job.

Comment 9

[Armen [:armenzg]]

Updated prometheus-vm and looking at the log where I can the first locale going through:
200909101602 7:43:47 Firefox mozilla-central WINNT_x86-msvc de 20090909041701 20090910045239