637833 - Split up final verification

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

I am not sure if by platform or by a number of locales but it takes too much time right now if this builder has any value on finishing before we release.

Feel free to close this or modify if I don't understand this correctly.

Comment 1

[Armen [:armenzg]]

I started a couple of jobs and each one took 20 hours and 17 hours.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Could you look up the slavenames for those jobs ? Did they hit VMs ? How does the elapsed time compare to previous releases for the same branch ?

It's possible that something in the chain is much slower now. We
 * query AUS for an update
 * do a HEAD request on the mar url (download.m.o)
 * get redirected to a mirror and do a HEAD there

We know that AUS changed to be MPT and PHX prior to the release, previously it was just MPT. The PHX webheads may be slower than MPT, and DNS is now 3 steps to get to the colo-specific load balancer:
  aus2.mozilla.org 
   -> aus2.3crowd.mozilla.net  (CNAME)
   -> aus01.zlb.phx.mozilla.net or aus2.acelb.sj.mozilla.com (CNAME)
   -> IP
where previously we had
  aus2.m.o 
  -> aus2-mozilla-org.geo.mozilla.com (CNAME) 
  -> IP

There's lots of other candidates - download.m.o may have been slow, or the mirrors themselves, or ESX buried by load.

Comment 3

[Armen [:armenzg]]

The slaves for the first two were moz2-linux-slave26 and moz2-linux-slave19.

There is no previous logs for that builders on pm01 and pm03.

The run of today on IX machine took 5 hrs, 13 mins, 21 secs.

Comment 4

[Rail Aliiev [:rail]]

Attachment: split per platform

This patch splits final verification per platform. ReleaseFinalVerification class stays backward compatible.

The builders tested in staging and worked as expected. master-dump also shows expected changes for the list of builders triggered by mirror uptake monitoring poller.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 521323 [details] [diff] [review]
split per platform

http://hg.mozilla.org/build/buildbotcustom/rev/76a6cfcddf47