Closed Bug 637833 Opened 9 years ago Closed 9 years ago

Split up final verification

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: rail)

References

Details

(Whiteboard: [releases][automation])

Attachments

(1 file)

I am not sure if by platform or by a number of locales but it takes too much time right now if this builder has any value on finishing before we release.

Feel free to close this or modify if I don't understand this correctly.
I started a couple of jobs and each one took 20 hours and 17 hours.
Could you look up the slavenames for those jobs ? Did they hit VMs ? How does the elapsed time compare to previous releases for the same branch ?

It's possible that something in the chain is much slower now. We
 * query AUS for an update
 * do a HEAD request on the mar url (download.m.o)
 * get redirected to a mirror and do a HEAD there

We know that AUS changed to be MPT and PHX prior to the release, previously it was just MPT. The PHX webheads may be slower than MPT, and DNS is now 3 steps to get to the colo-specific load balancer:
  aus2.mozilla.org 
   -> aus2.3crowd.mozilla.net  (CNAME)
   -> aus01.zlb.phx.mozilla.net or aus2.acelb.sj.mozilla.com (CNAME)
   -> IP
where previously we had
  aus2.m.o 
  -> aus2-mozilla-org.geo.mozilla.com (CNAME) 
  -> IP

There's lots of other candidates - download.m.o may have been slow, or the mirrors themselves, or ESX buried by load.
The slaves for the first two were moz2-linux-slave26 and moz2-linux-slave19.

There is no previous logs for that builders on pm01 and pm03.

The run of today on IX machine took 5 hrs, 13 mins, 21 secs.
OS: Mac OS X → All
Priority: -- → P5
Hardware: x86 → All
Whiteboard: [releases][automation]
Assignee: nobody → rail
Priority: P5 → P2
This patch splits final verification per platform. ReleaseFinalVerification class stays backward compatible.

The builders tested in staging and worked as expected. master-dump also shows expected changes for the list of builders triggered by mirror uptake monitoring poller.
Attachment #521323 - Flags: review?(catlee)
Attachment #521323 - Flags: review?(catlee) → review+
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.