Split up final verification

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
8 years ago
5 years ago

People

(Reporter: armenzg, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [releases][automation])

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
I am not sure if by platform or by a number of locales but it takes too much time right now if this builder has any value on finishing before we release.

Feel free to close this or modify if I don't understand this correctly.
(Reporter)

Comment 1

8 years ago
I started a couple of jobs and each one took 20 hours and 17 hours.
Could you look up the slavenames for those jobs ? Did they hit VMs ? How does the elapsed time compare to previous releases for the same branch ?

It's possible that something in the chain is much slower now. We
 * query AUS for an update
 * do a HEAD request on the mar url (download.m.o)
 * get redirected to a mirror and do a HEAD there

We know that AUS changed to be MPT and PHX prior to the release, previously it was just MPT. The PHX webheads may be slower than MPT, and DNS is now 3 steps to get to the colo-specific load balancer:
  aus2.mozilla.org 
   -> aus2.3crowd.mozilla.net  (CNAME)
   -> aus01.zlb.phx.mozilla.net or aus2.acelb.sj.mozilla.com (CNAME)
   -> IP
where previously we had
  aus2.m.o 
  -> aus2-mozilla-org.geo.mozilla.com (CNAME) 
  -> IP

There's lots of other candidates - download.m.o may have been slow, or the mirrors themselves, or ESX buried by load.
(Reporter)

Comment 3

8 years ago
The slaves for the first two were moz2-linux-slave26 and moz2-linux-slave19.

There is no previous logs for that builders on pm01 and pm03.

The run of today on IX machine took 5 hrs, 13 mins, 21 secs.
OS: Mac OS X → All
Priority: -- → P5
Hardware: x86 → All
Whiteboard: [releases][automation]
(Assignee)

Updated

8 years ago
Assignee: nobody → rail
Priority: P5 → P2
(Assignee)

Comment 4

8 years ago
Created attachment 521323 [details] [diff] [review]
split per platform

This patch splits final verification per platform. ReleaseFinalVerification class stays backward compatible.

The builders tested in staging and worked as expected. master-dump also shows expected changes for the list of builders triggered by mirror uptake monitoring poller.
Attachment #521323 - Flags: review?(catlee)
Attachment #521323 - Flags: review?(catlee) → review+
(Assignee)

Updated

8 years ago
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.