596841 - Track slow mirrors for update tests, report on the slow ones and blacklist them for subsequent runs/locales

Status: RESOLVED WONTFIX

(Release Engineering :: General, enhancement, P4)

Description

[LegNeato]

We occasionally run into a situation where update tests take forever because certain mirrors are slow. We should be able to set a threshold and have the update test:

1) Log when a mirror hits the slow threshold, for later investigation
2) Blacklist the mirror for the remainder of the run if at all possible

Comment 1

[LegNeato]

The behavior should be configurable, so if you specifically want to repeat / test the mirrors w/o blacklisting you can.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so this bug is more than only Mozmill. With the fix on bug 569477 we will see the download url in our report but blacklisting is a part of releng. It's not something we can do on our side.

Reopen and => Releng

Comment 4

[[e-mail] (:bhearsum)]

Henrik,

My understand of this is that Christian wants Mozmill to ignore any slow mirrors for the remaining of that test run -- not blacklisting of the mirror overall. Christian, do I have that right?

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to comment #3)
> Ok, so this bug is more than only Mozmill. With the fix on bug 569477 we will
> see the download url in our report but blacklisting is a part of releng. It's
> not something we can do on our side.

Doesn't IT (specifically justdave in my experience) usually do the pulling of mirrors?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> (In reply to comment #3)
> > Ok, so this bug is more than only Mozmill. With the fix on bug 569477 we will
> > see the download url in our report but blacklisting is a part of releng. It's
> > not something we can do on our side.
> 
> Doesn't IT (specifically justdave in my experience) usually do the pulling of
> mirrors?

Yes. Technically, we're able to as well, though.

Comment 7

[LegNeato]

(In reply to comment #4)
> Henrik,
> 
> My understand of this is that Christian wants Mozmill to ignore any slow
> mirrors for the remaining of that test run -- not blacklisting of the mirror
> overall. Christian, do I have that right?

Yep. My understanding is that the update verification scripts download and apply the update one-by-one on every platform for a subset of localizations. It seems silly if the 1st one hits a slow mirror to allow later ones to also use that slow mirror. We could then go back and spotcheck the blacklist, making sure any super-slow mirrors with sustained issues are pulled from the queue or their admins notified.

The bug was filed because for 2 releases now update testing took longer than expected due to slow mirrors. Of course, there are other things to do to speed up this particular process/testing...

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's not in our hands which mirrors we get. We are simply doing a ui focused update process. There is no way for us to select a specific mirror. Given that, we could end-up in an infinite loop with restarting Firefox until we get a mirror which is not marked as slow. In cases like those we should simply call RelEng and start the test again once the slow mirror has been pulled off from the list of available ones.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With bug 569477 fixed, we now include the url of mirrors in the reports. I will take care of displaying this data in the initial result view on our dashboard (bug 569477).

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Even with bug 616044 and bug 607800 fixed, we now log the patch size and the duration of the download. That way we can calculate the transfer speed. Everything will bubble up on our dashboard in the next two weeks.

Comment 11

[Chris Cooper [:coop] (he/him)]

For pre-release testing (i.e. test channels), RelEng is switching to using internal mirrors: bug 613620. 

Our final verification scripts are smart enough to retry now, so we don't tend to get hung up on slow mirrors.