1636956 - Stop regular update checks and downloads of various services

Status: NEW

(Testing :: Raptor, enhancement, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As started to discuss on bug 1626604 comment 10, we should stop running certain update checks, which could cause additional downloads while the test is running.

Here some of the update timer prefs I found and which might need a timestamp as best +7 days or such. It would allow us to use some of those for regression testing).

browser.safebrowsing.provider.google.nextupdatetime
browser.safebrowsing.provider.google4.nextupdatetime
browser.safebrowsing.provider.mozilla.nextupdatetime

Services to disable completely would be:

app.update.disabledForTesting
browser.search.update
extensions.update.enabled

Not sure how to handle this one:

services.settings.server

Tarek, is it something you could have a look at?

Comment 1

[Tarek Ziadé (:tarek)]

Before we disable some of the update features in the conditioned profile builder, I would like to understand why we want to move away from a realistic profile to a profile with disabled features. It sounds like we'll have less realistic conditions.

The whole point of conditioned profiling is to let those updates happen when the profile is built and updated, so they don't happen during the tests. If you have updates occuring during your raptor test, it means you have to deactivate something in raptor's side.

So I don't understand why you want to do this in the pre-conditioned profile phase. is this just to make the tarball smaller like you said in https://bugzilla.mozilla.org/show_bug.cgi?id=1626604#c12 ?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sorry when it was unclear. This is not about stopping certain web-specific features, but only delaying further updates of the content so that it doesn't conflict with page load tests. That means the data for safe browsing will be downloaded, and afterward the timer prefs will be updated.

Also update checks for Firefox, search engines, or extensions don't seem to be needed at all and do not affect any web feature. Those would only add noise. So that's why I proposed to disable these.

Is Raptor not the only harness which makes use of those conditions profiles? Maybe there is something I don't know yet. If that is the case should we set all the timer prefs in Raptor instead?

Comment 3

[Tarek Ziadé (:tarek)]

Is Raptor not the only harness which makes use of those conditions profiles?

we're working on sync tests (that use the full profile) so having services enabled is pretty much mandatory,
and more complex scenarios.

delaying further updates of the content so that it doesn't conflict with page load tests
Maybe there is something I don't know yet. If that is the case should we set all the timer prefs in Raptor instead?

Yeah it sounds like you need to deactivate more stuff right before the page load starts, to set up an isolated
test environment with no call to any service whatsoever - if that's what we want to test.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Greg and Dave, any feedback from your side regarding this bug? As it looks like we would have to move it over to the Raptor component.

Comment 5

[Greg Mierzwinski [:sparky]]

I agree, it sounds like something we should move to the Raptor component.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Yes, these should probably go into testing/profiles/raptor or testing/profiles/perf.

Comment 7

[Greg Mierzwinski [:sparky]]

:acreskey, do you have any thoughts on this bug?

Comment 8

[Andrew Creskey [:acreskey]]

Just trying to fully understand the context.

In general I'm in favor of tests that are as realistic as possible.

So, in an ideal world:
-The conditioned profile is updated (and it fetches the updated safebrowsing lists, extension block lists, does the update check etc).
-We start the pageload test with this profile and there's no update to do

But it sounds like the safebrowsing downloads and update checks are still happening even though the profile was conditioned that same day.

I know that we want to avoid adding a browser settle delay since it increases test duration.

I'm most curious about whether or not we always see safebrowsing downloads with conditioned profiles.
That sounds like it's a bug?

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It looks like bug 1456705 may be related.