803181 - Change update background download interval from 10 minutes to 1 minute and update check interval from 24 hours to 12 hours

Status: VERIFIED FIXED

(Firefox :: Settings UI, defect)

Description

[Robert Strong (they/them - no direct email)]

With the CDN we don't have to worry about bandwidth as much and we should change these settings.

Comment 1

[Robert Strong (they/them - no direct email)]

Daniel, just a heads up that we are planning on changing these two values.

Comment 2

[Robert Strong (they/them - no direct email)]

Adding some AUS people to give them a heads up.

Comment 3

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

So there would be no way to distinguish the first request for the day from the second request?

Does changing this timer affect only AUS or does it potentially affect blocklist or anything else like the idle-timer?

Comment 5

[Robert Strong (they/them - no direct email)]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #4)
> So there would be no way to distinguish the first request for the day from
> the second request?
Correct. Similarly, nightly has a 2 hour interval and aurora has an 8 hour interval.

> Does changing this timer affect only AUS or does it potentially affect
> blocklist or anything else like the idle-timer?
Only the app update (AUS) check is affected.

Comment 6

[Jake Maul [:jakem]]

A bit of info on how we're planning to deal with the additional peak/burst bandwidth we expect to get from this change:

1) Testing on beta first, to try and ascertain the actual increase... Naively 10min -> 1min seems very large, but actual usage patterns and billing practices make this somewhat fuzzy. We should have time to back this out (or compromise on the exact values) before it hits Release.

2) A tweak to the initial AUS throttling - instead of 48 hours after release, throttle sooner... 24-36 hours after release. Given the presumably faster uptake this should hopefully still get us the initial population we want to see. The benefit here is around 95th percentile billing... this compresses that initial spike into a small enough window to mitigate most of the billing impact.

3) Ideally, a "phased" approach to un-throttling ~5ish days after release. This would mitigate the second spike we get currently when that happens. With a little bit of thought and scripting we can probably make this automated, to avoid repeated manual AUS pushes.

4) AUS throttling in the case that the bandwidth simply becomes far too much for some reason.


On a side note: It would be good to revisit how the update algorithm works in general. I'd really like to see a smarter algorithm instead of the current fixed chunk size w/ fixed sleep interval. Something that's at least a little adaptive to the user's available bandwidth (ex: measure one chunk, then set an appropriate/conservative KB/s throttle for the rest of the file). This is out of scope for this bug, but I wanted to get it on record as a potential future improvement. I think this is the only way we can have a truly good update experience across a wide range of client bandwidth scenarios... one-size-fits-all is easier, but inherently starves fast clients and chokes slow ones.

Comment 7

[u279076]

Preemptively adding verifyme to this bug so we can check it after it lands in Beta.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Jake Maul [:jakem] from comment #6)
> 2) A tweak to the initial AUS throttling - instead of 48 hours after
> release, throttle sooner... 24-36 hours after release. Given the presumably
> faster uptake this should hopefully still get us the initial population we
> want to see. The benefit here is around 95th percentile billing... this
> compresses that initial spike into a small enough window to mitigate most of
> the billing impact.

The 48 hours has shrunk to ~24 hours based on monitoring ADI, ie 2 days gets you too many people, and even that might be too much. Maybe throttle at 50% for that first 24 hours, then jump to 100, and ramp off later. 

> On a side note: It would be good to revisit how the update algorithm works
> in general. I'd really like to see a smarter algorithm instead of the
> current fixed chunk size w/ fixed sleep interval. Something that's at least
> a little adaptive to the user's available bandwidth 

That sounds great for the user, but possibly harder to manage bandwidth on our side. Not that we have a lot of control now, given that Firefox et al will continue any download they start without checking with AUS again.

Comment 9

[Daniel Einspanjer [:dre] [:deinspanjer]]

(In reply to Robert Strong [:rstrong] (do not email) from comment #5)
> (In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #4)
> > So there would be no way to distinguish the first request for the day from
> > the second request?
> Correct. Similarly, nightly has a 2 hour interval and aurora has an 8 hour
> interval.
> 
> > Does changing this timer affect only AUS or does it potentially affect
> > blocklist or anything else like the idle-timer?
> Only the app update (AUS) check is affected.

Metrics has been specifically avoiding using AUS as a proxy for ADI (active daily installs) for a couple of years now.  When nightly and aurora were changed to ping more frequently, we acknowledged the fact that would prevent us from being able to analyze the offer and uptake trends using data from AUS for those channels.

There have been several occasions within that time that release engineering has requested analysis on the AUS data to look at update offers and migration of install-base.  :nthomas is very familiar with these cases, so I would trust him to consider the impact of this change.

Making this change to beta and release would prevent us from doing this analysis on those channels as well.  We would be able to see how many AUS requests there are for a given channel on a given date, but we would have to divide that number by the check frequency to get an *estimate* at the number of installations involved, and that number would be known to be inaccurate because we would not have any clear indicator how many unique installations are responsible for a given number of requests, only an assumed maximum.

Comment 10

[Robert Strong (they/them - no direct email)]

Comment on attachment 672891 [details] [diff] [review]
patch rev1

Alex knows about these changes and since these changes only affect Beta and Release so requesting aurora and beta so I can land this on Nightly, Aurora, and Beta at the same time.

Comment 11

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-central
https://hg.mozilla.org/mozilla-central/rev/ca6044e25cd2

Pushed to mozilla-aurora
https://hg.mozilla.org/releases/mozilla-aurora/rev/a242353aa94d

Pushed to mozilla-beta
https://hg.mozilla.org/releases/mozilla-beta/rev/29385b6db206

Comment 12

[u279076]

Otilia, can you please make sure this gets assigned to someone at Softvision for verification? We need this verified for Firefox 17.0b3, 18.0a2, and 19.0a1.

To test:
* grab a build from October 20th
* verify app.update.interval=43200
* verify app.update.download.backgroundInterval=60

Thank you

Comment 13

[Virgil Dicu [:virgil] [QA]]

Mozilla/5.0 (X11; Linux i686; rv:19.0) Gecko/19.0 Firefox/19.0
Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/18.0 Firefox/18.0
Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/17.0 Firefox/17.0

Checked Nightly, Aurora and Beta on Ubuntu 12.04, Mac OS 10.7 and Windows 7:

- all builds have "app.update.download.backgroundInterval" set to 60
- "app.update.interval" has different values for every branch: 
*beta: 43200
*aurora: 28800
*nightly: 7200      

Judging by comment 10, I assume these are expected?

Comment 14

[u279076]

I believe so Virgil. Thanks for testing.

Comment 15

[Daniel Veditz [:dveditz]]

(In reply to Jake Maul [:jakem] from comment #6)
> I think [adjusting to client bandwidth] is the only way we can
> have a truly good update experience across a wide range of client
> bandwidth scenarios... one-size-fits-all is easier, but inherently
> starves fast clients and chokes slow ones.

For example, it takes a full minute at dial-up speeds to download one 300K chunk, at which point with this change we'll be ready to grab the next. Using byte-range requests in that scenario is probably not efficient on the server end, and for the user we're no longer a "background" update but seriously downgrading their browsing experience.

Do we have any data on how many users see what sorts of network speeds?

Comment 16

[Robert Kaiser]

Just as a random idea on what comment #15 says, would it make sense if the "interval" would be counted from *finishing* the last chunk to the start of the next?

Comment 17

[Robert Strong (they/them - no direct email)]

(In reply to Robert Kaiser (:[e-mail]) from comment #16)
> Just as a random idea on what comment #15 says, would it make sense if the
> "interval" would be counted from *finishing* the last chunk to the start of
> the next?
Agreed and I assumed that network's nsIIncrementalDownload already did this