1025048 - Investigate why 20% of requests for 29.0.1 complete updates come from versions where partials are available

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Chris AtLee [:catlee]]

I've been looking into update logs to understand what types of updates contribute to our overall CDN bandwidth, and to understand what effect serving partials to more users would have.

I noticed a seemingly high percentage of users requesting complete updates that should be using partial updates.

My data comes from hive with this query:
insert overwrite local directory '/home/catlee/download_counts' select ds, ua_major, ua_minor, request_url, count(1) as c from v2_anonymous_logs where domain='download.mozilla.org' and ds >= '2014-05-01' and ds <= '2014-05-31' and request_url like '/?product=firefox-29.0.1%' and ua_family = 'Firefox' group by ds, ua_major, ua_minor, request_url;

I then post-processed this data:
- Use HEAD requests to figure out the size of each URL requested
- Count how many requests and how many bytes were for complete/partial updates

For 29.0.1 we have partial updates available to 27.0.1, 28.0, 29.0 users. So I also counted how many requests for partial updates came from users on those versions. Here's a summary of these results:

Complete updates:         23,417,672,547,025,403b (58.12%) n=642,694,681 (41.65%)
Partial updates:          16,871,014,243,483,169b (41.88%) n=900,354,424 (58.35%)
Redundant completes:       4,781,919,026,860,840b (20.42%) n=130,647,218 (20.33%)

This is showing that about 20% of requests for completes (by bytes or request count) were from versions where we had partial updates available. Breaking this down by platform shows that OSX builds are a bit more problematic in terms of % requesting completes that shouldn't (29.9%), but the overall impact of that is lower because OSX only makes up 1.6% of requests for complete updates. 98.1% of requests for complete updates are for windows.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

As mentioned in last week's channel meeting - if it's possible to go back a couple of releases to see if this is approximately the same rate with them (or a recent regression in update request code) that would be useful data to have.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Chris, do we know if the requests for a complete update were alone, or followed by a download of a partial mar first? I wonder if some of the users ended-up with a broken mar file, which couldn't be installed, and therefore Firefox requested the full mar file. Something as what we also test with Mozmill.

Comment 3

[Chris AtLee [:catlee]]

(In reply to Henrik Skupin (:whimboo) [away 07/19 - 08/01] from comment #2)
> Chris, do we know if the requests for a complete update were alone, or
> followed by a download of a partial mar first? I wonder if some of the users
> ended-up with a broken mar file, which couldn't be installed, and therefore
> Firefox requested the full mar file. Something as what we also test with
> Mozmill.

No, it's pretty hard to tell from the data *why* the complete update was requested. I can only assume it attempted the partial and failed back to the complete for versions where we offered a partial, but perhaps there are other cases where it's doing this.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So we don't have IPs for those requests, which we could correlate? If users are falling back to full update mar files, we should have a request to the partial mar a couple of seconds/minutes earlier. Maybe that would help if possible?

Comment 5

[Robert Strong (they/them - no direct email)]

While analyzing bug 1284915 using telemetry data it became apparent that at least for the majority of these cases that this is due to a networking error causing application update to try downloading the complete mar file. See bug 1284915 comment #55 for bugs to help address this.