383676 - Scan AUS logs for lost orphan groups, and decide which ones to upgrade, which to abandon

Status: RESOLVED INVALID

(Release Engineering :: General, defect, P3)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

By accident, we discovered that some users were left on a really old version of a beta. They were not upgraded to later beta, or a release version. Fixing this specific case is being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=383666

We need to scan the AUS logs to see if there are any other cases like this, and fix them also.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: grep of AUS logs

each line contains day-of-month, channel-name, #occurrences-that-day.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From offline discussions:

Looking through the AUS logs for the last month, we see lots of unexpected channels. A quick look through the numbers shows approx 20-25% of the FF1.5 beta channel is on some previously unheard of channel! While its still very small ( <1% ) of our overall installbase, the fact that there are any mystery channels at all is a surprise. Attached is a logfile listing them. For completeness, I note that it is possible there is a bug in the script used to parse the AUS logs, but this will need investigation.

For right now, we are ignoring the problem. As part of the major-update-on-beta work, we are only looking at "Firefox/1.5" connections, because those are the only ones "at risk" during the upcoming major-update rollout. Anyone who is not on "Firefox/1.5" on beta will not see the major update beta test work at all. Same for when we do major update on real/live channel.

After the major update project has rolled out, we should investigate the surprise channels in the attached list. In theory most of "Firefox/1.5" will have migrated to 2.0.0.4 by then, so once we exclude "Firefox/2" and "Firefox/3", anything left behind is a mystery channel and is therefore of interest for this bug. If at all possible, we should try to identify what product is in each surprise channels, and figure out how to upgrade them to a supported version. 


Separate discussion arose about adding some sort of URL validation. In AUS? In Firefox? The point was that even if we do a whole bunch of cleanup here, finding and upgrading different "orphaned" channels, there is nothing to stop more people coming in with new surprise channels tomorrow. After a couple of years, we'd be back in the same mess again. We need to think this through some more.

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #1)
> Created an attachment (id=267690) [details]
> grep of AUS logs
> 
> each line contains day-of-month, channel-name, #occurrences-that-day. 

This has a lot of junk in it; I'll get a better one today.

Comment 4

[Daniel Veditz [:dveditz]]

I especially like the "release-foo-cck-bar-cck-foo-bar-baz" channels, for all permutations of foo,bar,baz order.

A lot of the one-off junky ones (not the weird cck release ones) look like snippets of user agent. Could be the log-parsing script, but we should make sure AUS isn't interpreting things the same way.

I was directed to this bug in response to a mail about old versions being downloaded but I think that has to be a completely different problem as presumably all these unknown channel folks are getting no update.rdf and shouldn't be upgrading.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaked summary to line up with Q3 goals.

enumerate orphaned versions of FF/TB
    * decide which to ignore
    * decide which to schedule migrating forward to supported versions in Q4 

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Q3 goal, so upgrading severity

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Missed Q3, carried forward to Q4.

Comment 8

[Samuel Sidler (old account; do not CC)]

John asked me to comment here with some stats and my thoughts on which users we should try to bring forward and which to ignore. This comment is specifically about Firefox releases not about Thunderbird ones. We can look at that later.

First off, I'd like to flat out ignore all releases that have less than 100k daily AUS pings. That includes (data from July 10):
  * 1.5 through 1.5.0.11
  * 2.0.0.1, 2.0.0.2, 2.0.0.3, 2.0.0.4, 2.0.0.5, 2.0.0.7, 2.0.0.8,
    2.0.0.9, 2.0.0.10

Note that the cumulative number of users on 1.5 through 1.5.0.11 is ~142k, well under the number of users on 1.5.0.12 (~800k).

Next, I think we should prioritize with the oldest releases first. Currently, the oldest release with more than 100k users happens to be 2.0, which has just over 300k users. These users are actually *more* at risk than 1.5.0.12 users (depending on how you look at it; Firefox 2 included new security features over 1.5). In order by importance, I think we should try to bring forward all users of the following releases:
  * 2.0 (~300k)
  * 1.5.0.12 (~800k)
  * 2.0.0.11 (~220k)
  * 2.0.0.12 (~230k)
  * 2.0.0.13 (~120k)

I'm alright with ignoring the ~2.4m users on 2.0.0.14 since I believe they'll update in short order. I'm concerned, however, with the number of users on 2.0.0.11 and 2.0.0.12. I can see why we might have a lot more users on 2.0.0.11 since that was the "one week later" release to fix a compatibility bug caused in 2.0.0.10. But why are there so many users on 2.0.0.12?

In any case, my proposal is to bring forward (with priority) at least Firefox 2.0 users first then Firefox 1.5.0.12 users. We can start to worry about the next in the 2.0.0.x line after that.

Thoughts? What sort of timeframe could we be looking at to produce the appropriate bits? What's the strategy for updating 2.0 users, given they receive an update notice every 2.0.0.x release? Is there something we can do specifically for them?

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

A lot has changed since june2007, and I now think this can be closed as INVALID. 

The metrics dashboard lets folks outside RelEng easily see what specific versions have how many users, making the work in this bug obsolete.

Using the metrics dashboard, the product-drivers can now easily see for themselves which specific versions have how many users, and then decide which versions are worth reoffering an upgrade from. 

For each upgrade reoffer, product-drivers should file a separate bug tracking that specific upgrade release work.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

WONTFIX'd bug#518508.

Filed bug#526409 for reprompting orphaned FF2.0.0.20 users.

Comment 11

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.