Closed Bug 383676 Opened 13 years ago Closed 11 years ago

Scan AUS logs for lost orphan groups, and decide which ones to upgrade, which to abandon

Categories

(Release Engineering :: General, defect, P3, major)

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: joduinn, Unassigned)

References

Details

By accident, we discovered that some users were left on a really old version of a beta. They were not upgraded to later beta, or a release version. Fixing this specific case is being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=383666

We need to scan the AUS logs to see if there are any other cases like this, and fix them also.
Attached file grep of AUS logs
each line contains day-of-month, channel-name, #occurrences-that-day.
From offline discussions:

Looking through the AUS logs for the last month, we see lots of unexpected channels. A quick look through the numbers shows approx 20-25% of the FF1.5 beta channel is on some previously unheard of channel! While its still very small ( <1% ) of our overall installbase, the fact that there are any mystery channels at all is a surprise. Attached is a logfile listing them. For completeness, I note that it is possible there is a bug in the script used to parse the AUS logs, but this will need investigation.

For right now, we are ignoring the problem. As part of the major-update-on-beta work, we are only looking at "Firefox/1.5" connections, because those are the only ones "at risk" during the upcoming major-update rollout. Anyone who is not on "Firefox/1.5" on beta will not see the major update beta test work at all. Same for when we do major update on real/live channel.

After the major update project has rolled out, we should investigate the surprise channels in the attached list. In theory most of "Firefox/1.5" will have migrated to 2.0.0.4 by then, so once we exclude "Firefox/2" and "Firefox/3", anything left behind is a mystery channel and is therefore of interest for this bug. If at all possible, we should try to identify what product is in each surprise channels, and figure out how to upgrade them to a supported version. 


Separate discussion arose about adding some sort of URL validation. In AUS? In Firefox? The point was that even if we do a whole bunch of cleanup here, finding and upgrading different "orphaned" channels, there is nothing to stop more people coming in with new surprise channels tomorrow. After a couple of years, we'd be back in the same mess again. We need to think this through some more.

(In reply to comment #1)
> Created an attachment (id=267690) [details]
> grep of AUS logs
> 
> each line contains day-of-month, channel-name, #occurrences-that-day. 

This has a lot of junk in it; I'll get a better one today.
I especially like the "release-foo-cck-bar-cck-foo-bar-baz" channels, for all permutations of foo,bar,baz order.

A lot of the one-off junky ones (not the weird cck release ones) look like snippets of user agent. Could be the log-parsing script, but we should make sure AUS isn't interpreting things the same way.

I was directed to this bug in response to a mail about old versions being downloaded but I think that has to be a completely different problem as presumably all these unknown channel folks are getting no update.rdf and shouldn't be upgrading.
Priority: -- → P3
Tweaked summary to line up with Q3 goals.

enumerate orphaned versions of FF/TB
    * decide which to ignore
    * decide which to schedule migrating forward to supported versions in Q4 
Blocks: 383666
No longer depends on: 383666
Summary: Scan AUS logs for lost orphan groups, and get them upgraded to supported/stable version → Scan AUS logs for lost orphan groups, and decide which ones to upgrade, which to abandon
Q3 goal, so upgrading severity
Severity: normal → major
Missed Q3, carried forward to Q4.
Assignee: build → nobody
QA Contact: mozpreed → build
Component: Release Engineering → Release Engineering: Future
QA Contact: build → release
John asked me to comment here with some stats and my thoughts on which users we should try to bring forward and which to ignore. This comment is specifically about Firefox releases not about Thunderbird ones. We can look at that later.

First off, I'd like to flat out ignore all releases that have less than 100k daily AUS pings. That includes (data from July 10):
  * 1.5 through 1.5.0.11
  * 2.0.0.1, 2.0.0.2, 2.0.0.3, 2.0.0.4, 2.0.0.5, 2.0.0.7, 2.0.0.8,
    2.0.0.9, 2.0.0.10

Note that the cumulative number of users on 1.5 through 1.5.0.11 is ~142k, well under the number of users on 1.5.0.12 (~800k).

Next, I think we should prioritize with the oldest releases first. Currently, the oldest release with more than 100k users happens to be 2.0, which has just over 300k users. These users are actually *more* at risk than 1.5.0.12 users (depending on how you look at it; Firefox 2 included new security features over 1.5). In order by importance, I think we should try to bring forward all users of the following releases:
  * 2.0 (~300k)
  * 1.5.0.12 (~800k)
  * 2.0.0.11 (~220k)
  * 2.0.0.12 (~230k)
  * 2.0.0.13 (~120k)

I'm alright with ignoring the ~2.4m users on 2.0.0.14 since I believe they'll update in short order. I'm concerned, however, with the number of users on 2.0.0.11 and 2.0.0.12. I can see why we might have a lot more users on 2.0.0.11 since that was the "one week later" release to fix a compatibility bug caused in 2.0.0.10. But why are there so many users on 2.0.0.12?

In any case, my proposal is to bring forward (with priority) at least Firefox 2.0 users first then Firefox 1.5.0.12 users. We can start to worry about the next in the 2.0.0.x line after that.

Thoughts? What sort of timeframe could we be looking at to produce the appropriate bits? What's the strategy for updating 2.0 users, given they receive an update notice every 2.0.0.x release? Is there something we can do specifically for them?
A lot has changed since june2007, and I now think this can be closed as INVALID. 

The metrics dashboard lets folks outside RelEng easily see what specific versions have how many users, making the work in this bug obsolete.

Using the metrics dashboard, the product-drivers can now easily see for themselves which specific versions have how many users, and then decide which versions are worth reoffering an upgrade from. 

For each upgrade reoffer, product-drivers should file a separate bug tracking that specific upgrade release work.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INVALID
WONTFIX'd bug#518508.

Filed bug#526409 for reprompting orphaned FF2.0.0.20 users.
Moving closed Future bugs into Release Engineering in preparation for removing the Future component.
Component: Release Engineering: Future → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.