Getting telemetry data for updates

RESOLVED WORKSFORME

Status

()

RESOLVED WORKSFORME
7 years ago
4 years ago

People

(Reporter: Ehsan, Unassigned)

Tracking

({meta, sec-want})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [telemetry-needed])

(Reporter)

Description

7 years ago
JP and catlee told me that Johnath and nthomas are looking to figure out if they can get some useful data inside the telemetry pings to figure out why some users get left stranded on a version.  I recently added the update status to the telemetry pings to figure out if the updates are applied or not.

What are the other things which people want to know to answer this question?  Once we know what data we need, we can get it added to telemetry pings easily.
Off the top of my head
The value of the prefs that control how to update.
Whether incompatible add-ons were found.
Whether the ui offering the update was displayed.
Yeah, to be clear, I haven't done any good thinking here to date, I just expressed an interest in understanding it further. It's something Robert and I have talked about in the past, too; I'll defer to him on which telemetry probes would be most helpful there.

... which bugzilla tells me he has done in the time it took me to write this comment.
Some service specific ones: (Which doesn't help with old stuck update problems, but will help us detect new stuck update problems since we're landed)

1. Success with service vs. success updater.exe without service installed
2. How many people have the service installed and disabled vs how many people have the service installed and enabled (app.update.service.enabled)
3. What is the value of people's app.update.service.errors.
Not relating to telemetry pings, but along the lines of the goal of figuring out why some users get left stranded on a version, we also have this bug:
Bug 711393 - Add crash reporting for updater and maintenance service
Also putting more resources on investigating individual instances of not updating via specific support cases might help this goal as well.  I would be more than willing to set up a gotomeeting or logmein with a willing user to investigate an update problem we can't figure out for example.
My motivations may not map super well onto telemetry, but here goes
* understand why some people never update to the latest version, as the number of people left behind on old releases is worrying (3.6 or rapid, doesn't matter). Are they stuck ? Why ?
* understand why we still get many requests for update files from previous releases, for many days after a new release comes out. Should the client be aborting current downloads if a newer update is found ? How do the metrics for requests to download.m.o map to an actual amount of users and mirror traffic?

So that might mean
 * do they query for updates at all
 * do they find an update. If there are errors, what type ?
 * how many times do they make requests to download.m.o (rather than the mirror that'll redirect them to), compared to the session length
 * how long does it take to download the mar file 
 * do they fail to download the update file (ie is the update code robust to flaky mirrors or can it get hung)
 * do they fail to apply them (permissions etc, or just not restart at all)
 * how many times have they failed to apply an update, or how many update attempts does it take to make it stick

I've not played with the telemetry system at all so I don't know how answerable those are in a once-a-day payload, particularly when some of those processes could well be multi-day.
(Reporter)

Comment 7

7 years ago
Telemetry data are mostly histograms, but I hear that these days it's possible to add other types of data as well.  Taras, Vladan?
Depends on: 718348
(In reply to Nick Thomas [:nthomas] from comment #6)
> My motivations may not map super well onto telemetry, but here goes
> * understand why some people never update to the latest version, as the
> number of people left behind on old releases is worrying (3.6 or rapid,
> doesn't matter). Are they stuck ? Why ?

Just to ensure we're all on the same page, Telemetry was added in Firefox 7. We can easily add probes for upcoming releases to track this type of information going forward. AFAIK, getting this information for 3.6 would require a Telemetry backport and any release of Firefox other than the current would likely introduce new problems as we would need a way to deploy the probes on the older systems without helping the users update to current. If we're to pursue this it is likely a better use of our time to simply help them update.

I think that lots of the questions that have been asked here are good ones. There is real value in understanding why users get left behind so that we can make improvements to the update system (if the problem is in fact with the update system).
I didn't mean to imply backporting telemetry there, and agree it's possible the lack of updating is a PEBKAC rather than software.
(In reply to Nick Thomas [:nthomas] from comment #9)
> I didn't mean to imply backporting telemetry there, 

Yeah. I didn't really think you were suggesting that. Just wanted to make sure everyone reading this bug was clear on when Telemetry was first introduced in Firefox.

>and agree it's possible
> the lack of updating is a PEBKAC rather than software.

I do think that there is value in understanding why users get 'stuck' on a certain release. (It will be valuable to know if users get stuck on Firefox 11, 12, 13,...) It will also be interesting to see what affect (if any) silent updates have on this situation.

Comment 11

7 years ago
Do you currently know (from update pings, telemetry or otherwise) when a user cannot update because they lack the necessary permissions to do so?
No and we will be getting this info as well as many other "under the cover" pieces of info in this bug.
Depends on: 731901
Whiteboard: [telemetry-needed]
Depends on: 763011
Depends on: 763012
Depends on: 763016
Depends on: 763017
Depends on: 763018
Depends on: 763022
Depends on: 763025
Depends on: 763243
Depends on: 763244
Depends on: 763245
Depends on: 763246
OS: Mac OS X → All
Depends on: 765304
Depends on: 767471

Updated

6 years ago
Keywords: meta, sec-want
Depends on: 774395
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.