1052519 - Capture application update's update.log in update hotfix forensics

Status: VERIFIED FIXED

(Firefox :: General, defect)

Description

[Gregory Szorc [:gps]]

The Firefox update hotfix currently reports the raw log from the hotfix itself and the install.log from the installer it runs. The hotfix does *not* report the update.log from the Firefox updater.

I talked to rstrong and this is apparently an oversight. We should collect the (sanitized) update.log as well.

Comment 1

[Gregory Szorc [:gps]]

rstrong: what files should we be looking for and uploading?

How should we QA this? I tried installing 30.0 and going to the about page to trigger an update, but I see no update.log after restart in either the install dir or my profile dir.

Comment 2

[Marco Mucci [:MarcoM]]

Hi Gregory, can you mark this bug as [qa+] or [qa-] for verification.

Comment 3

[Robert Strong (they/them - no direct email)]

Per bug 928173 comment #1
In the update directory, active-update.xml, updates.xml, updates/backup-update.log, updates/last-update.log, and updates/0/update.log if they exist for the updater.

You can get the root directory using "UpdRootD". The .log files will need to have paths munged.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Kamil is this something you might verify once we have a fix? If so, can you assign yourself as QA contact?

Comment 5

[Gregory Szorc [:gps]]

Attachment: Capture updater state as part of forensic upload

This is what I have so far. I'm doing some local testing to ensure this
works. I'm having a difficult time capturing an actual log file (XML
files come through just fine). I /think/ the updater might be purging
files or switching update directories after the hotfix runs.

Comment 6

[Gregory Szorc [:gps]]

rstrong: see comment on attachment. Do I need to capture the updater files before the hotfix's installer runs or something?

Comment 7

[Robert Strong (they/them - no direct email)]

Yes, the logs must be captures prior to running the installer. Can you gather these logs for the install failure case as well?

Comment 8

[Gregory Szorc [:gps]]

I /think/ what you are asking for is:

1) active-update.xml, updates.xml, updates/backup-update.log, updates/last-update.log, and updates/0/update.log from immediately before we run the installer
2) install.log from our installation attempt
3) The set from #1 *after* we failed installation

If that's not correct, please be explicit in your request. I'd like to have this implemented by EOW.

Comment 9

[Robert Strong (they/them - no direct email)]

Since the files in 1) are removed when the installer runs just gather them before running the installer. What I am asking is if they will be submitted for both successful and failed installation attempts. I am less concerned about the install.log and doubt it will be all that helpful for troubleshooting update problems though getting the install.log after the hotfix runs might provide some insight.

Comment 10

[Gregory Szorc [:gps]]

Attachment: Capture updater state as part of forensic upload

I think this is pretty close to what's needed. I'm still struggling to
test this, so not asking for review yet.

Comment 11

[Kamil Jozwiak [:kjozwiak]]

Bogdan, once this get's merged and you need help, let me know and help where I can.

Comment 12

[Gregory Szorc [:gps]]

Attachment: Capture updater state as part of forensic upload

The forensic upload now contains files related to the (possibly broken)
updater service.

The implementation is complicated by the fact that the updater files are
deleted as part of performing an install. So, we need to copy these
files before the installer is executed.

Comment 13

[Benjamin Smedberg]

Comment on attachment 8477067 [details] [diff] [review]
Capture updater state as part of forensic upload

r=me, presuming that rstrong verifies that this list of files is the one you want, and before we deploy this we get him to check a QA payload containing these files to make sure they are useful for diagnosis.

Comment 14

[Robert Strong (they/them - no direct email)]

Comment on attachment 8477067 [details] [diff] [review]
Capture updater state as part of forensic upload

Those are the files we are looking for. I didn't verify anything else since it has been reviewed and to avoid being the bottleneck to getting this landed.

Comment 15

[Gregory Szorc [:gps]]

https://hg.mozilla.org/releases/firefox-hotfixes/rev/ae543afdf66a

Comment 16

[Paul Silaghi, QA [:pauly]]

I'm trying to verify this. What exactly should I follow here? I'm not sure I see any differences in the hotfix log files in comparison with the ones before the fix.

Comment 17

[Gregory Szorc [:gps]]

There will be little difference in the hotfix log files after this bug. The only differences are that the log may reference files belonging to the updater.

The important thing to verify is that updater files are being included in the hotfix payload. And verifying that is a bit complicated.

First, you'll need to produce some files:

1) Install an old Firefox release
2) Trigger a background update
3) Shut down Firefox (but don't restart it)

At this point, c:\Users\<user>\AppData\Local\Mozilla\updates\* should exist. There will be a directory with a hashed value corresponding to the install location. The default install location on Windows hashes to E7CF176E110C211B on my machine. YMMV.

That directory should have files like:

* active-update.xml
* updates.xml
* updates/backup-update.log
* updates/last-update.log

Only 1 of those files needs to exist to test things. But the more the merrier.

You may want to make a backup copy of that directory because the Firefox update mechanism likes deleting the directory. And you'll want a backup copy around to manually replace so you don't have to go through the trouble of triggering the creation of these files again.

4) Make a backup copy of your updates directory
5) Start Firefox to finish the upgrade
6) Shut down Firefox

At this point, you've done the *preliminary* work to verify this bug.

7) Install an old version of Firefox that the hotfix will target (<29).
8) Verify the updates directory is present. Copy from backup if needed.
9) Launch Firefox
10) Trigger background add-on update check to install hotfix
11) Wait for hotfix to do its thing
12) Find the forensicsID from the %profile%/hotfix.v20140527.01.json file
13) Needinfo gps in this bug to look up the ID on the server so I can verify the data went over the wire successfully.

Comment 18

[Paul Silaghi, QA [:pauly]]

(In reply to Gregory Szorc [:gps] (away Sep 10 through 27) from comment #17)
> 13) Needinfo gps in this bug to look up the ID on the server so I can verify
> the data went over the wire successfully.
"forensicsID":"c748cdcf-a37d-44ea-ab50-a60f582eb00e"

Comment 19

[Gregory Szorc [:gps]]

The uploaded record didn't contain any sign of the code from this bug. What version of the add-on were you running?

Also, the log indicates you were at one time installed on the aurora channel and were upgrading to Firefox 30. This is almost certainly an old version of the hotfix.

Comment 20

[Paul Silaghi, QA [:pauly]]

(In reply to Gregory Szorc [:gps] (away Sep 10 through 27) from comment #19)
> What version of the add-on were you running?
The current version in production. Indeed, FF was upgraded to 30.

Tried again with the hotfix from bug 1061975:
"forensicsID":"cf87ed2c-ec15-4dcd-9c3b-18e06b5c06a1"

Comment 21

[Gregory Szorc [:gps]]

I do not see the UUID from comment #20 on the server :/

Does the log in your profile directory say it was uploaded?

Comment 22

[Paul Silaghi, QA [:pauly]]

There is nothing containing 'upload' in c:\Users\paul.silaghi\AppData\Roaming\Mozilla\Firefox\Profiles\9vs69ubk.8\hotfix-update\update.log

I might be doing something wrong.
At step 2, I did the update by Help/About.
At step 7, I installed another old FF, but I used the same profile from the previous steps. 
Is it ok ?

Comment 23

[Paul Silaghi, QA [:pauly]]

Tried again on FF 28 Win 7 x64 with the signed hotfix from bug 1061975:
"forensicsID":"ff2a53d2-4a87-4e63-8dce-ebd0ff348bb3"

Comment 24

[Gregory Szorc [:gps]]

I handed off support of the data for the update hotfix to Stephen Pohl.

Stephen: The easy way to find records for a given day is to adjust the map reduce job filter to only look at data from a given day. Then, hack up one of the jobs to save the record to disk if its ID matches when Paul posts. This is a complete abuse of map reduce, but it works and should take no more than a minute to run on a single day's worth of data.

Comment 25

[Stephen A Pohl [:spohl]]

Still sidetracked by v2 signing work, but I'm hearing that Tim may be in the loop about hotfix forensics and may be able to tackle this?

Comment 26

[Robert Strong (they/them - no direct email)]

Tim is taking on getting the latest hotfix deployed but isn't in the loop regarding hotfix forensics akaik. At present, the mac v2 signing work has to take precedence over this.

Comment 27

[Paul Silaghi, QA [:pauly]]

(In reply to Gregory Szorc [:gps] from comment #17)
> 12) Find the forensicsID from the %profile%/hotfix.v20140527.01.json file
> 13) Needinfo gps in this bug to look up the ID on the server so I can verify the data went over the    > wire successfully.
"forensicsID":"831d9a66-8ee4-4dbd-bd56-1662631d1381"
Tested with the current hotfix from bug 1061975 on staging.

Comment 28

[Stephen A Pohl [:spohl]]

I'm not set up to verify this anymore. Robert, are you able to confirm that this is working as expected now? Please let me know if I should set up the machine to verify this again, or if we're good here. Thank you!

Comment 29

[Robert Strong (they/them - no direct email)]

Hi Stephen, the logs you showed me awhile back is enough for me.