635161 - rename original callback file and rename the updated callback file to the original callback file name instead of copying the updated callback file over the original file

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Robert Strong (they/them - no direct email)]

The crash stats from Bug 634343 show that the majority of mismatched dll's on 1.9.2 have a Firefox version that is less than the dll version and with 1.9.1 the majority of mismatched dll's have a Firefox version that is greater than the dll version. I suspect that this can be improved by renaming the old firefox.exe, renaming the copy of firefox.exe that has been updated back to firefox.exe, and then deleting the old firefox.exe.

http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/updater.cpp#1779

Comment 1

[Robert Strong (they/them - no direct email)]

After removing a few questionable reports, for the last 3 months of Firefox 3.6.x crash reports with mismatched dll's there were:
total reports: 1109
reports with a firefox.exe version greater than the dlls: 50
reports with a firefox.exe version lesser than the dlls: 1059

Top 10
count   firefox.exe     dll
376	1.9.2.3951	1.9.2.3989
174	1.9.2.3743	1.9.2.3989
128	1.9.2.3667	1.9.2.3989
31	1.9.2.3909	1.9.2.3989
30	1.9.2.3743	1.9.2.3855
29	1.9.2.3743	1.9.2.3951
28	1.9.2.3743	1.9.2.3828
28	1.9.2.3989	1.9.2.4038
26	1.9.2.3743	1.9.2.3909
22	1.9.2.3855	1.9.2.3989

The vast majority have 3.6.13 dll's with a previous firefox.exe. The top 4 have a firefox.exe starting of 3.6.12 with each subsequent being for the previous release.

Comment 2

[Robert Strong (they/them - no direct email)]

After removing a few questionable reports, for the last 3 months of Firefox
3.5.x crash reports with mismatched dll's there were:
total reports: 6913
reports with a firefox.exe version greater than the dll's: 6803
reports with a firefox.exe version lesser than the dll's: 110

Top 10
count	firefox.exe	dll
1123	1.9.1.3986	1.9.1.3593
763	1.9.1.3986	1.9.1.3642
671	1.9.1.3776	1.9.1.3726
619	1.9.1.3986	1.9.1.3622
566	1.9.1.3986	1.9.1.3726
327	1.9.1.4038	1.9.1.3400
232	1.9.1.3986	1.9.1.3685
225	1.9.1.3986	1.9.1.3576
201	1.9.1.3834	1.9.1.3726
187	1.9.1.3951	1.9.1.3726

There were no 3.6.x crash reports with a dll version that didn't match the other dll versions whereas there were a couple of 3.5.x crash reports with a dll version that didn't match the other dll versions.

I am quite certain that bug 525390 which landed for Firefox 3.6 is the main reason for the reduction of mismatched dll's. I'm also very confident that fixing this bug will also significantly reduce the remaining crash reports with mismatched dll's.

Comment 3

[Robert Strong (they/them - no direct email)]

Surprisingly I was unable to cause a crash with an earlier firefox.exe version so removing whiteboard and keyword.

Comment 4

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

Comment 5

[Robert Strong (they/them - no direct email)]

Comment on attachment 514452 [details] [diff] [review]
patch rev1

Jim, after looking at mismatched dll's on crashstats I think something along these lines would be worthwhile to try to lessen the number of frankenfox's even more.

Comment 6

[Robert Strong (they/them - no direct email)]

Comment on attachment 514452 [details] [diff] [review]
patch rev1

btw: I am not planning on this for Firefox 4 but I would like feedback on this approach

Comment 7

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

Comment 8

[Robert Strong (they/them - no direct email)]

Stats for Firefox 4.0b12 and 4.0b12pre:

count  Firefox ver       firefox.exe ver   dll ver
243    Firefox:4.0b12      2.0.0.4051     2.0.0.4070
19     Firefox:4.0b12      2.0.0.4038     2.0.0.4070
6      Firefox:4.0b12      2.0.0.3960     2.0.0.4070
5      Firefox:4.0b12pre   2.0.0.4060     2.0.0.4068
3      Firefox:4.0b12      2.0.0.3882     2.0.0.4070
2      Firefox:4.0b12      2.0.0.4000     2.0.0.4070
1      Firefox:4.0b12      2.0.0.3869     2.0.0.4070
1      Firefox:4.0b12      2.0.0.4027     2.0.0.4070
1      Firefox:4.0b12pre   2.0.0.4028     2.0.0.4069
1      Firefox:4.0b12pre   2.0.0.4063     2.0.0.4060
1      Firefox:4.0b12pre   2.0.0.4066     2.0.0.4067
1      Firefox:4.0b12pre   2.0.0.4068     2.0.0.4069
1      Firefox:4.0b12pre   2.0.0.4069     2.0.0.4070

Comment 9

[Jim Mathies [:jimm]]

Comment on attachment 516490 [details] [diff] [review]
patch rev1

two nits - 

>+      if (rv != OK && NS_taccess(callbackBackupPath, F_OK) == 0) {

!NS_taccess(callbackBackupPath, F_OK)

>+        if (CopyFileW(callbackBackupPath, argv[callbackIndex], FALSE) != 0) {

just

if (CopyFileW()) {

should be fine.

Comment 10

[Robert Strong (they/them - no direct email)]

Attachment: patch with comments addressed

There is no approval flag for 2.x yet so requesting 2.0 just to get it on the radar

Comment 11

[Daniel Veditz [:dveditz]]

Given comment 8 it should block 2.0.1 as well, requesting blocking2.0 to get a .x+ blocking status.

Comment 12

[Johnathan Nightingale [:johnath]]

Comment on attachment 518584 [details] [diff] [review]
patch with comments addressed

Clearing the approval2.0 flag here - we should re-nom this for a branch release (hence .x+) but we shouldn't push it into 4.0 itself.

Comment 13

[Daniel Veditz [:dveditz]]

Comment on attachment 518584 [details] [diff] [review]
patch with comments addressed

Approved for 1.9.2.16, a=dveditz for release-drivers

Comment 14

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-1.9.2
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/b5267288c8e4

Leaving open until after this lands on trunk

Comment 15

[Robert Strong (they/them - no direct email)]

With the report from bug 635834 there are around .05 % out of 56344 Windows crash reports that have mismatched dll's. I'll provide details after I finish analyzing the data,

Comment 16

[Robert Strong (they/them - no direct email)]

(In reply to comment #15)
> With the report from bug 635834 there are around .05 % out of 56344 Windows
> crash reports that have mismatched dll's. I'll provide details after I finish
> analyzing the data,
btw: this is trunk only

Comment 17

[Robert Strong (they/them - no direct email)]

With the report from bug 635834 I see the following mismatched dll's out of a total of 56344 crashes on Windows trunk

dll filename           total  % of total crashes
browsercomps.dll        32        0.0050%
browserdirprovider.dll   0        0%
brwsrcmp.dll             0        0%
mozalloc.dll            22        0.0035%
xpcom.dll               23        0.0036%
xul.dll                 31        0.0049%

All of the version mismatches had a firefox.exe version less than the dll version which supports the conclusions in this bug that at times removing the updater's lock on firefox.exe and copying the updated firefox.exe over the original sometimes fails due to the file not being unlocked in time.

Comment 18

[Robert Strong (they/them - no direct email)]

Using the updated reports from bug 635834 (only Windows trunk so far for the dates from 20110103 to 20110201).

Skipped 98 out of 1386278 crashes due to the crash having an invalid name for firefox.exe or firefox.exe not having version information.

The percentage is out of 1386180 crashes

                      |       Total       |  dll ver <   |   dll ver >    |
Crashes               | 1386180 (100%)    |     N/A      |      N/A       |
Mismatch Crashes      |     805 (0.0581%) | 15 (0.0011%) |  790 (0.0570%) |
browsercomps.dll      |     773 (0.0558%) | 13 (0.0009%) |  760 (0.0548%) |
mozalloc.dll          |     435 (0.0314%) |  9 (0.0006%) |  426 (0.0307%) |
xpcom.dll             |     420 (0.0303%) | 11 (0.0008%) |  409 (0.0295%) |
xul.dll               |     472 (0.0341%) | 10 (0.0007%) |  462 (0.0333%) |


The percentage is out of 805 crashes with mismatched dll's

                      |   Total    | dll ver < | dll ver > |
Mismatch Crashes      | 805 (100%) |  15 (2%)  | 790 (98%) |
browsercomps.dll      | 773 ( 96%) |  13 (2%)  | 760 (94%) |
mozalloc.dll          | 435 ( 54%) |   9 (1%)  | 426 (53%) |
xpcom.dll             | 420 ( 52%) |  11 (1%)  | 409 (51%) |
xul.dll               | 472 ( 59%) |  10 (1%)  | 462 (57%) |

Comment 19

[Robert Strong (they/them - no direct email)]

Comment on attachment 518584 [details] [diff] [review]
patch with comments addressed

Per bug 641422 comment #6 this patch should be approval2.0=?

Comment 20

[Robert Strong (they/them - no direct email)]

Updated reports using the latest data from bug 635834

Firefox 3.5.x Mismatched File Version Report

Summary
----------------------------
     Lines processed: 463310
   Crashes processed: 463193
       Lines skipped:    117
    Total mismatches:  39943
  Percent mismatches:  8.62%

Skipped Line Breakdown
-------------------------------
  Invalid firefox.exe name: 101
      Missing version info:  16

Mismatched Versions
---------------------------------------------------------------------------
                             Total        | DLLVer < FFVer | DLLVer > FFVer
        Total mismatches: 39943 (100.00%) | 39348 (98.51%) |  595 (1.49%)
  browserdirprovider.dll:  4618  (11.56%) |  4028 (10.08%) |  590 (1.48%)
            brwsrcmp.dll:  2037   (5.10%) |  1628  (4.08%) |  409 (1.02%)
               xpcom.dll: 30057  (75.25%) | 30044 (75.22%) |   13 (0.03%)
                 xul.dll: 39363  (98.55%) | 39344 (98.50%) |   19 (0.05%)


Firefox 3.6.x Mismatched File Version Report

Summary
-----------------------------
     Lines processed: 4968472
   Crashes processed: 4967790
       Lines skipped:     682
    Total mismatches:    5447
  Percent mismatches:   0.11%

Skipped Line Breakdown
-------------------------------
       Invalid line format:   4
  Invalid firefox.exe name: 435
      Missing version info: 245

Mismatched Versions
---------------------------------------------------------------------------
                             Total        | DLLVer < FFVer | DLLVer > FFVer
        Total mismatches: 5447 (100.00%)  |  186 (3.41%)   | 5261 (96.59%)
  browserdirprovider.dll: 5242  (96.24%)  |   57 (1.05%)   | 5185 (95.19%)
            brwsrcmp.dll: 5169  (94.90%)  |   51 (0.94%)   | 5118 (93.96%)
               xpcom.dll: 2032  (37.30%)  |   85 (1.56%)   | 1947 (35.74%)
                 xul.dll: 1809  (33.21%)  |  162 (2.97%)   | 1647 (30.24%)


Firefox 4.0.x Mismatched File Version Report

Summary
-----------------------------
     Lines processed: 3324365
   Crashes processed: 3324162
       Lines skipped:     203
    Total mismatches:    1638
  Percent mismatches:   0.05%

Skipped Line Breakdown
-------------------------------
       Invalid line format:   8
  Invalid firefox.exe name:  44
      Missing version info: 154

Mismatched Versions
---------------------------------------------------------------------------
                             Total        | DLLVer < FFVer | DLLVer > FFVer
         Total mismatches: 1638 (100.00%) |   21 (1.28%)   | 1617 (98.72%)
         browsercomps.dll: 1540  (94.02%) |   15 (0.92%)   | 1525 (93.10%)
             mozalloc.dll:  821  (50.12%) |   14 (0.85%)   |  807 (49.27%)
                xpcom.dll:  770  (47.01%) |   12 (0.73%)   |  758 (46.28%)
                  xul.dll:  886  (54.09%) |   11 (0.67%)   |  875 (53.42%)

Comment 21

[Robert Strong (they/them - no direct email)]

This should be the final data set from before this patch landing which will be used to evaluate whether this patch made any difference. The Firefox 3.5 numbers are mainly to confirm that the work done for Firefox 3.6 did in fact improve things. The data below represents a total of 25,910,929 crash reports processed for Firefox 3.5, 3.6, and 4.0.

Firefox 3.5 Mismatched File Version Report
20101204 through 20110321

Summary
=============================
    Lines processed : 1487950
  Crashes processed : 1497571
      Lines skipped :     298
   Total mismatches :  120613
 Percent mismatches :   8.05%

Skipped Line Breakdown
===============================
 Invalid firefox.exe name : 260
     Missing version info :  38

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches : 120613 (100.00%) | 118892 (98.57%) |  1721 (1.43%)  |
 browserdirprovider.dll :  11960   (9.92%) |  10249  (8.50%) |  1711 (1.42%)  |
           brwsrcmp.dll :   5247   (4.35%) |   4187  (3.47%) |  1060 (0.88%)  |
              xpcom.dll :  89406  (74.13%) |  89379 (74.10%) |    27 (0.02%)  |
                xul.dll : 118890  (98.57%) | 118866 (98.55%) |    24 (0.02%)  |
                        +------------------+-----------------+----------------+


Firefox 3.6 Mismatched File Version Report
20101204 through 20110321

Summary
==============================
    Lines processed : 17550147
  Crashes processed : 17551884
      Lines skipped :     2987
   Total mismatches :    43963
 Percent mismatches :    0.25%

Skipped Line Breakdown
================================
      Invalid line format :   19
 Invalid firefox.exe name : 2253
     Missing version info :  716

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches :  43963 (100.00%) |   932 (2.12%)   | 43031 (97.88%) |
 browserdirprovider.dll :  42825  (97.41%) |   325 (0.74%)   | 42500 (96.67%) |
           brwsrcmp.dll :  42654  (97.02%) |   302 (0.69%)   | 42352 (96.34%) |
              xpcom.dll :  14310  (32.55%) |   385 (0.88%)   | 13925 (31.67%) |
                xul.dll :   6627  (15.07%) |   795 (1.81%)   |  5832 (13.27%) |
                        +------------------+-----------------+----------------+


Firefox 4.0 Mismatched File Version Report
20101111 through 20110321

Summary
=============================
    Lines processed : 6853167
  Crashes processed : 6861474
      Lines skipped :     638
   Total mismatches :    3149
 Percent mismatches :   0.05%

Skipped Line Breakdown
===============================
      Invalid line format :  41
 Invalid firefox.exe name : 200
     Missing version info : 398

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches :  3149 (100.00%)  |   85 (2.70%)    |  3064 (97.30%) |
       browsercomps.dll :  2984  (94.76%)  |   74 (2.35%)    |  2910 (92.41%) |
           mozalloc.dll :  1791  (56.88%)  |   70 (2.22%)    |  1721 (54.65%) |
              xpcom.dll :  1773  (56.30%)  |   68 (2.16%)    |  1705 (54.14%) |
                xul.dll :  2138  (67.89%)  |   69 (2.19%)    |  2069 (65.70%) |
                        +------------------+-----------------+----------------+

Comment 22

[Daniel Veditz [:dveditz]]

Comment on attachment 518584 [details] [diff] [review]
patch with comments addressed

Approved for the mozilla2.0 repository, a=dveditz for release-drivers

Comment 23

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-2.0
http://hg.mozilla.org/releases/mozilla-2.0/rev/8f41e6f646cf

I have a different / better fix for this in bug 386760 so this won't land on mozilla-central... leaving open until that fix is reviewed and landed.

Comment 24

[Robert Strong (they/them - no direct email)]

Bug 386760 fixes this using a different method so resolving -> fixed. Also, bug 386760 has tests that verify we can update the application binary.

Comment 25

[Robert Strong (they/them - no direct email)]

Latest numbers

Total crashes processed : 31761889

Firefox 3.5 Mismatched File Version Report
20101204 through 20110411

Summary
=============================
    Lines processed : 1711982
  Crashes processed : 1721575
      Lines skipped :     361
   Total mismatches :  137788
 Percent mismatches :   8.00%

Skipped Line Breakdown
===============================
 Invalid firefox.exe name : 309
     Missing version info :  52

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches : 137788 (100.00%) | 135850 (98.59%) |  1938 (1.41%)  |
 browserdirprovider.dll :  13507   (9.80%) |  11575  (8.40%) |  1932 (1.40%)  |
           brwsrcmp.dll :   6087   (4.42%) |   4894  (3.55%) |  1193 (0.87%)  |
              xpcom.dll : 101843  (73.91%) | 101810 (73.89%) |    33 (0.02%)  |
                xul.dll : 135854  (98.60%) | 135823 (98.57%) |    31 (0.02%)  |
                        +------------------+-----------------+----------------+


Firefox 3.6 Mismatched File Version Report
20101204 through 20110411

Summary
==============================
    Lines processed : 20277061
  Crashes processed : 20278718
      Lines skipped :     3797
   Total mismatches :    49995
 Percent mismatches :    0.25%

Skipped Line Breakdown
================================
      Invalid line format :   34
 Invalid firefox.exe name : 2907
     Missing version info :  857

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches :  49995 (100.00%) |   1039 (2.08%)  | 48956 (97.92%) |
 browserdirprovider.dll :  48685  (97.38%) |    370 (0.74%)  | 48315 (96.64%) |
           brwsrcmp.dll :  48465  (96.94%) |    351 (0.70%)  | 48114 (96.24%) |
              xpcom.dll :  16297  (32.60%) |    426 (0.85%)  | 15871 (31.75%) |
                xul.dll :   7903  (15.81%) |    877 (1.75%)  |  7026 (14.05%) |
                        +------------------+-----------------+----------------+


Firefox 4.0 Mismatched File Version Report
20101111 through 20110411

Summary
=============================
    Lines processed : 9750994
  Crashes processed : 9761596
      Lines skipped :     954
   Total mismatches :    3679
 Percent mismatches :   0.04%

Skipped Line Breakdown
===============================
      Invalid line format :  46
 Invalid firefox.exe name : 246
     Missing version info : 664

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches :  3679 (100.00%)  |    95 (2.58%)   |  3584 (97.42%) |
       browsercomps.dll :  3501  (95.16%)  |    82 (2.23%)   |  3419 (92.93%) |
           mozalloc.dll :  2110  (57.35%)  |    77 (2.09%)   |  2033 (55.26%) |
              xpcom.dll :  2067  (56.18%)  |    75 (2.04%)   |  1992 (54.15%) |
                xul.dll :  2531  (68.80%)  |    77 (2.09%)   |  2454 (66.70%) |
                        +------------------+-----------------+----------------+

Comment 26

[Robert Strong (they/them - no direct email)]

Going back just two weeks of Firefox 4 the numbers are tad better coming in at 0.02% vs 0.05% and the same is true going back 4 weeks.

Firefox 4.0 Mismatched File Version Report
20110329 through 20110411

Summary
============================
    Lines processed : 841619
  Crashes processed : 841894
      Lines skipped :     84
   Total mismatches :    175
 Percent mismatches :  0.02%

Skipped Line Breakdown
==============================
      Invalid line format :  1
 Invalid firefox.exe name : 16
     Missing version info : 67

Mismatched Versions
==============================================================================+
                        |      Total       | DLLVer < FFVer  | DLLVer > FFVer |
                        +------------------+-----------------+----------------+
       Total mismatches :   175 (100.00%)  |    5 (2.86%)    |  170 (97.14%)  |
       browsercomps.dll :   169  (96.57%)  |    4 (2.29%)    |  165 (94.29%)  |
           mozalloc.dll :    94  (53.71%)  |    4 (2.29%)    |   90 (51.43%)  |
              xpcom.dll :    98  (56.00%)  |    4 (2.29%)    |   94 (53.71%)  |
                xul.dll :   147  (84.00%)  |    5 (2.86%)    |  142 (81.14%)  |
                        +------------------+-----------------+----------------+