Closed Bug 749956 Opened 12 years ago Closed 12 years ago

Apparently permanent Windows toolkit\mozapps\update\test_svc\unit\test_0000_bootstrap_svc.js failure in head_update.js | failed: 16002 == succeeded plus 14 other failures

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: RyanVM, Assigned: bbondy)

References

Details

(Keywords: intermittent-failure)

Attachments

(4 files, 2 obsolete files)

logs from talos-r3-w7-066
1.93 KB, application/x-gzip
Details
screenshot of regedit at HKLM\Software\Mozilla\MaintenanceService
79.83 KB, image/png
Details
New installer that adds the certs + forces install
177.87 KB, application/octet-stream
Details
New installer script that adds certs + forces install. In tree patch.
11.17 KB, patch
ehsan.akhgari
: review+
Details | Diff | Splinter Review 
+++ This bug was initially created as a clone of Bug #715746 +++

Not sure if this is related to bug 715746 or not, so feel free to dupe as needed.

https://tbpl.mozilla.org/php/getParsedLog.php?id=11293309&full=1&branch=mozilla-central
Rev3 WINNT 6.1 mozilla-central pgo test xpcshell on 2012-04-28 06:34:43 PDT for push 8f377102841b

builder: mozilla-central_win7_test_pgo-xpcshell
slave: talos-r3-w7-047
starttime: 1335620083.18
results: warnings (1)
buildid: 20120428060001
builduid: 46db0e9eeb0648ce97434aec5a07a8d2
revision: 8f377102841b

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0000_bootstrap_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0110_general_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0111_general_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0112_general_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | -1 != -1 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0150_appBinReplaced_xp_win_complete_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0151_appBinPatched_xp_win_partial_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0160_appInUse_xp_win_complete_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0170_fileLocked_xp_win_complete_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | -1 != -1 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0171_fileLocked_xp_win_partial_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | -1 != -1 - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0180_fileInUse_xp_win_complete_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0181_fileInUse_xp_win_partial_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0182_rmrfdirFileInUse_xp_win_complete_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0183_rmrfdirFileInUse_xp_win_partial_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0200_app_launch_apply_update_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js | failed: 16002 == succeeded - See following stack:
Bug 718922 and bug 733188 are more likely to be relevant to your interests - 16002 == succeeded should mean either that the build is correctly signed but the cert isn't on the test slave, or that the build is signed with the wrong cert. Since we've seen it on multiple test slaves which haven't been freshly reimaged, I'd bet on this morning's infra outage, whatever that was, having broken signing. Silently, which is a Very Bad Thing according to bug 733188 comment 3.
Component: Application Update → Release Engineering: Automation (General)
Product: Toolkit → mozilla.org
QA Contact: application.update → catlee
Summary: Intermittent toolkit\mozapps\update\test_svc\unit\test_0000_bootstrap_svc.js failure in head_update.js | failed: 16002 == succeeded plus 14 other failures → Apparently permanent Windows toolkit\mozapps\update\test_svc\unit\test_0000_bootstrap_svc.js failure in head_update.js | failed: 16002 == succeeded plus 14 other failures
Version: Trunk → other
That last one is a retrigger on a build which had previously passed, so apparently it's one of two things - catlee wants to blame the tests, like the in-tree complete.mar being signed with a now-expired cert or something, and I want to blame whatever cert it is that's installed on the slaves, a la bug 718922.
So I downloaded the build from the first package and it seems the binaries are being signed by a cert with name:
Mozilla Fake SPC and that is issued by Mozilla Fake CA.

It should be signed by the Nightly Mozilla signature which is issued by a trusted root authority and not issued by our fake CA.  The issuer should be "Thawte Code Signing CA - G2" and the name should be: "Mozilla Corporation". 

This is bad because we will refuse to install updates from the service for an updater like this.  If I recall correctly the update would fail and we would use the fallback to the old updater for this case on the next browser restart.  But it's still very dangerous. 

This seems to be failing also on mozilla-inbound.  There I think it's because the test machines should manually have Mozilla Fake CA in the root authority.  I'm not sure what changed recently. 

I think bhearsum would potentially know what's going on and how to fix.
The first part of Comment 11 was related to mozilla-central.
So it sounds like 2 different problems to me:
1) mozilla-central builds shouldn't be signed with this cert at all
2) I think the root authority is not manually setup on some machines for where this is failing.
> 1) mozilla-central builds shouldn't be signed with this cert at all
> 2) I think the root authority is not manually setup on some 
> machines for where this is failing.

Actually I think 1 is probably not a problem. I think the nightly cert I'm thinking about is only used for Nightly builds. 

So I think it falls back to 2) for both.
> That last one is a retrigger on a build which had previously passed, 
> so apparently it's one of two things - catlee wants to blame the tests, like 
> the in-tree complete.mar being signed with a now-expired cert or something
> , and I want to blame whatever cert it is that's installed on the slaves, 
> a la bug 718922.

This error code is for authenticode updater.exe signature checks, not for the MAR files.  

The Issued to cert "Mozilla Fake SPC" is valid until 12/31/2039.
Maybe the Issued by "Mozilla FAke CA" expired though or is not installed. 
I don't have a copy of that so I cannot check.
The root CA is at https://bugzilla.mozilla.org/attachment.cgi?id=577619, and I think is also valid until 2039.
Brian - it looks like these tests are only failing on Win7. XP tests are still running fine.

It's also affected dozens of different win7 slaves starting today. Some of those have recent successful tests e.g.

talos-r3-w7-066 passed profiling_win7_test-xpcshell build at 2012-04-28 05:44:06 but then failed on birch_win7_test_pgo-xpcshell at 2012-04-28 07:37:21
> talos-r3-w7-066 passed profiling_win7_test-xpcshell build at 
> 2012-04-28 05:44:06 but then failed on birch_win7_test_pgo-xpcshell 
> at 2012-04-28 07:37:21

If it's related to something expiring I'm wondering if it will ever pass again though?
Maybe the time server or something is down and the current computer time on those machines are wrong?
I installed the root CA on my machine and the updater.exe from a failed build verifies fine. 

Did something change with the signing or server side lately?
Maybe a new cert? if so maybe was not installed to the SYSTEM account's root store?

No relevant changes have landed that I'm aware of by the way.
I should mention I'm running x64 Windows 7 and verified an x86 build that was failing on talos.
If it verifies fine on your machine, that suggests that the signing system is ok. We haven't changed anything on the test machines AFAIK.

Can you run the same unittests on your machine that are failing on the test systems?
Could you zip up this directory and attach it to this bug?
C:\ProgramData\Mozilla\logs

It should contain more information about the problem
talos-r3-w7-066 thinks the date is april 28, 2012 4:30pm
...from any machine that is failing
So it looks like the WinVerifyTrust command is succeeding like my test app is.  But it also looks like the registry is not setup correctly or not reading correctly.  Could you send me an export from HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService on the same machine?
In particular I think that on that machine this key:
HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4

has only 1 subkey named 0 and should have 2 subkeys. 
I'm guessing the 0 key holds name="Mozilla Corporation"
and issuer="Thawte Code Signing CA - G2"

The 1 key that I think is missing should I think hold name="Mozilla Fake SPC"
and issuer="Mozilla Fake CA"
looks like you're right about it missing subkey 1.

so...how did this subkey disappear?
Another possibly fun question: how did  talos-r3-w7-030 manage to do the single successful Win7 run of xpcshell since 7:30 this morning, apparently still being healthy as of 12:15? Is it refusing to do OPSI while everyone else is?
> screenshot of regedit at HKLM\Software\Mozilla\MaintenanceService
> looks like you're right about it missing subkey 1.
> so...how did this subkey disappear?

I don't know but I do not think it is the application code that is doing it.
I know there's a script that runs after each reboot to put the service back to its old state.  Maybe that got changed or is not working correctly?


> Another possibly fun question: how did  talos-r3-w7-030 manage to do the
>  single successful Win7 run of xpcshell since 7:30 this morning, apparently 
> still being healthy as of 12:15? Is it refusing to do OPSI while everyone else 
> is?

Has it failed and then succeeded for this problem? Or only failed after that success? I think it is something wrong in the script that the talos machines run after each reboot.
It's the weekend (which is the only reason you can still hear yourself think in here, without an absolutely constant bombardment of starring), so according to https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-w7-030 that was the only time it has run xpcshell between 7:30 Saturday and now. I did consider pushing to try and then triggering a couple hundred xpcshell runs just to see whether it really did affect every single Win7 slave, but since that's the only green run we've had, so far I'm going with the assumption that it does.
So this can be fixed by re-adding the reg entries that are missing, but I think probably the script that runs after each start would just remove them again.  So likely that script needs to be fixed/tweaked. 

Comment 49 lists the key that is missing with the 2 registry values that should be added.  I *think* those are correct.
I had a look at talos-r3-w7-007 (of comment #72). It runs attachment 579099 [details] on boot, and I've verified
* the registry doesn't have a 1 key at HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4 on this slave
* the system clock is correct
* that we didn't regress the updateservice.zip retrieved when swapping the host we're downloading from last week (ie sha1 hash hasn't changed between dm-wwwbuild01 and relengweb1.dmz.scl3, not sure the timing is be right anyway)
* the script runs without issue on boot (according to the task schedulers 'Last Run Result'
* a manual run has the same result, nothing appears on screen to verify
* running the mozillamaintenance_installer.exe as administrator yields a 'Program Compatibility Assistant' dialog saying the program might not have installed properly, and options of
  * Reinstall using recommended settings
  * This program installed ok
* choosing the latter still has no 1 key
* manually adding the 1 key (per comment #49) and rerunning does not show a dialog (or anything at all), I think the assistant remembers your previous choice. The 1 key is still present
* removing the 1 key and rerunning also shows no dialog, 1 isn't recreated
* rebooting in this state doesn't put the 1 key back
* going back and taking the Reinstall option doesn't yield a 1 key

I'm not sure how useful this is, but I'm happy to try other suggestions. We can also set up a slave loan. Did we look for a code change that might have triggered this ?
> * rebooting in this state doesn't put the 1 key back

Could you attach an export of the following registry key (including subkeys):
HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService

From:
1) A machine with the problem
2) A machine without the problem (XP machine is fine)

I want to make sure we're looking into the right problem.  I.e. I want to make sure we're trying to figure out how the 1 key got deleted. If the registry exports happen to match for example then that means that the setup is not as I suspected.  If they differ in the way we're discussing (missing a 1 key) then the problem is to figure out how the 1 key got deleted and to restore it to all machines. 

> * going back and taking the Reinstall option doesn't yield a 1 key

I hadn't seen the script that runs until you posted it, so it seems the 1 key was manually added to each machine, I think by bhearsum originally.

> Did we look for a code change that might have triggered this ?

The relevant application code is in both toolkit/mozapps/update/ and toolkit/component/maintenanceservice/:

The last change in toolkit/mozapps/update was on April 18th 2012:
http://hg.mozilla.org/mozilla-central/rev/62c44b96f769
A quick skim of this file indicates it's not related and also the timeline does not match up.

The last change in toolkit/components/maintenanceservice was on March 5th 2012:
http://hg.mozilla.org/mozilla-central/rev/7a690d2c2e23
Timeline there doesn't match up either.
I'm fairly certain I know what happened. 

I'm guessing the affected machines have Firefox release channel installed on them and recently got a software update.  The maintenance service installer removes the entire key: "SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4" on purpose.  If this is set on a user's machine it is a bad thing, it should only be set for test machines. 

I think the custom built maintenaceservice_installer that we use in that package manually adds SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4 with the 0 key.  The 1 key was manually added by bhearsum originally. 

So the fix would be to add a 1 key to that setup scrip that runs on each reboot that ensures the following key exists: 
SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\1
name="Mozilla Fake SPC"
issuer="Mozilla Fake CA"
Actually only the uninstaller removes it (SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4).  So I'm guessing somehow the uninstaller of Firefox Release was run on those machines after the update was taken?

In any case the fix is the same as described in Comment 89.
Adding this to the end of the script from attachment 579099 [details] should solve the problem:

reg add HKLM\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\0 /v name /t REG_SZ /d "Mozilla Corporation" /f
if ERRORLEVEL 1 exit /b 1
reg add HKLM\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\0 /v issuer /t REG_SZ /d "Thawte Code Signing CA - G2" /f
if ERRORLEVEL 1 exit /b 1
reg add HKLM\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\1 /v name /t REG_SZ /d "Mozilla Fake SPC" /f
if ERRORLEVEL 1 exit /b 1
reg add HKLM\SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4\1 /v issuer /t REG_SZ /d "Mozilla Fake CA" /f
if ERRORLEVEL 1 exit /b 1

Please try the script to make sure the reg entries are setup properly after running the script from task scheduler.
If you have to manually deploy the script to each machine, I can instead give you a new service installer file which does the same on each run.
If not for the need to have run the uninstaller, we'd have a smoking gun: this started at 06:30 Saturday morning, and at 06:00 the weekly addontester talos runs ran for the first time against 12.0 release builds. They are also (I think) the only thing we test that runs the installer rather than just using a zip.
I don't know anything about the weekly addontester talos runs but if we run the installers for those are you sure we don't run the uninstaller for them?
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/addontester-win32/1334959045/addontester_win7_test-addon-bm15-tests1-windows-build0.txt.gz looks a great deal like running the installer to put it in c:\talos-slave\talos-data\firefox\, and then just rm -rf'ing that without any uninstall.
Ya I don't see it running uninstall/helper.exe in the log.  I think somehow the uninstaller was run though, and regardless of how, the fix in Comment 91 will resolve this issue moving forward.   

I would like to find out how it was run though just so we know.
For future reference, patched installer file that manually installs the Mozilla Fake SPC cert in the registry.
When Firefox 12 is installed for the addontester job, presumably the existing Maintenance Service install is updated. Is there an uninstall of the existing service or is it simply overwritten ?
the installer is run again but not the uninstaller.
Attached file New installer that adds the certs (obsolete) — 
If you replace the maintenanceservice_installer.exe with the one in this attachment I think it will fix the problem.  It basically adds these 4 extra lines:

  WriteRegStr HKLM "${FallbackKey}\0" "name" "Mozilla Corporation"
  WriteRegStr HKLM "${FallbackKey}\0" "issuer" "Thawte Code Signing CA - G2"
  WriteRegStr HKLM "${FallbackKey}\1" "name" "Mozilla Fake SPC"
  WriteRegStr HKLM "${FallbackKey}\1" "issuer" "Mozilla Fake CA"
Attachment #619478 - Attachment is patch: false
Attachment #619478 - Attachment mime type: text/plain → application/octet-stream
Thanks Brian, this will be the easiest to deploy a change. I'll test it out on talos-r3-w7-007. A quick check though, the old maintenanceservice_installer.exe is 328072 bytes, the new one is only 186040. Is that expected ?
The old ones were built on elm, the new ones built on my machine locally.  So I'm not sure exactly why but probably due to the nsis version in use.
Testing progress
* created updateservice-2.zip with new maintenanceservice_installer.exe
* modified batch file to pull that file instead
* ran scheduled task
* now have 0 and 1 keys, as expected

So far so good, so I'll go ahead with a proper deployment. 

I also tried running the 12.0 installer, doing a custom install (which didn't mention the service at all). Still have the 0 & 1 keys (also as expected by Brian). Run that Firefox in a new profile, keys still there. Same after manual deletion and reboot. Dunno what caused this in the first place.
updateservice.zip   --> updateservice.zip.pre_bug_749956
updateservice-2.zip --> updateservice.zip

ie the zip file is deployed for jobs starting now.
Comments 105 through 107 are for jobs that started prior to the fix being deployed. We have green builds on inbound.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
(In reply to Nick Thomas [:nthomas] from comment #104)
> ie the zip file is deployed for jobs starting now.

Actually, all jobs where the slave rebooted (on the previous job) after 2129 should be OK.
Thanks for uploading the new zip, I think this is fixed now. If we see more oranges for this it is because the machine running the test has probably not been rebooted yet.
I'm re-opening this because this is causing problems on Ehsan's Oak branch work.  

We need to manually force an update of the maintenanceservice.exe from within the installer file because I think we did that originally as well in case we test older versions. 

I'll attach a new installer script file and new maintenanceservice_installer.exe that should be deployed again in the zip.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Additionally applying this change to the installer file I gave you last time:

-  ${GetParameters} $0
-  ${GetOptions} "$0" "/Upgrade" $0
-  ${If} ${Errors}
-    nsExec::Exec '"$INSTDIR\$TempMaintServiceName" install'
-  ${Else}
+  ;${GetParameters} $0
+  ;${GetOptions} "$0" "/Upgrade" $0
+  ;${If} ${Errors}
+    nsExec::Exec '"$INSTDIR\$TempMaintServiceName" forceinstall'
+  ;${Else}
     ; The upgrade cmdline is the same as install except
     ; It will fail if the service isn't already installed.
-    nsExec::Exec '"$INSTDIR\$TempMaintServiceName" upgrade'
-  ${EndIf}
+  ;  nsExec::Exec '"$INSTDIR\$TempMaintServiceName" upgrade'
+  ;${EndIf}

Which is basically commenting out other cmd line parameters and always passing forceinstall
Assignee: nobody → netzen

        Attachment #619478 -
        Attachment is obsolete: true

    
  

        Attachment #622409 -
        Attachment mime type: text/plain → application/octet-stream

    
    

    
  
Comment on attachment 622409 [details]
New installer that adds the certs + forces install

Could you please deploy this new binary as soon as possible since it is blocking ehsan's work on oak? Thanks so much for your help.

        Attachment #622409 -
        Flags: feedback?(nrthomas)

    
    

    
  
Comment on attachment 622410 [details] [diff] [review]
New installer script that adds certs + forces install. In tree patch.

r=me.

        Attachment #622410 -
        Flags: review?(ehsan) → review+

    
  
Blocks: bgupdates

    
    

    
  
Landed the script to track changes on m-c, but I'll leave this open until RelEng uploads the new binary into the zip.

http://hg.mozilla.org/mozilla-central/rev/6f6af0178099

    
    

    
  
Comment on attachment 622409 [details]
New installer that adds the certs + forces install

Jobs on slaves which rebooted at 15:44 or later should have this.

RelEng: The existing updateservice.zip was moved to updateservice.zip.pre_bug_749956_round2. And a new updateservice.zip (using this attachment and the existing maintenanceservice.exe from the zip) moved into place.

sha1's:
ff68faa7ea48936a9ec0cceb7e8f80996e7e49e6  
      updateservice.zip.pre_bug_749956   - oldest
49b180cd4afb585cdcbe0549b607f6f31247fefa
      updateservice.zip.pre_bug_749956_round2 - previous
7028fc48f313fb636940156ab72ecf48298a5c3a
      updateservice.zip - current

        Attachment #622409 -
        Flags: feedback?(nrthomas)

    
    

    
  
Thanks!
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED

    
  
No longer blocks: bgupdates
Whiteboard: [orange]
Product: mozilla.org → Release Engineering
Component: General Automation → General

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: