766585 - Update maintenanceservice.exe on test slaves

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Brian R. Bondy [:bbondy]]

Attachment: Updated maintenanceservice.exe with file version set to 0,0,0,0

The maintenanceservice.exe that we use in the zip we deploy on each reboot predates us even landing the maintenance service on m-c.  There are many changes to it since then.  I'd like to see if the intermittent errors in bug 723541 still happen using the latest maintenanceservice.exe.

The maintenanceservice.exe that I attached is the one from the latest m-c build and the only change is to set the version number to 0 so that it will always gets replaced. 

I don't know how to test this, so after updating the zip we deploy on each reboot, please keep an eye on tbpl and revert if there are any new errors.

Comment 1

[[e-mail] (:bhearsum)]

If this has to get rolled out by hand we NEED to test this advance. Rolling back the entire pool is far from trivial.

We might be able to deploy automatically....I don't recall if the batch file we run at boot pulls a new service properly or not.

Comment 2

[Brian R. Bondy [:bbondy]]

All that should be required is to update the maintenanceservice.exe inside the zip file.  There is nothing to do per machine manually.

Comment 3

[Brian R. Bondy [:bbondy]]

If you can easily only deploy a new zip to only oak or something like that, that would be ideal, but I don't think that's possible currently.

Comment 4

[(no longer active)]

You can change the script on one slave to point to another URL, and trigger tests on it, right?

Comment 5

[Rail Aliiev [:rail]]

FTR:

wget http://runtime-binaries.pvt.build.mozilla.org/updateservice.zip
wget -O maintenanceservice.zip https://bugzilla.mozilla.org/attachment.cgi?id=634940
unzip updateservice.zip
rm maintenanceservice.exe
unzip maintenanceservice.zip
zip updateservice-new.zip maintenanceservice.exe maintenanceservice_installer.exe 
scp updateservice-new.zip relengweb1.dmz.scl3.mozilla.com
ssh relengweb1.dmz.scl3.mozilla.com
chmod 644 updateservice-new.zip
sudo cp /var/www/html/runtime-binaries/updateservice.zip{,_pre_bug_766585}
sudo mv updateservice-new.zip /var/www/html/runtime-binaries/updateservice.zip


I'll check the status on some of the slaves in a bit.

Comment 6

[Rail Aliiev [:rail]]

Attachment: file properties

I rebooted one of the slaves twice (to be sure), but it seems like it doesn't updates the service. The script downloads and unzips the file properly to c:\updateservice, but it doesnn't look like service installer copies it.

Any idea how to debug this?

Comment 7

[Brian R. Bondy [:bbondy]]

Do you get the same results when you double click the maintenanceservice_installer.exe file?
 
Can you attach a zip of the logs found here?
C:\Users\All Users\Mozilla\logs

Comment 8

[Rail Aliiev [:rail]]

It turns out that W7 test machines get the file updated. However, maintenanceservice_installer.exe doesn't update the file on XP test slaves. Is this expected?

Comment 9

[Brian R. Bondy [:bbondy]]

No that's not expected. Could you attach the zip here and I'll try on a local XP machine?

Comment 10

[Brian R. Bondy [:bbondy]]

also teh logs mentioned in Comment 7 might indicate what's going on.

Comment 11

[Rail Aliiev [:rail]]

Attachment: current zip

Attached is the current zip file.

Comment 12

[Rail Aliiev [:rail]]

Attachment: logs

Logs

Comment 13

[Brian R. Bondy [:bbondy]]

Is this by chance an x64 machine? or is it an x86 machine?  It seems like the location it wants to install to didn't already have a maintenanceservice.exe from the logs.

If it's an x64 you need to check inside Program Files (x86)

I tried the zip package on my x86 XP machine and it worked correctly, it forced an upgrade of the v0.0.0.0 overtop of the existing v13 maintenanceservice (because the maintenanceservice_installer.exe in the zip package forces the install).

Comment 14

[Rail Aliiev [:rail]]

(In reply to Brian R. Bondy [:bbondy] from comment #13)
> Is this by chance an x64 machine? or is it an x86 machine?  It seems like
> the location it wants to install to didn't already have a
> maintenanceservice.exe from the logs.

The logs are from a 32-bit machine. I checked the status of 10-15 x86 XP test slaves, and it's the same.

Comment 15

[Brian R. Bondy [:bbondy]]

That's way messed up since my XP machine works with the same zip.

Could you try renaming the maintenanceservice.exe out of the way and then double clicking the maintenanceservice_installer.exe. Does that install the updated service?
Then you could restore it back to the old state and double click again, does it not replace it as before?

Comment 16

[Rail Aliiev [:rail]]

I renamed maintenanceservice.exe in C:\Program Files\Mozilla Maintenance Service and the installer worked properly...

Comment 17

[Rail Aliiev [:rail]]

Btw, do you rename maintenanceservice.exe as part of the installation program? There is maintenanceservice_tmp.exe in the same directory. Maybe the installer fails to rename te file somehow?

Comment 18

[Brian R. Bondy [:bbondy]]

> There is maintenanceservice_tmp.exe in the same directory. 
> Maybe the installer fails to rename te file somehow?

No that's normal. The maintenanceservice.exe file gets copied from wherever you execute it into %programfiles%\Mozilla Maintenance Service\maintenanceservice_tmp.exe.  Then a version compare is done and if it is newer it replaces the old file. 

Could you go to services manager and let me know the path of where it thinks the service is at? Control Panel -> Administrative tools -> Services
Then double click on Mozilla Maintenance Service

Comment 19

[Rail Aliiev [:rail]]

I checked the path to executable for Mozilla Maintenance Service on one of the x86 XP test slaves and it points to C:\Program Files\Mozilla Maintenance Service\maintenanceservice_tmp.exe

maintenanceservice_tmp.exe looks up to date, but maintenanceservice.exe is an older version.

Comment 20

[Brian R. Bondy [:bbondy]]

Ya I suspected that was the problem.  It should be pointing to maintenanceservice.exe.  I'll look at the code and try to find out how it could have gotten in that state. Maybe from some old bug.

Comment 21

[Brian R. Bondy [:bbondy]]

So the only way I can see that this could happen is if C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice.exe exists but the MozillaMaintenance service itself is not installed with Windows.
In that case the maintenanceservice_installer.exe would see that maintenanceservice.exe already exists and it would copy it in as maintenanceservice_tmp.exe. 
serviceinstall.cpp would then try to open the service, but get a "service does not exist" error, so it would install using the current path maintenanceservice_tmp.exe.

This would have happened because when we first landed the service, we would uninstall the service and then reinstall on each upgrade. 
There was an intermittent problem that happened rarely with this, but the test slaves are probably more prone to them because they run upgrades so often.

So conclusion:
1) I just wrote a patch which will fix any old computers that had those early Nightly builds and happened to have the rare intermittent problem.
2) I'll post a bug for this shortly and attach the patch.
3) After that bug is fixed, you should be able to proceed with this bug, I'll provide a new maintenanceservice.exe again but you don't need it, I'd just prefer it be the latest.

No manual process will be needed, the patch that I made will fix any computers that are in this state. 
Computers in this state aren't a problem, but it will use the latest installed service and not the latest service.

Comment 22

[Rail Aliiev [:rail]]

Thanks a lot. Sounds good to me.

Comment 23

[Brian R. Bondy [:bbondy]]

Attachment: Zipv2 - Updated maintenanceservice.exe with file version set to 0,0,0,0

Here's the updated bin, please try the upgrade on that XP machine, the path on it should be set to maintenanceservice.exe now.

Comment 24

[Rail Aliiev [:rail]]

\o/ 
Looks good to me now. I checked the status of C:\Program Files\Mozilla Maintenance Service\maintenanceservice.exe file on xp and w7 slaves and they have proper size, timestamp and version information. You may want to wait 4-6 hours until most of the slaves rebooted.

Feel free to reopen if something goes wrong with testing.

Comment 25

[Brian R. Bondy [:bbondy]]

Thanks!

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Rail, did we get this onto the ref images too ?

Comment 27

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #26)
> Rail, did we get this onto the ref images too ?

I haven't asked explicitly to do this since the change is very small and the way how we update the files is puppet-like. However, we can ask to take a snapshot to be sure unless somebody asked since then.