Closed Bug 1064771 Opened 11 years ago Closed 11 years ago

Update CI nodes for system updates and Flash

Categories

(Mozilla QA Graveyard :: Infrastructure, defect, P2)

Product:
Mozilla QA Graveyard
Component:
Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: andrei, Assigned: andrei)

References

Details

(Whiteboard: [sprint])

There are a number of security updates available for Windows. I've done a quick check on staging, available updates: - XP: 2 updates Windows Malicious Software Removal Tool - August 2014 (KB890830) Microsoft Security Essentials - (KB2902907) - Vista: (no staging Vista Machines?) there probably are, just couldn't due to lack of a Vista staging machine - Win7: - update Windows Update - 10 security updates available - 4 optional updates available - Win81: - 11 security updates - 1 optional update
As talked with Andreea yesterday we should wait for the next monthly updates, and probably even delay a further week. Given all the instability lately, I'm not that hesitated to get our machines updated immediately. :/
There is also a new Flash version available (see bug 1069794 comment 0) which should be handled simultaneously.
Right, so here the list of items to do: * Update Windows, Linux, and OX for latest system updates * Upgrade Flash to 15.x Not sure if there is even another Java update available.
Summary: Update Windows OS scl3 machines → Update CI nodes for system updates and Flash
Assignee: nobody → andreea.matei
Status: NEW → ASSIGNED
Andrei has a smaller todo list so he will take this.
Assignee: andreea.matei → andrei.eftimie
Staging is done. I've skipped 2 machines: - mm-osx-107 (which has issues and will be the new 10.10) - mm-ub-1204-64 (which is puppeted) Also I haven't been able to reproduce the Flash crash on mm-win-7-64 with the new flash 15.0.0.152 version. I've run multiple flash versions (15, 14, 13, and ran tests against FF 35 and FF 32). With these results I've left the normal Flash version on these Staging Win machines. We'll check back after the weekend to see if there's any crash reported. The issue might have been fixed in the meantime. I am referring to bug 980938. Staging ======= mm-osx-106 // OS update mm-osx-108 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-109 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-107 // NA, this is 10.10 mm-ub-1204-32 // OS update && Java 7u67 mm-ub-1204-64 // NA puppet mm-ub-1310-32 // OS update mm-ub-1310-64 // OS update mm-win-xp-32 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-64 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-32 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-64 // OS update && Java 7u67 && Flash 15.0.0.152
OS: Windows XP → All
(In reply to Andrei Eftimie from comment #6) > Also I haven't been able to reproduce the Flash crash on mm-win-7-64 with > the new flash 15.0.0.152 version. Please ask the Adobe guys on the appropriate bug for feedback. Maybe this crash has really been patched.
Small update. There have been no crashes recorded on any staging Win machine with regular Flash 15 installed (that would be bug 980938). We will go ahead and update to regular Flash 15 on our production machines as well.
Whiteboard: [sprint]
Next tuesday on 14th we'll have new releases for MS, java and Flash. Lets get this done in the next days.
Status update: Production ========== mm-osx-106-1 // OS update mm-osx-106-2 // OS update mm-osx-106-3 // OS update mm-osx-106-4 // OS update mm-osx-107-1 // NA, will be updated to 10.10 mm-osx-107-2 // NA, will be updated to 10.10 mm-osx-107-3 // NA, will be updated to 10.10 mm-osx-107-4 // NA, will be updated to 10.10 mm-osx-108-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-108-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-108-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-108-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-109-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-109-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-109-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-osx-109-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-ub-1204-32-1 // OS update && Java 7u67 mm-ub-1204-32-2 // OS update && Java 7u67 mm-ub-1204-32-3 // OS update && Java 7u67 mm-ub-1204-32-4 // OS update && Java 7u67 mm-ub-1204-64-1 // OS update && Java 7u67 mm-ub-1204-64-2 // OS update && Java 7u67 mm-ub-1204-64-3 // OS update && Java 7u67 mm-ub-1204-64-4 // OS update && Java 7u67 mm-ub-1310-32-1 // OS update mm-ub-1310-32-2 // OS update mm-ub-1310-32-3 // OS update mm-ub-1310-32-4 // OS update mm-ub-1310-64-1 // OS update mm-ub-1310-64-2 // OS update mm-ub-1310-64-3 // OS update mm-ub-1310-64-4 // OS update mm-win-xp-32-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-xp-32-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-xp-32-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-xp-32-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-vista-32-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-vista-32-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-vista-32-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-vista-32-4 // OS update && Java 7u67 && Flash 15.0.0.152
Can we please stop updating the machines on a day when we have to run tests for a release build? Especially if its a RC build. We should only update the nodes when no release tests are in the queue.
Oh and just a note as what I already mentioned to Andrei on IRC yesterday. OS X 10.10 does not support Java 7. So we might wanna make use of Java 8 in the future.
(In reply to Henrik Skupin (:whimboo) from comment #11) > Can we please stop updating the machines on a day when we have to run tests > for a release build? Especially if its a RC build. What gave you the idea I updated the machines _before_ or _during_ the RC testruns? Machines were updated _after_ all RC testruns were completed. > We should only update the > nodes when no release tests are in the queue. Yep, this is exactly what happened.
To reiterate a bit, a timeline: - RC testruns were finished by ~05:30 - machine updates were done between ~11:30-16:30 (in the order they are mentioned in comment 10) - an ondemand update run was triggered at ~17.30 *All times are UTC+2* The only problem we've seen is with a particular machine: mm-ub-1204-64-3 which didn't start any ondemand update testrun, failing at Jenkins level: > 07:29:35 Started by user anonymous > 07:29:35 [EnvInject] - Loading node environment variables. > 07:29:35 Building remotely on mm-ub-1204-64-3 (12.04 ubuntu linux 64bit) in workspace jenkins/workspace/ondemand_update > 07:29:35 > 07:29:35 Deleting project workspace... java.io.IOException: Failed to mkdirs: jenkins/workspace/ondemand_update > 07:29:35 at hudson.FilePath.mkdirs(FilePath.java:1069) > 07:29:35 at hudson.model.AbstractProject.checkout(AbstractProject.java:1407) > 07:29:35 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671) > 07:29:35 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) > 07:29:35 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580) > 07:29:35 at hudson.model.Run.execute(Run.java:1684) > 07:29:35 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > 07:29:35 at hudson.model.ResourceController.execute(ResourceController.java:88) > 07:29:35 at hudson.model.Executor.run(Executor.java:231) > 07:29:35 Archiving artifacts > 07:29:35 ERROR: Publisher hudson.tasks.ArtifactArchiver aborted due to exception > 07:29:35 /usr/lib/firefox/jenkins/workspace/ondemand_update does not exist. > 07:29:35 at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483) > 07:29:35 at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460) > 07:29:35 at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:181) > 07:29:35 at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:172) > 07:29:35 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2462) > 07:29:35 at hudson.remoting.UserRequest.perform(UserRequest.java:118) > 07:29:35 at hudson.remoting.UserRequest.perform(UserRequest.java:48) > 07:29:35 at hudson.remoting.Request$2.run(Request.java:328) > 07:29:35 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > 07:29:35 at java.util.concurrent.FutureTask.run(FutureTask.java:262) > 07:29:35 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 07:29:35 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 07:29:35 at hudson.remoting.Engine$1$1.run(Engine.java:63) > 07:29:35 at java.lang.Thread.run(Thread.java:745) > 07:29:35 Email was triggered for: Failure - Any > 07:29:35 Sending email for trigger: Failure - Any > 07:29:35 #95453 is still in progress; ignoring for purposes of comparison > 07:29:36 #95453 is still in progress; ignoring for purposes of comparison > 07:29:36 Sending email to: mozmill-ci@mozilla.org > 07:29:36 Finished: FAILURE Thanks Henrik for noticing and getting the machine offline. I'm investigating what the problem with the machine might be.
Found the issue with `mm-ub-1204-64-3`. It was not caused by the update directly, but it was caused by me. The default terminal had cwd set to `/usr/bin/firefox/`. After I installed the updates, and restarted Jenkins (as I did on all other machines), Jenkins assumed the following home folder `/usr/bin/firefox/jenkins` in which the `mozauto` user doesn't have write permissions, thus coming to the failures mentioned in the above comment. In total 7 ondemand update testruns were affected. I retriggered all of them. The machine is back online. I filed the following mozmill-ci issue https://github.com/mozilla/mozmill-ci/issues/506 which should avoid any similar problems like this in the future.
(In reply to Andrei Eftimie from comment #13) > What gave you the idea I updated the machines _before_ or _during_ the RC > testruns? > Machines were updated _after_ all RC testruns were completed. Those were the pulse triggered jobs, but not the ondemand_update jobs which have to run by QA manually in the evening. Those are most critical and we cannot risk to break them. For everything else please see the personal email.
Status update: mm-win-7-32-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-32-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-32-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-32-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-64-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-64-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-64-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-7-64-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-32-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-32-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-32-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-32-4 // OS update && Java 7u67 && Flash 15.0.0.152
mm-win-8-64-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-64-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-64-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-8-64-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-32-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-32-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-32-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-32-4 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-64-1 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-64-2 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-64-3 // OS update && Java 7u67 && Flash 15.0.0.152 mm-win-81-64-4 // OS update && Java 7u67 && Flash 15.0.0.152 All machines have been updated. Some notes: 1) OSX 10.7 machines were skipped. They will be fully updated to 10.10 in the near future 2) As usual Linux is not getting any more Flash updates from Adobe 3) Ubuntu 1310 doesn't have the latest Java update in the releases PPA (we will switch to 1404 as LTS soon, so not a big problem) 4) 64bit Windows machines (win-7-64, win-8-64 and win-81-64) all got the 32bit Java version. Firefox would not see the 64bit plugin at all. (Some of them were already running a 32bit version). So I made sure they all got the same version.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.