Closed Bug 1083130 Opened 11 years ago Closed 11 years ago

14/10/14 - Update CI nodes for Java, Flash and MS updates

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

Product:
Mozilla QA Graveyard
Component:
Infrastructure
Version:
Version 2
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: AndreeaMatei, Assigned: AndreeaMatei)

References

Details

Attachments

(1 file)

multiple-crashes.jpg
153.32 KB, image/jpeg
Details
Henrik, do we want all of these right now or should we wait a bit? (for Java 8 maybe). I'll start on staging after your ok.
Flags: needinfo?(hskupin)
At least the MS updates are scaring me off. So lets wait some days and observe possible feedback on those. If all is fine, I don't see why holding off from updating the nodes. Java 8 is the way to go for sure! And maybe the Flash update can fix the crash we see on the Vista machines?
Flags: needinfo?(hskupin)
Hardware: x86 → All
Summary: Update CI nodes for Java, Flash and MS updates → 14/10/14 - Update CI nodes for Java, Flash and MS updates
The crash unfortunately is not only on vista, it's on windows in general, but we've seen it most on vista and win 7. I'll try on staging the new release and if we still get crashes tomorrow I'll update to debug. There are way too many crashes lately, some without .extra file :(
Uhm, it appears the archive for this release is not up yet, in order to get the normal and the debug version: http://helpx.adobe.com/flash-player/kb/archived-flash-player-versions.html I see the last release is and for 14th (this release), there are some older versions. (Released 10/14/2014) Flash Player 11.2.202.411 (32.39 MB) (Released 10/14/2014) Flash Player 13.0.0.250 (172.47 MB) (Released 9/9/2014) Flash Player 15.0.0.152 (Win Firefox and Mac) (132 MB) (Released 9/9/2014) Flash Player 15.0.0.167 (Win IE only) (33.78 MB)
Assignee: nobody → andreea.matei
Status: NEW → ASSIGNED
Updated OSX on staging for Java and Flash (except 10.6).
Updated remaining machines: Windows and Linux except 1204-64 - which is offline due to bug 1038646 and 14.04 machines. Note: * Win 8.1 32 - For flash debug, the option in settings to disable notifications was checked at Notify me to install updates, but all those were greyed out, unable to change anything there. * Also, for 8.1 machines (both 32 and 64), the older versions don't get uninstalled with the script, I had to uninstall them manually. * Linux 13.10 has Java 8.11 (not .25), was the only one available I'll have to wait for the flash window on linux, hope they'll appear until I leave.
(In reply to Andreea Matei [:AndreeaMatei] from comment #6) > * Win 8.1 32 - For flash debug, the option in settings to disable > notifications was checked at Notify me to install updates, but all those > were greyed out, unable to change anything there. You might need extra admin previligies you can request by using the context menu. Or open it via a admin console. > * Also, for 8.1 machines (both 32 and 64), the older versions don't get > uninstalled with the script, I had to uninstall them manually. If this doesn't work anymore we should get this fixed. It's not helpful to keep around brokeness and do everything by hand. > * Linux 13.10 has Java 8.11 (not .25), was the only one available 13.10 gets updates? I didn't expect that. > I'll have to wait for the flash window on linux, hope they'll appear until I > leave. I have seen some others on a production machine yesterday! So this check should be done again 1 or 2 days after the upgrade again.
Not sure where else to put this. Here's a breakdown by machines of all (reported) crashes in the last 4 days (oct 17's _might_ only be partially captured): 17-20 oct = total 87 crashes ========= 4 x mm-win-81-32-1 3 x mm-win-81-32-2 1 x mm-win-81-32-3 4 x mm-win-81-32-4 2 x mm-win-81-64-1 2 x mm-win-81-64-2 6 x mm-win-81-64-3 3 x mm-win-81-64-4 3 x mm-win-8-32-1 4 x mm-win-8-32-2 1 x mm-win-8-32-3 4 x mm-win-8-32-4 5 x mm-win-8-64-1 2 x mm-win-8-64-2 2 x mm-win-8-64-3 1 x mm-win-8-64-4 5 x mm-win-vista-32-1 3 x mm-win-vista-32-2 4 x mm-win-vista-32-3 3 x mm-win-vista-32-4 3 x mm-win-7-32-1 4 x mm-win-7-32-2 1 x mm-win-7-32-3 4 x mm-win-7-32-4 6 x mm-win-7-64-1 5 x mm-win-7-64-2 2 x mm-win-7-64-3 3 x mm-win-7-64-4
Attached image multiple-crashes.jpg —
I wanted to submit them, but for every reported crash we have, I see we have 12 crash reports waiting in about:crashes. If the ratio holds for the rest of the machines, this yields a total of 1044 crash reports. Here's one of them: https://crash-stats.mozilla.com/report/index/4cb684a4-188e-4fed-9ff8-d601c2141020 I will assume most (if not all) are related to the Flash issues, and that they'll be fixed by the reintroduction of the Debugger version. It's not manageable to manually submit 1044 reports across 28 machines.
This are crashes for Flash, yes! I thought that Andreea already reverted the Flash installations on the Windows machines to use the debug version of Flash!? We talked about it already a week ago. If that hasn't happened yet this has to be done ASAP. Given that we dont want to use the latest Flash player for that but what has been installed on the other machines, it would be a replacement. Who will take care of that? Andreea? Shall we file a separate bug?
Flags: needinfo?(andreea.matei)
I have installed on staging so far, the debug version from the latest flash as provided by Jeromie. Finished on Friday and I checked yesterday the results, all was fine, but haven't got a chance to start on production and today we have beta so I have waited as we established before regarding beta days. I'd say we use this bug going forward, I'll start with Windows in the morning. I don't expect those numbers to be real crashes (meaning those that can be submitted) as most of them are probably with no extra file.
Flags: needinfo?(andreea.matei)
Ok, sounds good then. And in regards of the .extra file, we can always expect that this file can be missing in case of Flash crashes. So it's a very good indication.
Counted 61 crashes in todays runs (across beta & regular daily builds)
Can someone tell me why we are never updating the jenkins master machines? Both haven't gotten upgrades for 133 days! Also it's running 12.04. We have to upgrade it to 14.04 so we get further updates.
I updated staging for all the latest updates, so we can check if something helps here with the HTTP session issue.
Something is totally broken with the Java installation on staging Windows nodes. The icon on the desktop does no longer point to the javaws executable. Checking more we have a ProgramData folder under C:\ which has some single jawa executables, but nothing else. Is anyone verifying the installation of tool before reconnecting machines?
Some more things here... * Can we please stop installing additional software on the test machines? I mentioned that a couple of times and now I see Notepad++ installed again on the Win7 32bit node. * Who replaced the Jenkins shortcut on Windows XP staging with a URL shortcut? Thats nothing which ever has been discussed what we want to do. This should be a command shortcut. * On Windows machines Microsoft Security Essentials are active? This should have been disabled across all boxes. I disabled real time protection now. * All Java installations at least on Windows have automatic updates enabled! I assume that we have to be more careful in case of system updates! I believe that someone simply installed every available update and didn't uncheck those we do not need. This might be the reason why the Security Essentials were active again. So lots of things are in a strange state, which should put us in a state of updating the documentation for update steps.
Flags: needinfo?(andreea.matei)
Finished windows on production, besides XP where I only installed flash debug, all have java as well. Flash is certainly with updates off, for java i'll have to recheck tomorrow as per previous comment (maybe something is wrong with the script there too, as it is with uninstalling older versions).
What I have just seen with the script is that in some cases it cannot import the regex patch directly from the mapped network drive. Maybe we have to copy the files first to the local disk.
So I would suggest that we try to meet up tomorrow and work on updates for the Mana update documentation. It would be good to know what is not clear from your side.
We had no crashes today, this update fixed the problem.
Great to hear! That will lower the amount of work for duty.
We had a meeting to update the documentation and clear remaining questions. I wanted to continue today with OSX and Linux updates, but we have 33.0.2 release and tomorrow beta, so most likely on Wednesday. Depending on when the runs will be done today, I might get to check the windows machines in the evening, to remove essentials and recheck autoupdate on Java.
Flags: needinfo?(andreea.matei)
Todays problem with very slow build processing was clearly the security essentials situation. It's known for months that this causes problems, and we talked about it last Thursday. Since today this tool has not been disabled. So for most of our builds we run an extra time longer. I disabled the realtime protection for all Windows machines except Windows 8 and 8.1, where it is not installed. I would suggest that we really get this disabled, and never ever installed again. Please do that as soon as you can.
I talked with Marc yesterday, and we wonder if we should keep Security Essentials installed but disabled. If we do another OS wide update, it may not get installed and will not be active. So when disabled it usually should stay disabled even signatures are getting updated.
Updated Flash and Java on OSX machines.
As mentioned in comment 17 the shortcut links for Jenkins are broken on Windows. This has not been fixed yet for a good number of slaves. Andreea, can we please get this fixed ASAP? Thanks.
Flags: needinfo?(andreea.matei)
Btw some of the Windows 8 machines showed me the modal overlay for the Windows 8.1 offer. I disabled that via gpedit.msc and will update the documentation appropriately.
Updated ubuntu machines. Tomorrow morning I'll recheck them for the updates window and then go through all windows to fix autoupdates for java where the script might have failed and the jenkins icon.
Flags: needinfo?(andreea.matei)
(In reply to Andreea Matei [:AndreeaMatei] from comment #29) > Updated ubuntu machines. The update for Java 8 is not done yet on the Ubuntu boxes. When you switch to a new major version you really want to remove and purge the old installer: $ dpkg -l | grep oracle ii oracle-java7-installer 7u72+7u60arm-0~webupd8~2 Oracle Java(TM) Development Kit (JDK) 7 ii oracle-java8-installer 8u25+8u6arm-1~webupd8~1 Oracle Java(TM) Development Kit (JDK) 8
Flags: needinfo?(andreea.matei)
Solved that on al ubuntu's on production. Going to staging now. What's left here for later (after the release testing) is to recheck windows for the autoupdate java and the jenkins shortcut on the desktop. Maybe also have a look on essentials, to see it's disabled.
Flags: needinfo?(andreea.matei)
Checked windows machines except win vista, win xp and win-8-32-1, win-8-64-1 - I'll continue in the morning.
All machines checked, also on staging, XP has Java 7 as it's not supported anymore and the autoupdate couldn't seem to get disabled with java 8.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: