769972 - Java is choking on leap second.

Status: RESOLVED FIXED

(Mozilla Metrics :: Metrics Operations, task)

Description

[Eric Ziegenhorn :ericz]

Servers running java apps such as Hadoop and ElasticSearch and java doesn't appear to be working.  We believe this is related to the leap second happening tonight becuase it happened at midnight GMT.

Comment 1

[Eric Ziegenhorn :ericz]

Elevating to blocker.  I believe we need to restart Java everywhere, and possibly reboot servers but need some feedback from Hadoop owners, etc.

Comment 2

[Corey Shields [:cshields]]

opening this bug up

Comment 3

[Pedro Alves]

Still needs to be confirmed, but I was able to fix one of the issues with an elasticsearch server that I have installed by manually adjusting the date "date --help" (there was a service restart involved, but no reboots)

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

We are updating kernels and rebooting HBase clusters right now.

Comment 5

[Ricardo Pardini]

For those machines that shouldn't be rebooted:

/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;

Then restart affected Java applications.
This stops ntpd, sets the date manually to the current date, confirms it.
You may or may not get the bug back after restarting ntpd.

Comment 6

[Corey Shields [:cshields]]

(In reply to Ricardo Pardini from comment #5)
> For those machines that shouldn't be rebooted:
> 
> /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;

we are injecting this fix into all systems through our base puppet module as we speak

Comment 7

[Mina Naguib]

For what it's worth, we've stabilized our java apps across our servers simply via:

date; date `date +"%m%d%H%M%C%y.%S"`; date;

The CPU of the JVMs drops instantly when that is run. There was no need to stop/restart ntpd nor the JVMs themselves.

Comment 8

[Ricardo Pardini]

(In reply to Mina Naguib from comment #7)

> The CPU of the JVMs drops instantly when that is run. There was no need to
> stop/restart ntpd nor the JVMs themselves.

I've got mixed results, some machines go back to 100% when ntpd is restarted, some don't. Out of uncertainty, I'm keeping ntpd stopped for now. I will bring some back online later and report.

Comment 9

[Ricardo Pardini]

> I'm keeping ntpd stopped for now.
> I will bring some back online later and report.

I've brought ntpd back online now on all my servers, and it seems stable.
It definitely caused the CPU issue to reappear some time before, but no longer.

Comment 10

[Eric Ziegenhorn :ericz]

Socorro and most Hadoop stuff is back up.  Everything else hadoop-related can wait until Monday.

Comment 11

[Corey Shields [:cshields]]

For reference, this is the fix that is getting pushed out:  http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/