Social Enterprise

Were you hit by the leap second?

It caused more damage than Y2K, and lasted for a solitary second. Were you caught up in this weekend's leap second carnage?

Welcome to the new Australian financial year, the new carbon tax regime and an internet that is shaking its collective head to itself.

Over the weekend, a leap second was injected to realign our clocks with the slowing rotational speed of the Earth.

Unfortunately, for many Linux and Java users, this leap second caused systems to hang and thrash.

Rebooting the affected machines fixed the problem, as did stopping "ntpd" and manually setting the date with
/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date;

Another workaround and discussion as to the root cause is found on the linux-kernel mailing list.


(Screenshot taken by schmidtzl)

Swept up in the carnage were sites such as Reddit, FourSquare, Yelp, Qantas and LinkedIn.

Was your site hit by the leap second? Leave a comment below to let us know. We'd also love to hear what configuration of servers were or were not hit over the weekend.

About

Some would say that it is a long way from software engineering to journalism, others would correctly argue that it is a mere 10 metres according to the floor plan.During his first five years with CBS Interactive, Chris started his journalistic advent...

29 comments
Gisabun
Gisabun

Slept through it. :-)

gurusnord
gurusnord

All our Production Ubuntu servers, primarily Lucid and Precise, running NTP were hit. Anything java related would spin up all cores to 100% utilization. This was very frustrating and in the end rebooted to fix before learning of the simple date fix. I'm surprised this would even be an issue these days with the Linux kernel.

phatkat
phatkat

I have mix of new & old Linux systems here and they fared well over leap-second & all logged that it happened. I also have Mac OS X & Windows of various versions which fared well also, no issues I saw when I came in on Monday. However, I read that on Linux mailing list that some Linux systems had issues that was caused by the ntp daemon was in a endless loop which the solution was either to manually enter the correct date or reboot.

peter_erskine
peter_erskine

So the answer's "no" - where I work there wasn't any problem.

TobiF
TobiF

There's a window every three months for adjusting UTC (calculated time, based on cesium clocks etc.) to be aligned with GMT (Measured time, based on observations on earths rotation. Some times, an additional second is added to UTC, other times, a second is subtracted from UTC, so that UTC, ideally, should never differ from GMT by more than 1 second. This practice has been going on for many-many years. So I don't think that just the leap second in itself is to blame here.

Glitchw
Glitchw

Win 7 Pro "shut down" my netbook to protect it from an "unexpected error". It was streaming a US radio station in the at the time . Oddly enough, it was shortly after midnight EDT, 0400 UTC, 2300 CDT at the stream source. Aparently occuring when my local computer rolled over to the next day. Other computers here (most Win 7), were all OK.

pgit
pgit

All minor. I did notice that one web cam I frequent started reporting the date as Jan 1 1970.

DAS01
DAS01

The leap second has been applied before AFAIK (note for Suresh). A few months ago on (UK) radio I heard a debate about whether it should be abolished and time be allowed to drift apart from the earth's movement or not. There are one or two cogent reasons for doing that. However, my preference at the time was for keeping the leap second and I did not think it would have any consequences, but it does, it seems... I found this interesting nugget about Google on the BBC website: In a blog post written last year, Google explained how it planned to avoid the leap second issue by using a tactic it called a "leap smear". This involved incrementally adding tiny fractions of time - a couple of milliseconds - gradually over the course of a day. "This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day," explained Christopher Pascoe, the company's site reliability engineer. This is as good a summary of the situation and arguments either way as any:- http://www.bbc.co.uk/news/science-environment-16625614

Suresh Mukhi
Suresh Mukhi

I was not even aware that there was such a thing called a "leap second". How would this affect anybody's systems anyway? We have our own internal clock.

jetsethi
jetsethi

Windows Server 2008 somehow jumped back in time 3 hours. Luckily there wasn't any critical services running on it.

linux for me
linux for me

It would seem that the leap second got them too, but that site has been down for a couple of days.

Deadly Ernest
Deadly Ernest

and the only differences between them that i can see, off hand, is the Windows system was the only one that was in constant time checking mode as I usually don't do that, while my son does and his Windows system locked up on him for no discernible reason, and that is the only thing we can think of that may have caused it.

wizard57m-cnet
wizard57m-cnet

it was my little Acer Aspire One netbook, I had it booted into Slax via USB, and the system clock got all confused and jumped ahead 12 hours or so! Set the clock, rebooted and all was well.

Nobby6
Nobby6

Our Slackware servers running 13.1 and 13.37 were not affected.

Deadly Ernest
Deadly Ernest

weren't affected by it at all. But it will also depend on how tight their timing sync is set and the coordinating server.

Suresh Mukhi
Suresh Mukhi

For this information. A second can make all the difference.

rocket ride
rocket ride

Someone at Google deserves an extra banana or three for that one. Just sayin'. None of my home systems or the ones at work seemed to notice a thing.

Deadly Ernest
Deadly Ernest

each other if the times cease to match or get out by too much.

mark.cooper
mark.cooper

All our Windows servers were not affected.

pgit
pgit

I'd roll back to 1970 and do it over if i could...

JCitizen
JCitizen

That was a good post! :)

khiatt
khiatt

I have a server that keeps time for the network. Every time a user logs in, the computer time is synced with the primary server. That server updates it's clock once a week. Never had a time related issue. Even the antivirus software doesn't care unless it's more than an hour off. Granted, I don't have any extremely time sensitive functions that require absolutely perfect synchronization, but a web server crashing because the time changed 1 second?! How do they survive Daylight Savings Time, and why doesn't the computer world end on a leap year? :)

Deadly Ernest
Deadly Ernest

I've seen some systems set to allow a few minutes variation work, and others crash when there is NO TOLERANCE at all set in the system. I try to avoid such set ups as much as possible in order to reduce the workload of dealing with such issues.

JCitizen
JCitizen

that when you provide your own domain and server it is not as much of a problem; especially inside your LAN. But if communicating across long distances happens; it can be a problem for devices that rely on hand shakes with different devices, and even the speed of light/electricity can have an affect on this process. Timing is critical, obviously or the atomic clock would not be a standard for the web. In this instance, the movement of the earth becomes more critical than exact timing; therefore the constant need to adjust to reality instead of science in a perfect vacuum.