Oct 3, 2021
The Largely Untold Story Of How One Guy In California Keeps The World’s Computers Running On The Right Time Zone. (Well, Sort Of)
Down the rabbit hole: my brief odyssey into the esoteric world of the tight-knit time zone data maintenance community who quietly keep the world’s computers from avoiding DST-related-meltdowns
I run a small YouTube channel.
And now and again, I record short videos documenting how to “do” certain things using Linux.
I make them as much for myself as for my 300-odd subscribers.
Because Linux, or rather doing things with it, tends to … you know … be quite complicated. And I can’t always remember how I got X to work three months later.
It’s nice to create documentation I can refer back to and it’s even better if others find it interesting as they occasionally tell me they do. But, for now at least, that’s about all there is to it.
Yesterday evening, I recorded a short video describing how to look up the time zone database (tzdb) to find the right way to denote the time zone on a certain world clock program (gworldclock). I wasn’t expecting that the video would give Netflix a run for its money. It hasn’t. But it has brought me into contact with a world so wonderfully weird that it could well be the stuff of fiction. Thankfully it isn’t.
As most techies know, time zone setting is a fairly elementary feature of computing which most operating systems bake into their graphical user interfaces (GUI). Time zones are attached to locales. Setting a locale is often done on the basis of rough geolocation which users can manually override. Once set, users typically never need to change it unless they move countries. “Ah, I see Paul has been up to his usual tinkering” might be a clever excuse for being late to a Zoom meeting. But it would very seldom be a truthful one (Eggert, the world time zone coordinator, will be introduced later).
Linux, of course, tends to excel in letting users jump under the bonnet and denote changes themselves. That’s largely why I, and many, love it. So we get to see from a closer distance exactly how all this arcane technology works.
Typically, when I’m making my videos, I’ll do a bit of research before setting up shop and hitting the record button. I don’t script most of these, but I like to keep a few notes at the ready.
And this particular production process led me to a series of completely unexpected discoveries sometime in the early hours of yesterday morning.
Perhaps I had simply drunk too much coffee (at most hours of most days I usually have). But I found them all oddly engrossing as they came to me in slow waves of weirdness, one crescendoing after the other:
- All Linux and Mac-based computers pull their time zones from a massively important database — the time zone database. The process of defining time zones is centralized. This is actually quite a big deal in its own right because people tend to grossly underestimate how pivotal Linux is to … the entire internet and technology as we know it. It may constitute a small percentage of desktop users and be an OS largely favored by nerds and computer developers. But in server-land it’s actually the dominant operating system, especially on the public cloud infrastructure that is rapidly usurping the diminishing role that on-premises infrastructure has to play in getting packets of data from hosts to users (in normal language: making the internet work). AWS instances, for instance, default to Amazon’s Linux spin-off. Virtually all the world’s supercomputers used for everything from weather forecasting to simulating physics experiments run on Linux. Android is a fork of Linux. (I don’t play the “will this be Linux’s year on the desktop?” game. People have been wrong too many times and I don’t particularly care either way). If you’ve ever used an Android device, received a weather forecast, or accessed a website (you’ve probably done all those things), then you’ve benefited from the existence of Linux.
- The time zone database — which is sometimes called the Olson data or zoneinfo database — has a fascinating history. Unlike most databases, it’s been deemed interesting enough to have a wikipedia entry maintained about it. Just as curiously, the database has also been the subject of litigation. In fact, it’s been deemed so essential to the operation of computers worldwide that ICANN — which approves top level domains (TLDs) among many vital internet-relation functions — has brought it under its auspices. This generally only happens when something is really fricking essential to the internet. Or more accurately, the upkeep of the database is now the responsibility of the Internet Assigned Numbers Authority (IANA) — which sounds vaguely like the sort of organization that would feature either in an Orwell novel, The Martix, or be the go-to name for an intelligence service looking to create complete stultification about what a front company did (“So … what do you for a living.” “Oh, I’m a time zone specialist at the IANA”. *For the sake of clarification: I’m not alleging that either IANA or ICANN is a front company of any intelligence service although that would probably make good fodder for a bad conspiracy theory).
But wait, there’s more. Lots more, in fact.
You thought time zones were straightforward or even boring or at least eminently controversial?
So I did I. Until last night.
Not only are time zones apparently a longstanding menace for computer developers, but the time zone maintenance community is currently, it seems, mired among some procedural dispute regarding how this essential database should best be maintained. Of course that’s an interesting fact in its own right: there is a world time zone data community.
In fact, The Register recently described them as being no less than “up in arms” about the direction the project was proceeding down. When this saying is being rolled out in reference to a group of middle age computer scientists who discuss time zones in their spare time, you know that something serious must be going down.
The community’s water cooler, apparently is a listserv that looks pretty much like what you’d expect an actual listserv to look like.
It sports the characteristic bald look favored by computer scientists around the world. Its markup unsullied by a single line of CSS. Hyperlinks its most high-tech feature.
But to understand why the time zone data community can prove to be so oddly mercurial, its decision-making process volatile, you have to watch Tom Scott’s video first — no cheating, all the way through.
As Tom points out, the best way to deal with time zones is simply not to deal with time zones by integrating a library into your code. Like hangovers, they’re best avoided at any cost. And like bad hangovers, once you already have one, time zone problems are all sorts of hassle to fix.
The maddening series of problems and exceptions and inconsistencies he speaks of with the air of a man teetering ever closer to the brink of insanity? The ones that thankfully any developer can now avoid by simply integrating the time zone database?
That’s the thankless work of the people maintaining the time zone database. Tom Scott’s video may have driven him mad for five minutes. For these guys, tackling these issues is a career-long endeavor.
As befits the significance of their mission, the minds at the top of the time zone management community are serious guys. Tech titans and heavyweights.
Such as noted Java expert Stephen Colebourne who recently took fire at a recent series of updates to tzdn.
At least, that’s how I understand it. (I asked Colebourne to explain things more simply and he kindly indulged me with an answer; a difference of vision among time zone enthusiasts might be the neatest summary anybody can advance. Colebourne and the rest of the time zone community, I believe, remain on good terms. But you could say that professionally speaking they’re no longer on the same time zone. Sorry, I had to.)
In a blog bearing the utterly fantastic title “Big problems at the timezone database” Colebourne alleges that:
Paul Eggert is the project lead of the timezone database hosted at IANA, a position referred to as the TZ Coordinator. He is an expert in the field, having been involved in documenting timezone data for decades. Unfortunately, he is currently ignoring all objections to an action only he seems intent on making to solve an invented problem that only he sees as important.
Colebourne’s biting critique has already drawn the ire of other time zone mavericks on the Coordinator’s side. They charge that Colebourne is being overly hasty in his pronouncements. He’s jumped the clock, so to speak.
Not only can’t the time zone titans currently agree on the best way to carry the timezone database forward, it seems. But the entire process of codifying and standardizing time zones is also decidedly contentious political business with a long and tumultuous history to go with it.
Those who enter the fray need to be therefore not only technical heavyweights but also prepared to have the occasional audacity to stand up to countries like the Hashemite Republic of Jordan and tell them that their attempt to prematurely end DST is unacceptable and will not be promulgated in the database.
Practically speaking, the time zone maintenance process is tightly controlled and shared with major industry stakeholders (apparently). But ultimately — or at least on paper — the time zone database represents a sort of final arbitration of its decisions. At least as to how time zones are interpreted technically.
Like many other nations — the Germans probably aren’t to be reckoned among them — the Jordanians have had enough with summer time this year and are throwing caution to the wind and moving the clock over one month ahead of schedule. They’re going all YOLO on DST. Again.
But it could be worse.
Weary time zone mavericks are bursting to the seams with horror stories of African states who made rash time zone decisions on only four days’ notice. Warlords and dictators, in the midst of nervous breakdowns, have been known to take out their anger on time zone conventions. Perhaps they’re secretly just trying to troll people like Eggert.
And so time zone politics are an occupational hazard in this little corner of the tech universe. Political leanings — or rather adjudications — are sometimes called for. Those who rise to the heights of this part of the open source community rarely do so without having taken at least one stand on one contentious zone.
The ramifications of these decisions don’t go unnoticed, either. A thread from Hacker News by Y Combinator provides a rare look inside the obscure listservers where much of the coordination for the project happens and shows that those who are most closely involved are acutely aware of the weightiness of the project..
As one user commented in response to another poster raising doubts over whether time zones could really be that big of a deal:
“We have a client in Azerbaijan who was very angry their timezone was shown as Asia/Yerevan. As you may know, Azerbaijan is in a state of cold of war with Armenia.”
And here’s something else I learned: Pre-independence Ireland once operated its own time zone — Dublin Time — which was stamped out by the British as punishment for the Easter Rising.
All this is why the official source code for the world time zone community reads less like a text database and more like .. something you may never have seen before even if you’re the type to pore over with curiosity the commented lines that some developers leave in their code.
I would call it an entirely sui generis form of literature. Part historic intrigue (those lines are commented out). Part actual code that can be read by computers. And part trivia reference.
It is suffused, from file to file, with the sense of heaviness that could only come from men and women who have spent large parts of their life tracking the evolution of time zones and who (one sometimes feels) are lamenting the complete minefield they have waded into. Eggertian I would call it.
These are the people for whom adjudicating upon such matters as France’s attempted move to the decimal system and the International Atomic Agency’s periodic decision to add nuclear seconds to the world time system are the bread and butter that they grapple with at work every day. While we’re planning content marketing or just developing code — on computers with time zones set by their labor.
Time zone data insiders say that every single one of these high stakes deliberations represents a near Y2K disaster that must be averted. If Hollywood thought more imaginatively, it could even be the stuff of blockbusters. Another rogue country has pulled a time zone trick. Better think about a response before it’s too late.
Although most computer users have never heard of the project or its maintainers, the decisions of the time zone maintainer and his cohorts affect the daily computing lives of hundreds of millions of users worldwide.
After all, a keystroke on the Americas file could change EST to CST (or PST).
Zoom meetings could be missed in their millions, vast numbers of transactions incorrectly recorded in ledgers. The consequences of their actions are, frankly, rather enormous.
And the man heading all of this up:
Meet Paul Eggert. The Incumbent Time Zone Coordinator. (TZC). And Where The Buck Totally Stops When Nations Pull Surprise DST Changes. Oh, And Whose Database May Very Well Be Sitting Somewhere On Your Hard Drive.
The most intriguing part of this rabbit hole ( — thankfully it was a brief one, at least comparatively speaking. I spent the best part of a month attempting to figure out the best home internet setup):
At the helm of this project is one individual. One guy.
Paul Eggert, a computer scientist who teaches at the Department of Computer Science at the University of California’s LA Campus.
Eggert’s students speak of him in awed tones. With reverence.
He is said to be an almost impossibly hard grader.
A-grades from the Time Zone King are not to be taken lightly. This is a man, after all, whose codebase helps hundreds of millions of users know what time zone they’re in and who — for the past ten years — has gone to bed knowing that hundreds of millions of computers are using his code to know what time zone they’re in. He’s lived under that pressure for over a decade. And by all accounts thrived.
So having chanced upon this fascinating circus of the absurd, last night, I started a thread on the Today I Learned subreddit — dedicated to sharing interesting things that you learned today.
Posts there are prefaced with TIL and Reddit is full to the brim with geeky technology fans like me.
I was pretty sure that at least a few others would find Eggert’s role as world time zone master intriguing.
Within a few seconds of posting, there were ten upvotes.
And by the time I took the screenshot to embed here, more than 3,700 Redditors had propelled my post sharing the news of Eggert and his project to near the top of the subreddit, which has some 26 million subscribers.
News spread fast across the online community.
The thread was cross-posted to UCLA’s own subreddit (/r/ucla) where some of Eggert’s own students learned that the guy who had been grading their own tests also got up to some weird stuff at night. Not pole dancing kinda weird. To the best of everybody’s knowledge, Paul Eggert doesn’t have a raging cocaine habit or spend his evenings playing strip poker. But he does dedicate time to maintaining the global time zone system.
Redditors — including me — began roasting (or trolling) Eggert. Sorry. It was really hard not to. It still kind of is.
As befits Reddit’s internet culture, Redditors were quick to share a comic from xkcd which depicts a thoroughly fascinating facet of the internet that remains grossly under-explored (time-dependent, I would love to): The fact that vast swathes of the internet depend upon tiny projects and crews — like the work of Eggert et al. — to perform truly essential services for computer-using humanity, which is increasingly almost all of it.
The project that Eggert and a tiny team of cohorts maintain is at the bottom of a stack upon which (literally) hundreds of millions of computers rest. xkcd depicted that visually in this excellent web comic.
It tells a sad history but also an inspiring one. Because — as a somewhat weary technology-focused marketer — that’s exactly how I think about projects like these.
Untold millions have been made by startups announcing dubious advents upon existing technologies heralded with the breathless fanaticism of companies announcing that they have found a way to turn air water into oil. Many of these will vanish into oblivion within a few short years. The time zone database won’t. Because it can’t.
And those at the very bottom of the tech stack — those tirelessly and thanklessly maintaining open source projects upon which so much of the world’s computing derives — languish in comparable obscurity.
Yesterday I learned of the existence of the time zone database (tzdb) which provides authoritative data upon time zone configurations and is used by hundreds of millions of computers around the world every single day (in addition to Linux and MacOS, it can be mapped onto Windows).
I’ve reinstalled the Ubuntu computer I’m typing this post on probably dozens of times over the year.
While many things on the Linux Desktop tend to go either periodically kaput (NVIDIA drivers!) or not work at all, I can’t recall even a single time when the time zone detection program has worked anything less than flawlessly. Daylight saving time even left it unfazed.
Had that not been the case, I might have missed countless Zoom meetings. Perhaps run late to job interviews. Who knows what else might have happened. Without the time zone project, my life may have taken a different course.
In recent years, the project has fallen under the purview of ICANN. Its code reads like a cross between a JSON file and a historical novel.
And while I’m sure the project has many noteworthy contributors, there’s ultimately one guy who’s responsible for maintaining it.
The Time Zone King. His name is Paul Eggert.
And he’s a computer scientist based out of UCLA.
We probably all owe him a ‘thank you’.
I’ll have to add UCLA formally endorsing the “time zone king” alias as one of the highlights of my writing career on Medium:
— With thanks to:
The Time Zone King and everybody who has contributed to the zoneinfo project;
Stephen Colebourne;
