The Worst Server Setup Mistake You Can Make

This is a blog about the development of Yeller, The Exception Tracker with Answers

Read more about Yeller here

There’s one server setup mistake you can make that will screw you up for years to come. It’ll haunt your business, your developers and your server admins, and be hugely expensive to fix. It’s surprisingly easy to make. The mistake?

Setting the timezone to anything other than UTC

Use UTC

Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC. Use UTC.

It seems like it’s going to be innocent. What’s the problem with setting my server’s timezone to EST (or your timezone of choice) you ask?

Why Not {PST, GMT, PDT, etc}?

  • if you use a timezone with Daylight Savings (which is most of them), once a year (upon DST change) many scheduled jobs will just “go missing”. Of course, nobody ever runs nightly cron jobs for backups, or sending emails or anything else important, EVER THIS IS A WELL KNOWN FACT
  • DST is different in different timezones, and at some points the overlap will cause a bug when interacting with other systems (I’ve heard of this cause billing delays when one system was billing another with different days and billing was getting rejected because the dates were different).
  • I have (in the past), seen servers where the system timezone is set to PST, but the database is set to UTC, causing more crazy bugs, because for 8 hours a day, calling now() in the database has a different date from the application that talks to it.
  • There are thousands of bugs out there that are caused by timezones
  • UTC assuming code interacting with non UTC assuming code can, and has caused silent data corruption

I’m sure I speak for everyone here when I assure you that I have never made any of these mistakes personally and they have never cost me weeks of my life. ;-)

  • migrating later than the start of the infrastructure can be really painful, because of your app getting coupled to the timezone it’s in.
  • At some point, you’ll want to run servers in different continents, or hire employees across timezones, both of which will be made more painful if your system doesn’t use UTC throughout.

There are millions of stories like this in the wild - debugging timezone related bugs is incredibly painful, migrating off of one of the other timezones to UTC is painful as hell as well.

So, Why UTC?

How? Well UTC doesn’t have daylight savings. It’s universal, well agreed upon (though sadly it still has leap seconds, which can get kinda weird). Nearly every mature engineering organization runs all their servers on UTC, and you should too.

tl;dr

Use UTC as the only timezone for your servers.

Do it first

It’s IMPORTANT that you do this at the start of your server setup. Moving the system timezone is not to be taken lightly. Thankfully, doing it at the start is trivially easy, and saves you from hundreds of problems down the road.

Often servers you rent from hosting providers aren’t set to UTC, so make sure you change this as soon as you set up the server.

Other things you should do

Use bcrypt

(for the record, Yeller’s servers are, and always have been in UTC. Lesson well learned from other workplaces)

This is a blog about the development of Yeller, the Exception Tracker with Answers.

Read more about Yeller here

Looking for more about running production applications, debugging, Clojure development and distributed systems? Subscribe to our newsletter: