Log in

entries friends calendar profile Goode Trouble Previous Previous Next Next
Backups - I Am My Avatar
Big Warm Fuzzy Public Heart
Backups are hilarious. Everybody talks about them and nobody does them. Everybody loses ALL THEIR SHIT and screams and cries and screams again. Humans are funny.

For personal use the answer is pretty easy:

1. Buy a Mac
2. Buy an external USB drive - a simple teeny one that has more capacity than your hard drive and does NOT require a separate power cord
4. Set up Time Machine, which comes with your Mac
5. Keep hooking that drive up for a while every day
6. That's it. Time Machine backs up automatically, you don't have to think about it. As a bonus you can get back to versions of any file as of particular dates. But the main win is that you can restore your entire machine if you must replace your drive or your entire computer.

Sure, the external drive could fail, but your main drive and your external drive aren't going to fail on the same day. It's more than good enough. And it is BETTER than your fancy backup solution that is complicated that you never use.

The only backup that matters is a backup that you actually use.

For professional use it's a little trickier. At work we've come to favor using Amazon S3 for backup of client projects.

For a while we had a Drobo. It was shiny, but too shiny. It is shininess tempted folks to try to set it up as a network Time Machine time capsule... thing... which never worked. Eventually we just bought people individual external drives: simpler is better.

Then we hooked it up in the basement to back up certain projects nigthly. That was better, and it was sweet that it could handle a failed drive so gracefully... until it started to get really really slow and we realized it was old and unsupported and scary to rely on.

We also tried a bargain-basement RAID setup, with two external drives. This was not good. Linux kept changing its mind about what to call the array and the individual drives, no matter how much I worked to lock those things down. It shouldn't be hard to figure out which drive is the dead one.

So we moved to Amazon S3. It's awesome: it's not in your basement, it's geographically redundant (data centers in widely separated states), it's highly redundant on the local level too, and you don't worry about ANY of that shit. You just pay the bill.

Amazon S3 is shiny and you can do fancy things with it, but we try to keep it as stupid as possible: we use a simple command line tool called s3cmd to move things to and from S3. We make an archive file of an entire project and upload that to S3; we keep three weeks of those archives. It's too simple to screw up.

OK, we did do one fancy thing with S3: I wrote a nice nodejs-powered tool to manage the backups.

You can pick up the code here on github. It's nifty, but it has one flaw: it downloads each project locally, archives it, and uploads the archive, then proceeds to the next project. And this takes a long time. So since we have a lot of clients with big projects, we can only do weekly backups, not daily.

So the next generation of this tool will probably do all the work for each project on the server that hosts that project, allowing them all to back up to S3 in parallel. Which means setting up a separate S3 "bucket" for each project, for security reasons. And... other stuff. So for now, we stick with the simple version.

Tags: , , , , ,

6 comments or Leave a comment
waider From: waider Date: January 2nd, 2014 07:40 am (UTC) (Link)
Out of curiosity: why not use one of the S3-backed network drive options to put your time machine backups on S3?
boutell From: boutell Date: January 5th, 2014 04:38 am (UTC) (Link)
That's not a bad idea, but I'm talking about two separate problems really - time machine is a solution for our individual dev macs, while the S3 based solution is for backups of client projects via rsync and mysqldump and yadda yadda yadda and involves a Linux host.
waider From: waider Date: January 5th, 2014 10:08 am (UTC) (Link)
You've reminded me, someone did a Linux time machine clone with a newer version of rsync than I'm familiar with, which allows you to specify a 'previous backup' directory which rsync will sym- or hard- link to (not sure which) in order to minimize disk requirements in the same way TM does.

That is:
First rsync backup copies everything
second and subsequent rsync only copy changes; unchanged files are linked to previous backups of the same file.

For full TM compatibility it'd have to be hard links as that way you can thin out your old backups without destroying the later ones.

Just remarking on this FYI in case it's of use.
boutell From: boutell Date: January 6th, 2014 12:47 am (UTC) (Link)
Hard links are a very cool solution that was underused for a really long time
From: hatter Date: January 2nd, 2014 10:51 am (UTC) (Link)
6a) when you have a few spare dollars, buy another drive like your first one. Run it as your new time machine drive for a few days, then grab your old drive and take it to live with a friend/parents/etc. Your neighbour's place is better than not doing it, but a few cities away is good if you can. Protects from theft, clumsiness and malice; protects from fire/flood/earthquake/The Great One/electrical surges/etc.

the hatter
boutell From: boutell Date: January 5th, 2014 04:39 am (UTC) (Link)
Also a good idea.
6 comments or Leave a comment