The fall  

tl;dr — I bought a drobo then dropped the hard drive I was going to copy onto it. Got emotional on twitter about it.

  • visit your friends
  • see their big tv, which is not a tv, but a screen, and the drobo blinking beside it
  • they call it a drobo, you complain about space
  • when they give you a spare hard drive, take it home and say thank you
  • look at your pile of drives
  • listen to the fans
  • add this new drive to the pile
  • believe Jason when he mentions the drobo is good
  • google the drobo thing
  • look at the drobo site
  • gasp
  • quit your job (unrelated)
  • spend hours moving your files around
  • wonder what you’ve done with your life
  • see it’s all garbage, but there are some things
  • look at the drobo site
  • gasp again, doubly
  • remember Aaron said something about selling his drobo
  • ask Aaron if he’s selling his drobo
  • be grateful Aaron says his drobo doesn’t work well
  • remember the internet is about not paying full price.
  • look at amazon, it is cheaper
  • add a five-bay drobo to your amazon cart
  • close the window
  • return to amazon, look at your cart
  • add a book you want, like you’re just buying a book
  • close the window again
  • jason scott will tweet about newegg
  • newegg is not amazon
  • not amazon is better than amazon
  • you will order from new egg
  • you will panic momentarily, but tweet about it
  • when no one tweets that you are stupid, you will close twitter.
  • wait some days
  • get the drobo
  • realize the drobo wants empty drives, not full ones
  • play a shell game of moving things around drives until each is empty
  • start with the smallest
  • put each empty drive in the drobo, one by one
  • call them their sizes: 500gb, 750gb, 1tb, 2tb
  • when you are done, see there is really not so much space
  • there is still one more drive, which is biggest and still two-thirds full
  • see the problem
  • try copying some files around, the very big ones
  • leave copying for overnight
  • when it fails, try again
  • there’s still not enough room, think some more
  • if there were a place to move these three terabytes, then there would be room in the drobo
  • for hours, keep shuffling
  • do the math
  • take a trip to another city
  • return home and tell yourself this trip was not so expensive
  • google more hard drives
  • order another big hard drive
  • do some things because it is the weekend
  • realize that nothing ships sunday
  • see that for twenty dollars, you can get it on tuesday which is sooner
  • look at bank account
  • look at shipping options
  • do it anyway
  • wait (but not as long)
  • work all day and all night until it is tuesday
  • check the tracking
  • out for delivery
  • you have not slept, but that’s unimportant
  • hear the sound of a truck
  • hear the truck drive away
  • wait some more, until later
  • see the fedex truck and run to the door
  • hear the buzzer
  • run back
  • tell him one second
  • be thankful and smile
  • run up the stairs
  • take a picture of the box
  • unpack the box
  • take a screenshot of drobo dashboard
  • unmount all the hard drives
  • unwire the firewires, unplug the plugs
  • start turning around the drobo
  • bump into the thing
  • what is the thing
  • this time is so slow
  • the thing it is falling
  • the drive it is falling
  • the drive that is spinning
  • the drive with the stuff
  • the one with the stuff
  • stick out your foot
  • foot under the falling
  • the crunch of the thing
  • oh shit
  • wait a second
  • no worries, don’t worry
  • unwrap the new drive and it goes in the drobo
  • now there is room
  • plug it in
  • it is beeping
  • why is it beeping?
  • is it really beeping?
  • tweet about it. tweeting will help
  • delete the tweet.
  • unplug the drive
  • put the drive in the freezer
  • tweet some more
  • delete those tweets, too
  • take it out of the freezer and plug it in
  • it will beep
  • and keep beeping
  • until you unplug it
  • now there is space
  • you have some of the files
  • when you sleep, you will wonder
  • what has been lost?
  • and what would it cost?
  • and does it all matter?
  • and when will you know that something is missing?
  • and what of the rest?
  • and you will dream
    that the knowing
  • was always out of reach
Ring toss  

(I’m going to edit this post a few times. Do not yet consider it complete.)

I get the Believer magazine as a couple of files in Dropbox, including a PDF (for reading/hightlighting in GoodReader on iPad or Preview on Mac) and an IDML export of the original InDesign file.

Some non-printing characters are find&replace-d in InDesign

Each article, hed, and footnote is clicked, Cmd-A, Cmd-E, saved as an RTF to a folder in Dropbox called Export RTFs. Filenames are important.

Each of those files is selected.
Filename is copied.
File is dragged onto an Automator script generously concocted by @waldojaquith that basically turns these gears:
— uses textutil to export from RTF to HTML
— normalizes the linebreaks with BBEdit
— tidy on the textutil util
— runs a bunch of Find&Replace for the markup texutil keeps (a bunch of spans and stuff)
— concatenates
— BBEdit opens with marked up text.

textutil -convert html -stdout "$1"
tidy --wrap 0 --drop-font-tags yes --output-html yes --show-body-only yes --drop-proprietary-attributes yes --char-encoding utf8 --logical-emphasis yes --vertical-space yes --show-warnings no -quiet; exit 0

Cmd-S to save; Cmd-V to paste the filename; Cmd-W to close;

Repeat for each file. (Yes, this should be a bash script or anything else, but given that this whole thing involves ~ 50 files and only once/month, it’s oddly meditative and quick)

Open all those HTML/TXT files in BBEdit at once.

A lot of regex/grep replaces in specific files and for specific errors the Applescript hasn’t yet picked up on.
Here are the those grepfors: https://gist.github.com/71f25235d290d3cb849b

That’s where I’m up to today.
– I have to read the whole PDF to look for smallcaps.
– Can’t yet batch the whole folder
– Some of those replacements in grepfor can be rolled back into the Automator
– Automator should become a python or shell script

From there, each snippet file gets read and re-read a number of times to iron out the flaws.

Once OK, I copy/paste all the parts into the database through an app called Sequel Pro, which @kissane told me about. Before that, it was all phpmyadmin, which had a habit of actually timing out. (Also, scary how hard it was to back up.)

Screenshot of Sequel Pro on a large screen.

Then some PHP stuff happens that actually generates a bunch of flat files. Scott (my predecessor) built this and it’s actually pretty amazing. He pretty much wrote an equivalent of Movable Type from scratch (this was 2004 or something!)

Recently we added CloudFront to take some load off the hosting and speed things up.

I dream of moving this to Casein or WordPress.