Crowdsourcing My Media Management Blues Away

This is is not a blog post about sex. (Nor yet is it a song about Alice.) Nope, it’s a blatant attempt to harness the brains and experience of my loyal readership in an attempt to solve some of my digital media management problems.

Short version: I want software suggestions for library management software across several flavors of media. My wants are specific.

Long Version: There’s a category of software out there that helps you manage your media files. iTunes is probably the most well-known such piece of software. It will (or it would last time I used it, years ago) scan your hard drives for music files, let you sort them in various ways, and (this was the part I really liked) help you clean up the metadata — regularize the file names, sort out the seven different ways an artist was spelled and capitalized, import new and better metadata for your files from external web sources, provide you with cover art, all that sort of goodies. In short, even if you ignored the publisher’s main purpose for the software (selling you $.99 songs) it was really a handy and nifty piece of software. (Presumably, still is.)

I don’t use iTunes any more because, for awhile there, it was buggy and sluggish and prone to crashing on my Windows XP machines. And, because I never got into buying music online. At some point I had a system crash, and when I reinstalled aps, iTunes didn’t make the cut. Time marches on.

Time marches on, we learn stuff. One of the things I’ve learned is that software from open source projects is always better for my needs. Blogging software is what really taught me the lesson, but it’s widely true for me: if I can find a piece of open-source software with ongoing development and a substantial user community, the quirks and ease-of-use issues (like WordPress’s damnable fetish for frequent near-mandatory upgrades that aren’t reliably backward-compatible with its older installations) are massively outweighed by the facts that development is rapid, solutions are usually available from the user community, and data migration (both in and out) is almost always possible (usually, easy).

So these days when I go looking for software to solve a problem, I’m invariably looking for something that is free and open-source and is in active development. Doesn’t need to be the category leader, but it does need to have a critical mass of fans and enthusiasts.

So, moving on, last year I bought a Kindle. And it’s a nifty little thing. I was afraid the device wouldn’t “go away” in my hands, vanishing from consciousness like a paper book does while I’m escaping into it. But it does. Kindle really works for me as a reading device.

But I find that I don’t use it much in the “push the button, send money to Amazon, buy a sharply-restricted electronic book for almost as much money as I’d pay for a real book” mode that it was designed to stimulate. Instead, I wind up using it to read a huge variety of prose that’s “out there” (in a profusion of formats) that I would formerly have planned to read on my desktop computer, but never would have actually gotten around to reading (because I already spend too many hours sitting on my fat ass in front of a computer). Now, I upload all those files to my Kindle, carry it everywhere, and read where I’m comfortable — including, at the end of my day, in bed.

It’s great. However, like most proprietary devices, the Kindle’s ability to handle “a profusion of formats” is pretty limited. Early on, I realized I needed “an iTunes for ebooks”. I wanted something that would keep all my reading files in one place, help me regularize the names and manage the metadata, and ideally convert a wide variety of file types (html, .pdf, .lit, .doc, .txt, and more) into formats that my Kindle can handle (basically, .mobi).

I went looking. I was in luck. I found Calibre.

Calibre is a huge and free open-source software entity that does a lot of things, not all of them pertinent to my purposes. And it’s got some hairy scary bits pretty close to the surface (as for example when it invites you to fine-tune huge nasty long regular expressions if you go looking at the menus that govern how it converts files between formats. Trust me on this one, boys and girls — you don’t want to mess with regular expressions unless you can think like a programmer. It is not for “mere users”.)

But in the mansion of robust open-source software, that sort of grues in the closets are just a price you pay for free and luxurious digital living. And what luxury! Calibre’s library management function lets you pull in just about every ebook format under the sun. It offers a bunch of tools for cleaning up, regularizing, and editing the meta-data (title, author, ISBN, publisher, series, cover art, whatever) and can connect you to online databases where you can get more and better metadata for your ebooks. Once you have all your books in a nice pretty list with the author names all spelled the same and such, you’ll discover that Calibre has been very friendly about what it did with your old dirty files (left them alone, right where it found them) and how it stores your new clean files (in a new directory that automatically gets updated as you make changes within Calibre, sensibly organized alpha by author, with metadata stored in the files, no ugly database files on your hard drive to take up extra space and get corrupted).

This “library management” function is just one of the three key parts of a good piece of media management software. Calibre also offers a ton of very robust format conversion options. Here, the grues get hairier fast and have bad breath, but the bottom-line functionality is impressive. If you can get there from here, Calibre will get you there. Pretty much any readable file I’ve found, Calibre has turned into a legible .mobi for my Kindle. (The exception is old .pdfs, the kind that are just images with no text data in them; Calibre will try to break them up into page sizes that aren’t too small to read on Kindle’s small screen, but it doesn’t always work well enough for a pleasant reading experience. Yes, there are grue-infested options for pursuing that problem; my failure in select cases says more about my own limits than about the limits of this particular software.)

After library management and format conversion, the final thing you want in a piece of media management software is a way to sync your files up with your devices. In the case of Calibre, this means it needs to be aware when you plug in your Kindle (or whatever e-reader device you use, Calibre supports many/most), show you what’s on there, and make it easy to move files from your library onto your device. Which Calibre does, although not (unless I just haven’t discovered those options because I don’t want them anyway) automatically, the way iTunes did back when I used it.

So, that’s my plug for Calibre, presented here because I’m trying to describe what I want by example.

I’ve got movies on my hard drive, some dirty, some not. Their metadata is a mess! I want a Calibre for movies! (More on this later.)

I’ve got music on my hard drive, lots of it! Some of it dates back to the original Napster era. Lots is in folders I ripped from my CDs before I put them in deep storage. The metadata is all over the place. I want a Calibre for music, one that’s not iTunes and is open source.

You’d think I could just Google for what I want, using key phrases like “like iTunes” and “open source” and “library management” and “media management” and “metadata”. And I’ve done that. But man-o-man, there’s a ton of software out there! It’s not well-described, so you have to download it to try it; and most of the programs that look promising don’t have all three parts: library management, format conversion, and device syncing ability. Or, if they do, there are fundamental flaws. For instance, it’s common to discover that the library management function insists on editing the metadata on your original files, or stores your improved metadata in some bloated and ugly internal database where it does nobody any good. You can waste weeks — and I have! — trying different alternatives and rejecting them.

So anyway, music is the least important to my media life — when I listen to music at all, I generally just have satellite radio running in my background. But I’d love to have an open-source program with iTunes-like functionality. It’s been my experience that some of the features I’m looking for are built into many of the programs that used to be called “MP3 player” software; but the ones I’ve looked at either aren’t device aware, haven’t been good at format conversion, or haven’t done a good job of handling and storing the cleaned-up metadata. Is anybody using anything that does all that? I’d love to hear specific recommendations.

Likewise, movies. Movies have a ton of formats and most of those formats have really nasty metadata by the time they’re sitting in some random folder on one of my many hard drives. This ugly metadata situation is so chaotic, I mostly don’t bother with looking at the movies on my computer any more. Indeed, I gave The Nymph my video iPod years ago because I couldn’t be bothered. (I now realize that I was frustrated by the then-primitive functionality that iTunes had for handling video metadata, but at the time, I didn’t know what my problem was, I just knew it was a bigger pain in the ass than I had time for.) But if I could get a handle on the metadata of my movies and get them organized into some sort of decent video library, it would be worth getting another portable video device (or tickling The Nymph without mercy or surcease, until she gives me back my video iPod).

In recent weeks, I’ve actively been working on the movie problem. Every time I use Calibre to clean up a bunch of files and load them neatly onto my Kindle, I get inspired to go trawling the web for some software that will help me get my movies under similar control. In the process, I’ve downloaded and tested quite a few promising-sounding software packages, but nothing has come close.

For example, I looked at Miro, which is an amazing and ambitious open-source piece of video player software that also makes it really easy to get at and download a wide variety of video from internet sources. It has a robust development community, so what I write about it now may not be true six months from now; but the version I looked at (2.5.4) is awfully primitive about handling metadata for the files already on your hard drive. It will scan them, thumbnail them, and let me play them, and maybe let me adjust a filename with a right-click menu option, but it doesn’t have (make that “doesn’t seem to have”, it’s always possible I’m an idiot and missed a whole set of features) any robust features for library management. It’s a great movie player and a superb way to suck in video from elsewhere, but it’s not — isn’t trying to be — an iTunes or Calibre for movies.

Another program I looked at is called XBMC. XBMC is visually stunning, and designed to be; because it’s intended to be the visual interface on your home theater PC, the software you can use with your remote control while sitting on your couch, to watch (on your big living room TV) all the movies that you’ve got available on your local computers. Like Miro, it’s a really excellent movie player for your PC; but also like Miro, it (so far) is seriously lacking in the library management functions. I’m a little bit less confident in that assertion than I’d like to be, because I found the XBMC interface confusing, and I’m not sure I ever found all the features. I did find hints of a complicated way to set up formulas for renaming all your files in useful ways while scanning them into your movie library, but no easy way to handle the metadata cleanup chore. (Despite the lack, I came way from the experience wishing I actually had a media PC hooked to my big TV in the living room.)

There are quite a few more open source and feature-heavy apps out there that deal with video files, and I’ve looked at a fair few. But so far, I haven’t found anything that does the basic iTunes/Calibre library management functions with ease and panache. Video format conversions are hard, and I’d be willing to overlook gaps in that functionality; and since I don’t have a portable video device at the moment, the ability to sync with one is at the bottom of my personal feature list. Once again, suggestions would be very very welcome, especially if they came with enough commentary to guide me. “Have you looked at MovieWhizzer?” will not help me very much; a comment that would help a lot more would be “I use MovieWhizzer, and although it doesn’t sync with devices, it’s an ace at file conversion and metadata handling, though it does store the metadata in ugly semi-encrypted FreeDumbFlatBase files”. You get the idea.

My thanks to anybody who read through this whole entreaty, and more thanks (heaps of ’em!) in advance to anybody who can point me in useful software directions. Crowdsource me, baby!