Managing an iBooks Metadata Mess
You’ve downloaded a shiny new EPUB from somewhere that isn’t Apple’s rechristened iBooks Store, and it’s on your Desktop in OS X 10.9 Mavericks. You drop that EPUB file on the iBooks icon, the book opens, and you read it with pleasure. Then, one day, you look at the Categories view in iBooks and you notice that the EPUB is part of an ever-growing category known as Uncategorized. This astonishes you because the same EPUB, when you synced it from your Mac to your iPad through iTunes, shows up categorized as Fiction. What gives? Why is it Uncategorized on your Mac and Fiction on your iPad?
It’s because of an odd bug in iBooks. It all has to do with metadata (embedded information about your ebook), and how iBooks handles that metadata for EPUBs that originate from outside the iBooks Store.
(Things are going to get geeky here, so if that frightens you, skip down to the end of the article where I explain how to work around this bug.)
EPUBs and Metadata — Some months back I wrote a piece about managing book metadata in iTunes, “Managing Books in iTunes: This Novel Has a Nice Beat,” 14 January 2013. In it, I described two files that, between them, control how iBooks lists a book.
An EPUB is actually a specially constructed .zip archive that contains a bunch of other files. One of the two files that I talked about can be found inside every EPUB: the .opf file. Among other things, the .opf file contains a bunch of metadata statements — that is, specific bits of code that describe the EPUB. One of these (or maybe more than one, but only the first one is recognized by iBooks) describes the EPUB’s subject. For example, a statement like <dc:subject>Fiction</dc:subject> says that the EPUB is Fiction. Ideally, when the EPUB ends up in iBooks, it will be
assigned to the Fiction category.
iBooks, however, doesn’t use the raw metadata in the .opf file for assigning the EPUB to a category. Instead, it uses a special file designed by Apple for use in iBooks (and, as you might guess from its name, in iTunes). This file is named iTunesMetadata.plist. The iTunesMetadata.plist contains a bunch of metadata entries about the EPUB. One of those entries, named “genre”, is the one that iBooks uses to assign an EPUB to a category.
But, you may be thinking, where does that iTunesMetadata.plist come from? Good question. When you buy an EPUB from the iTunes Store, the iTunesMetadata.plist is already inside the EPUB. However, for EPUBs that come from other sources, there is no such file. Before Mavericks, iTunes would make this file when you added the EPUB to your book library in iTunes; what is interesting is that in this bright new shiny iBooks-on-the-Mac era, iTunes still does this job!
When you add an EPUB to iBooks from a source other than the iBooks Store, iBooks sends out a call to iTunes to look into the EPUB’s .opf file, scavenge through the metadata as best it can, and use what it finds to create an iTunesMetadata.plist for the EPUB and fill it with data. So, if the EPUB contains the statement <dc:subject>Fiction</dc:subject> in its .opf file, the “Fiction” part of the statement becomes the “genre” in the iTunesMetadata.plist file that iTunes creates for the EPUB.
Great! Except for that bug I mentioned. And here it is: iBooks apparently assigns a newly added EPUB to a category before iTunes has a chance to create the iTunesMetadata.plist file and let iBooks know about it. And, since the EPUB has no iTunesMetadata.plist, and hence no “genre” entry, at the time the EPUB is assigned to a category, it ends up being categorized as Uncategorized.
However, when iTunes finishes its iTunesMetadata.plist construction project, the EPUB in your iBooks library on the Mac does have the file. Consequently, the next time you sync your Mac’s iBooks library with your iPad (and you do that via iTunes, so, even if iTunes wasn’t running when you added your EPUB to iBooks, it can build the file when you sync), the EPUB that you sync to your iPad contains the iTunesMetadata.plist file, and the book shows up on your iPad categorized as Fiction.
And that’s why your EPUB is Uncategorized on your Mac but is Fiction on your iPad.
The Workaround — Because iBooks 1.0 on Mavericks lacks the Get Info command that iTunes has, you have no way of editing the “genre” for the book to fix it. However, here’s what you can do to make iBooks on your Mac display the EPUB in the proper category.
First, make sure that iTunes is running. Next, open the new EPUB in iBooks. This causes iBooks to make a copy of the EPUB, store it deep inside your ~/Library folder, and to ask iTunes to add an iTunesMetadata.plist file to the book.
Next, drag the EPUB from the iBooks Library window to your Desktop. This causes iBooks to make another copy of the EPUB that it just created and put it on your Desktop. At this point you have three copies of the book: the original (with no iTunesMetadata.plist file), the copy inside of your ~/Library folder (with its new iTunesMetadata.plist file), and a new copy on your Desktop (also with an iTunesMetadata.plist file inside).
Now for the switcheroo: Select the EPUB in iBooks (make sure you are looking at All Books view), and choose Edit > Delete (or just press the Delete key). iBooks asks you if you are sure you want to delete the book from your Mac. You are, so click Delete.
Finally, drag the copy of the EPUB that iBooks created on your Desktop back into iBooks. When you look at the EPUB’s category in iBooks it now reflects the “genre” entry in the iTunesMetadata.plist file.
(You might think you could edit the “genre” statement in an EPUB’s iTunesMetadata.plist file directly, using a text editor like BBEdit or Xcode’s Property List Editor, and you’d be right. Unfortunately, it’s harder than just exporting and reimporting, since iBooks hides your EPUBs deep within your Home directory’s Library; you can find them here: ~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books. However, iBooks also expands each EPUB file into a folder, and names that folder with a unique string of essentially random characters, so good luck figuring out which book goes with which folder!)
This is a silly workaround for a silly but incredibly annoying bug, but what can you do? Wait for Apple to bring back Get Info in iBooks? It may be a while…
You can find the book folder with spotlight if you can remember enough of the original filename. Not much use for batch changes, though. I can't figure out why they bother to upack the epub; on a modern desktop it surely shouldn't be a speed issue.
I still prefer a hierarchical organization in Finder though. What I want most is a simple epub reader that works like Preview. Open the darned file and leave it alone. FBReader will do that, but it's got a ways to go before it's good. I've tried to get the reading part from calibre to exist independently of the management part, but without much success. Something that would allow even minor edits would get many bonus points (and my money! Especially if it could work on snow leopard.)
As with most deeply buried folders in ~/Library, you should approach that book folder as you would a hive of bees: it's very easy to get stung unless you really know what you are doing.
I agree with you - I think this sort of organization is annoying and unnecessary.
The best argument I can make for Apple in this case is that there may be some database work going on behind the scenes as well, as with iPhoto, such that messing with the files behind the app's back could cause database corruption. As to why that sort of an architecture would necessary, I don't know.
Indeed, I was horrified when I (finally!) figured out where and how Apple was storing books in the new Mac iBooks. iTunes did this much better!
One of iTunes strengths has always been that underneath the metadata db layer the app itself uses the actual media files are stored in a simple, standard folder hierarchy you can easily look into and modify when required. It's a pity the designers of Mac iBooks didn't use the same approach with the new app.
I was horrified too when I discovered what happened to Mountain Lion's iBooks Get Info pane because I filled the comment spaces of all 1300 PDFs I created with information I got from Wikipedia, Discogs, 45cat, MusicVF, Amazon, iTunes & others.
I must be lucky, categories are working ok for me. For example, your book from TidBits, "Take Control of iBooks Author" (ePub) shows up under reference. Everything else is also in the right category. The only Uncategorized books are proofs I am working on. Maybe I am misunderstanding when this bug occurs.
Depends on when and how the books ended up in your Mac's iBooks library.
If they come over from iTunes when you installed Mavericks, no problem.
If you added them to an iOS device first and synced them to your Mac, also no problem because iTunes had a chance to add the iTunesMetadata.plist file before they ended up in your Mac's iBooks library.
But if you add them manually to your iBooks library on the Mac, and they don't have the iTunesMetadata.plist already in them, they end up uncategorized.
Ok that makes sense. Thanks. (This is why you write these articles and not me) ;)
Update: Just downloaded the iBooks update 1.0.1 and some books that had been categorized ok were moved to Uncategorized. I deleted them and re-downloaded from iCloud and they were again properly categorized.
Great article, thank you.
I'm continually frustrated by the way iTunes and iBooks 'handles' Playlists for books. Whilst only loosley related to your article my problem is metadata related and I wondered whether you had any thoughts.
I want different books on different devices and my wife and I share the Mac with one user account. We can share iTunes OK but iBooks and book management is frustrating me.
Books are synced from your Mac to your devices via iTunes, as they always were. Each device has a syncing pane for books when it is connected to iTunes. In that syncing pane you can choose which books sync to a device and which won't. Every device has its own syncing settings.
Beyond that, I have no advice to offer because I don't know enough about your individual setup.
I tried the "workaround" (moving to desktop, etc.) and that didn't work for me. I can't find a way to add additional categories; the original 5 categories plus Uncategorized aren't useful for the way I would like to categorize my books. I have added new collections but don't know if new categories can be added. Any insights into this issue would be most appreciated. Thanks.
There is no way in iBooks to add categories, because they aren't decided by the user but by the book's publisher. If a publisher does not include metadata in a EPUB that specifies its category (such as by including the dc:subject code within the EPUB, as I describe in the article), it will continue to show up as Uncategorized no matter what you do…because it really has no category.
To recap: the problem only affects EPUB books (not, say, PDFs) that you obtain from a source other than the iBooks Store and that you add manually to iBooks in Mavericks. If the EPUB was ever in iBooks, or in iTunes before the release of Mavericks, my workaround won't work because the EPUB already has the special iTunesMetadata.plist file inside of it.
If the EPUBs you're talking about meet the above conditions, make sure that iTunes is running when you add them to iBooks, make sure you delete them from iBooks AFTER you drag a copy back out (work from the All Books view to make sure you are deleting them from your library and not just from a collection). Then add the copies back in to iBooks. If the EPUB contains metadata that describes its genre or subject, that will be the category assigned to it in iBooks.
I also want to reiterate that Collections are the only method that iBooks currently provides for you to arrange your books into your own categories; the Category view in iBooks is the view that shows you how your books' _publishers_ have categorized them.
I had wondered about adding Categories as the 5 included ones were so broad and vague; not very useful. I think I will probably not pay much attention to Categories and just use Collections for my personal "categorizations."
That's probably your best option unless or until Apple provides iBooks with a way to edit a book's category, as you formerly could in iTunes.
But the "5 included ones" you mention are not the only categories you might see in iBooks, since each book you buy from the iBooks Store comes with a category applied to it, and that category can be almost anything that the publisher of the book wants it to be.
Note that "genre"="subject"="category" in iBooks. "Subject" is the accepted label for that metadata which you find inside of an EPUB's .opf file; in the iTunesMetadata.plist file that iBooks adds to an EPUB, that same information is labeled "genre"; in iBooks itself that information is called a "category"! Confused? You're not the only one…
Great info. Thanks for sharing, Michael.
Do you know if there is a way to revert back to the pre-iBooks-on-the-Mac era and have all my books and PDF's back into iTunes (but still on Mavericks)?
I'd gladly give up the ability to read them on my computer if I can be able to manage everything back in iTunes.
Thank you!!
No, sorry: not that I know of.
I appreciate the work-around provided.
The best document organizer that Apple provided was iTunes before iBooks. I was able to create reading lists to group documents regardless of media type. For example, I could see a list of all of my Take Control books sorted as I desired.
The effect of rudely discarding all the user provided metadata in the migration of books from iTunes to iBooks was capped by the seemingly arbitrary choice to ignore the original date added and change every document date added to the migration date. This, along with the unrecoverable loss of other data, greatly confuses document management. For example, I could easily sort Take Control books into Current and Legacy lists.
I use the word "rudely" as I attribute the discarding of user data as a matter of ignorance or thoughtless rather than malice on the programmer's part.
Thanks for the article. An easy way to edit an epub book's metadata (or change it's cover, etc.) is to download the free program Calibre (an excellent ebook manager/metadata editor). Then drag the book out of iBooks to a folder, open the book in Calibre, edit the metadata as desired, save back out to ePub format, and then drag back into iBooks (after deleting the book in iBooks). If you're changing the cover, be sure to drag the FOLDER that contains both the epub book and the separate jpg cover file, as exported by Calibre, into iBooks.
I would advise some caution here, and encourage keeping the original around in iBooks until you've verified the modified version. Every time I've tested Calibre with respect to converting or modifying one of our Take Control ebooks, it has caused some level of "damage." So while I haven't tried these particular steps, I inherently do not trust Calibre to modify books.
I wish I understood these directions:
"First, make sure that iTunes is running. Next, open the new EPUB in iBooks. This causes iBooks to make a copy of the EPUB, store it deep inside your ~/Library folder, and to ask iTunes to add an iTunesMetadata.plist file to the book."
the ePUB for EVERY SINGLE BOOK I OWN? A batch file that contains all the ePUB data? I have an iBooks folder with /books, /downloads / temporary / Updates. I apologize but I do not fully understand the directions. Pictures are worth a 1000 words! :-) Hopefully Apple fixes this in future updates.
No, not for every single book you own. Books that you purchased from the iBooks Store already have the genre metadata. Books that you have imported into iBooks on the Mac from iTunes also have the genre metadata. You only have to do the shuffle that I describe in the post for new books that you manually add directly to the iBooks app in Mavericks. And you only have to do that if you care whether or not the genre (Category) information associated with the book is correct.