Next Issue 08-Dec-03 — We’re taking next week off from our regular publication schedule so we can all spend some relaxing time with families and friends over the Thanksgiving holiday here in the U.S. I plan to continue moderating TidBITS Talk many of these days other than a few around Thanksgiving itself, and we’ll be using some of the extra time to work on assembling our annual gift issue. Look for the next issue of TidBITS on 08-Dec-03, and for those partaking in the holiday festivities, enjoy! [ACE]

Submit Your Holiday 2003 Gift Ideas — We may not approve of ads for Christmas sales appearing the day after Halloween, but it is now once again a decent time to start thinking about the year’s annual spectacle of consumer confidence. For us at TidBITS, that translates into sharing the best holiday gift ideas from readers in our traditional holiday gift issue, slated for publication during the second week in December. So tell us what gifts you’re planning to bestow on your friends or family, or what gifts you’re hoping to receive yourself. As in previous years, we’ll be collecting ideas in TidBITS Talk, so please send your suggestions to <[email protected]>. We’ve already started threads for specific categories, and there’s a "Respond (via email)" link for you to use at the bottom of each message in the TidBITS Talk Web archive (check out the Hot Topics in TidBITS Talk article at the end of this issue for direct links). As always, please suggest only one product or idea per message, give the reason why you’re recommending it, make sure to include a URL or other necessary contact information, and recommend only others’ products. Also try to concentrate on items that haven’t appeared in past years. Thanks! [ACE]

<https://tidbits.com/getbits.acgi? tlkthrd=active>

<https://tidbits.com/getbits.acgi?tbiss=560>

<https://tidbits.com/getbits.acgi?tbiss=609>

<https://tidbits.com/getbits.acgi?tbiss=659>

Apple Introduces 20-inch iMac — Just in time for the holiday shopping season, Apple has released a new version of the flat-panel iMac that offers a 20-inch LCD screen with a resolution of 1680 by 1050 pixels. The screen is also brighter then the smaller models and offers a higher contrast ratio. As far as other functional specs go, the 20-inch iMac is identical to the 17-inch iMac, with a 1.25 GHz PowerPC G4 processor, 256 MB of RAM, 80 GB hard drive, SuperDrive, and all the usual ports. The one interesting deviation from the 17-inch iMac is with the weight – the 20-inch iMac weighs 40.1 pounds (18.2 kg), 17.3 pounds (7.8 kg) heavier than the 22.8 pound (10.4 kg) 17-inch iMac. The new 20-inch iMac is available immediately for $2,200 in the stock configuration, with all the usual build-to-order options. The 15-inch iMac remains priced at $1,300 and the 17-inch iMac still costs $1,800. [ACE]

<http://www.apple.com/imac/>

<http://www.apple.com/imac/graphics.html>

Apple Releases 1.8 GHz Dual Processor G5 — Apple has released a new model of the Power Mac G5, its 64-bit minitower computer aimed at professional and high-end Macintosh users. The new model offers dual 1.8 GHz PowerPC G5 processors (each with a 900 MHz frontside bus) with prices starting at $2,500. The new systems slip into the middle of Apple’s G5 lineup, replacing a $2,400 configuration with a single 1.8 GHz processor; otherwise, the specifications for the Power Mac G5s remain unchanged, save that they all now ship with Mac OS X 10.3 Panther. (For details on the changes in the Power Mac G5 architecture compared to previous systems, see "Apple Announces 64-Bit Power Mac G5s" in TidBITS-685.) However, Apple has reduced the starting price of the 1.6 GHz single processor system by $200 to $1,800. The new dual-processor model serves to highlight benefits of Mac OS X’s multiprocessor capabilities, particularly in computationally intensive areas like video and media production. [GD]

<http://www.apple.com/powermac/>

<https://tidbits.com/getbits.acgi?tbart=07241>

Security Update 2003-11-19 Helps Panther, Jaguar — Apple released a pair of security updates last week via Software Update, bringing both Mac OS X 10.2.8 Jaguar and Mac OS X 10.3.1 Panther up to the same security level. Security Update 2003-11-19 improves OpenSSL security and fixes a potential problem with third-party applications using a print library. On Macs running Jaguar, the security update also fixes issues with Mail, QuickTime for Java, Personal File Sharing, and a few other vulnerabilities that were implemented in the first release of Mac OS X 10.3 Panther (see "Security Update 2003-10-28 Released" in TidBITS-704). The update for Jaguar is a 4.6 MB download; the update for Panther is a 1.3 MB download. [JLC]

<http://docs.info.apple.com/article.html? artnum=120278>

<http://docs.info.apple.com/article.html? artnum=120277>

<https://tidbits.com/getbits.acgi?tbart=07418>

Creo Eases File Sharing with Tokens — Have you found sharing files via email frustrating? Email, as we’ve been predicting, is becoming increasingly unreliable, thanks to spam-overloaded servers and inaccurate filters, and, when we’re talking about file sharing, encoding format troubles and attachment size limitations. Creo, makers of the Six Degrees program for managing email-based workgroups, has come up with an alternative called Tokens, which works with both the Mac and Windows. Rather than attach your huge Keynote presentation to an email message to multiple people, you use the $50 Tokens Creator to create a several-kilobyte "token" that points back to a compressed and encrypted version of the Keynote file on your hard disk. When your recipients receive the token you sent, they double-click it to open it in the free Token Redeemer, which retrieves the presentation from your computer. Your computer must be left on and connected to the Internet to serve the file; if there are other network obstructions (such as firewalls) between the recipient and your computer, Token Redeemer automatically retrieves the file via Creo’s Tokens relay service (which allows up to 5 GB per month of transfer). After a basic installation on both sides, no setup is necessary, and no one has to worry about user accounts or passwords. Creo also offers a $600 Tokens Server, which comes with 10 licenses for Token Creator and handles the file serving duties for all of them. Tokens is definitely a 1.0 product, but it’s interesting, and could serve an important role in simplifying file transfer. [ACE]

<http://www.creo.com/tokens/>

Silly us. We thought things would ease up after our initial launch and release of the 1.1 update to Joe Kissell’s "Take Control of Upgrading to Panther" two weeks after Panther shipped. On the contrary, we’re working as hard as ever, and orders continue to flow in as more and more people hear about Take Control from places like Apple eNews and a number of user group newsletters.

"Take Control of Customizing Panther" Update — Most notably, we’ve now released the 1.1 update to Matt Neuburg’s "Take Control of Customizing Panther." We’re really pleased with it, since we’ve made significant changes and additions that place it squarely where we want the Take Control series to evolve. It’s more than twice as long, with 47 content pages, up from 22, and we focused the writing to provide a more concrete set of tasks to perform to customize Panther to your liking.

<http://www.tidbits.com/takecontrol/panther/ customizing.html>

New sections include a discussion of using some of our favorite third-party utilities to customize Panther, coverage of how you can manage Services better, three ways of setting your Internet helper applications, and small tweaks you can apply to Safari and Terminal. Also, since we discovered that many readers didn’t have much, if any, experience with Mac OS X, we added a large section listing the basic customizations you can perform using System Preferences and Finder Preferences.

In accordance with the feedback we received after publishing the update to Joe’s book, we added internal links to the Changes in Version 1.1 section, so you can jump quickly to the new bits. We also added a tip about which pages largely stayed the same for people who want to print only the changed pages; however, Matt added and changed so much that most pages are different. One tip regarding printing: in the Print dialog, choose Layout from the Copies & Pages pop-up menu, choose 2 from the Pages Per Sheet pop-up menu, and choose Single Hairline from the Border pop-up menu. The large font we used prints quite nicely when reduced, so that two pages fit on each piece of paper.

Since we believe early adopters should be rewarded, and since our goal is to provide the best experience possible, we’ve sent email to almost everyone who purchased the 1.0 version, telling them how they can get the 1.1 version for free.

Why almost everyone? When you order through eSellerate (see below), there’s a checkbox titled "Send me information about product updates and offers." We can’t currently change the wording of that checkbox, so if you deselect it, we won’t send you email about the free update – you’ve opted out. If that applies to you, send us email at <[email protected]> and we’ll let you know how to get the update and add you to the list.

Similarly, if you didn’t receive email notification because you have changed email addresses or because your ISP is employing stupid spam filters, send us email and we’ll fix you up. That said, dealing with these delivery problems generates a lot of unnecessary work for us, so please try to ensure that you can receive mail reliably at the address you give when ordering.

Ordering System Changes — We’re now trying eSellerate’s service for fulfilling our orders. Although we had no general complaints with Kagi, we found that dealing with our own merchant account was draining vast amounts of time and energy away from editorial work, and we can’t afford that right now. Overall, eSellerate appears to be working well for us, and one very real advantage is that you can download directly from the final order screen, rather than waiting for your email receipt to arrive.

eSellerate also hosts our downloads now, which lets us eliminate the encrypted archives that required entering passwords into StuffIt Expander, a process that proved surprisingly troublesome. We still compress our PDF files because so many Web browsers are set to display PDF files in a window rather than download them for later reference. To make things easier for people who want to download at work on a Windows machine and print there, we used StuffIt Deluxe to create a Zip archive that any recent version of StuffIt Expander should be able to expand. People who download using Safari in Panther will find the Zip archive expanded automatically without launching StuffIt Expander.

New Ebooks Coming Soon — Although most of our efforts have been aimed toward our existing two titles so far, we have two more books in the works right now: "Take Control of Sharing Files in Panther," by Glenn Fleishman, and "Take Control of Users & Accounts in Panther," by Kirk McElhearn. You can read more about them on the Take Control home page, and stay tuned for even more titles that are still in the outline stage.

<http://www.tidbits.com/takecontrol/>

Some things are inevitable. Death, taxes, and disk crashes. One day you will try to open an important file, only to receive a dire error message. Or perhaps you’ll discover that an entire folder has vanished. Worse yet, maybe your Mac won’t even boot, thanks to some sort of disk corruption.

Fortunately, you have a full backup of all your data, so you just restore the missing data from your backup, and you’re back in business. What’s that you say? The last time you backed up was during the Reagan administration?

If an ounce of prevention is worth a pound of cure, then surely the most effective disk repair program is actually a reliable backup utility. My favorite is Dantz’s Retrospect. But whatever backup program you choose, you must use it regularly, so you have a current backup when your hard disk is called to that great clean room in the sky. (See the TidBITS article series, "Have You Backed Up Today?" for more details on setting up a good backup strategy.)

<http://www.dantz.com/>

<https://tidbits.com/getbits.acgi?tbser=1041>

Yet just as so many of us would prefer to lose weight by taking a magic pill rather than through diet and exercise, we’d rather fix a corrupted disk with a disk utility, rather than restoring from a backup, even when a recent backup is available. It can take many hours to do a full restore from a backup, whereas a good disk utility can often fix minor disk errors in minutes.

Some of the Macintosh world’s favorite disk repair programs have recent upgrades, and here I’ll compare the Norton Disk Doctor tool in Symantec’s Norton Utilities 8.0 ($100), Alsoft’s DiskWarrior 3.0 ($80), Micromat’s Drive 10 1.1.4 ($70), SubRosaSoft’s DiskGuardian 2.2 ($70), and Apple’s Disk Utility (free). Although these programs contain a wide variety of disparate features, I concentrate on their disk repair functions in Mac OS X. I chose not to include Prosoft Engineering’s Data Rescue X, because it recovers files onto another disk and does not repair the damaged disk itself.

<http://www.symantec.com/nu/nu_mac/>

<http://www.alsoft.com/DiskWarrior/>

<http://www.micromat.com/drive_10/drive_10_ introduction.html>

<http://www.subrosasoft.com/>

<http://www.prosoftengineering.com/products/data _rescue.php>

The user interface and ancillary features of a disk repair program are secondary, because in the event of disaster you care about only one thing: will it get my data back? So let’s concentrate on the heart of the issue: what are the most common disk errors you may experience, and which disk repair programs can save your bacon when you’re unfortunate enough to suffer disk corruption?

In my experience, most people run into three general categories of disk problems: hardware failure, bad sectors, and damaged directories. After a brief examination of how you get started with these programs, given that you can’t repair an active startup disk in Mac OS X, I look at the worst type of problem – hardware failures, after which I examine the soft errors and compare the performance of the disk repair programs.

Booty Call — One disadvantage of Mac OS X is that a disk repair program can’t safely check the startup disk. Despite this, and the fact that Apple’s official line is that checking startup disks is not supported, Norton Disk Doctor and DiskGuardian both allow checking of the startup disk, although they warn against doing so. I consider messing about with startup disks under Mac OS X dangerous, and I advise you not to do it.

Fortunately, there is a simple solution. Restart in "single user mode" by holding down Command-S while the Mac is starting up. In single user mode, you’re dropped into a command line version of Mac OS X, without windows or a mouse pointer. Type "fsck -y" to check (and repair, if necessary) the boot disk, after which you restart the Mac by typing "reboot" (sans quotes for both commands). Disk Utility and fsck rely on the same engine, so running fsck in single user mode is exactly the same as running Disk Utility.

If you’re uncomfortable with the command line for even two commands, you have an alternative. If you boot your Mac using the Mac OS X Installer CD, you can run Disk Utility from there. At the first screen in the Installer, choose Open Disk Utility from the Installer menu.

What about the other disk repair programs? All except DiskGuardian (for now, but a new version is expected soon) come on bootable CDs. Insert the CD, turn on your Mac, and hold down the C key to force the computer to boot from the CD. Bootable CDs are essential in the event that your hard disk is so badly damaged that your computer won’t even start up.

It’s Dead, Jim — Let’s look at what can go wrong now. Hardware failure can result from the electronics on the drive’s controller board burning out, or the heads or the arm developing mechanical problems. Sometimes a problem with the lubricating grease prevents the disk from spinning or the read arm from moving, causing a problem known as "sticktion."

These problems are caused by dropping the disk, by defective components, by static discharge, or even by sheer age. Usually the disk won’t even show up on the Desktop. As far as the disk utilities go, Drive 10 can detect hardware errors with a "Unit Ready" test, which is just what it sounds like. Drive 10 asks the drive "Are you ready?" and the drive replies "No." This test is mostly helpful for confirming hardware errors you probably already suspect.

Hopefully there’s no vital data locked in the dead drive. No software program can repair a disk with hardware problems. If you desperately need to retrieve the data, your only option is a data recovery company such as DriveSavers, who I’ve found to be professional, competent, and expensive. (Also see Jeff Carlson’s report in "DriveSavers to the Rescue" in TidBITS-495.)

<http://www.drivesavers.com/>

<https://tidbits.com/getbits.acgi?tbart=05530>

Extended Warranties — If your drive has just suffered a hardware failure, you’re probably checking the warranty and hunting for your receipt. Don’t despair, even if your computer or external hard drive is out of warranty from the manufacturer. Many hard drives, even those installed in computers, are also covered by longer warranties provided by the OEM (original equipment manufacturer), the company that actually made the drive. If you’re willing and able to open a case and remove the drive mechanism, you can take advantage of this warranty.

Here’s the trick. Although you may have bought the computer from Apple, or the drive from VST, those companies don’t make drive mechanisms. Instead, your drive was probably made by IBM, Seagate, Maxtor, HP, Western Digital, or another hard drive manufacturer. These companies often offer an independent warranty on their drives, and it’s often longer than the warranty on your Mac or external hard drive. Drive mechanism warranties may be two years, and some run as long as five years. Better yet, if you’ve lost the receipt, the company can sometimes look up a drive’s serial number to verify that it’s still under warranty.

Both HP and IBM have replaced dead drives for me, without a receipt, simply by checking my serial numbers. Even better, IBM didn’t make 14 GB drives anymore, so they replaced my dead one with a 20 GB unit.

SMART Stuff — Some modern hard drives have a feature called SMART, which stands for Self-Monitoring, Analysis, and Reporting Technology. Originally developed by IBM, SMART-compliant drives constantly perform diagnostics to monitor variables like drive temperature, spin up time, and how well the heads stay on track. By noting when these physical parameters slip out of spec, SMART can predict some types of drive failure before they happen, giving you time to back up your data and buy a new drive.

Unfortunately, most hard drives have no way to tell you about an impending disaster; you need a program to query the drive’s SMART statistics. Both DiskWarrior and DiskGuardian can check the drive at regular intervals and alert you if they find trouble.

Using SMART works well if your disk has a factory defect that’s slowly getting worse, because it gives you time to move your data to a new disk. But SMART doesn’t help you deal with the occasional bad sector or corrupted directory, much less catastrophic hardware failures. Since I don’t have a drive with impending hardware failure, I was unable to test the SMART capabilities of DiskWarrior and DiskGuardian.

Unsavory Sectors — Now it’s time to move from pure hardware problems to problems that could be either physical (hardware) or logical (software). Disks are broken up into sectors. Each sector usually holds 512 bytes of data. There are two types of problems that lead to bad sectors: hard errors and soft errors.

Hard errors are caused by physical damage to the disk surface. Dropping the disk and manufacturing defects are the most common causes of hard errors. Although there’s no way to repair hard errors, they can be "fixed" by "sparing" the bad sector. Disks maintain a small number of spare sectors in reserve; when a hard error occurs, the drive controller maps out the bad sector and uses one of the spare sectors in its place.

How are these bad sectors spared? SCSI hard disks provide a SCSI command – "reassign" – to spare a bad sector. A low-level format also spares any bad sectors. On pre-SCSI disks, a low-level format was the only way to fix bad sectors.

Many modern disks, including many internal ATA, FireWire, and USB drives, automatically spare sectors with hard errors the next time the sector is written. That’s helpful, but if data is stored on that sector, programs may be unable to read it successfully, causing problems and making it difficult to spare. Erasing the disk with Apple’s Disk Utility spares any bad sectors if you select the "Zero all data" option.

What about soft errors? In addition to the 512 bytes of data stored in each sector, a few additional bytes hold an error correction code (ECC). When the sector is written, the drive’s controller computes and records the ECC. When the file system later reads that sector, it checks the ECC to make sure the data hasn’t been corrupted. If the ECC doesn’t match the data, it’s called a soft error. The disk surface is fine, but the data on that sector has become scrambled.

Soft errors can be caused if the disk is jarred while it’s writing or if power is lost while writing, either of which can leave a sector half written. Large magnets (such as can be found in electric motors) next to hard disks also tend to have bad effects on the data. As with hard errors, most modern disks repair soft errors automatically the next time the sector is written.

Bad Sector Detector – Norton Disk Doctor, DiskGuardian and Drive 10 (but not DiskWarrior or Disk Utility) claim to detect bad sectors using a test called either a "defective media check" or a "surface scan." Using a proprietary tool that creates soft errors on disks, I tested each program.

Although Norton Disk Doctor claims to be able to find and repair bad sectors, its defective media check didn’t detect the bad sectors on my test disk, erroneously giving it gave a clean bill of health.

DiskGuardian detected the bad sectors, although it took several hours to run a full check. Unfortunately, it didn’t tell me which files used the bad sectors, so I had no way of finding out which files were damaged and would thus need to be restored from backup. DiskGuardian lacks the capability to repair bad sectors.

Like DiskGuardian, Drive 10 detected the bad sectors, but didn’t identify which files were damaged. Confusingly, Drive 10’s report describing the damage claimed it could fix the bad sectors, but I couldn’t find a command to fix them. Micromat tech support confirmed the report was wrong; Drive 10 can’t fix the bad sectors it finds. It’s too bad, since Drive 10 could fix the bad sectors merely by writing zeros to them.

Although Disk Utility cannot scan for bad sectors, it can fix bad sectors on modern disks if you erase the disk with the "Zero all data" option selected.

I must rate all these products unacceptable in dealing with bad sectors. Even though two could detect bad sectors, none of them could tell you which files contain bad sectors, making it impossible to learn which files you should restore from your backup. Only Disk Utility successfully fixed the bad sectors, but at the price of erasing the entire disk.

Ripping the Yellow Pages — We’ve now looked at pure hardware failures, and bad sectors, which can be either hard errors or soft errors, and so far, our disk repair utilities don’t help much at all. Now it’s time to move on to problems that exist entirely in software, the most common type of which are errors in the directory, which tracks the files and folders on the disk. In the case of directory errors, there is nothing wrong with the drive mechanism or the disk surface; instead, the directory information that’s necessary to locate your data on the disk has simply become scrambled. Often your data is intact, if it could just be located.

As an aside, people with important data sometimes use mirrored disks or RAID arrays, which faithfully duplicate each byte on the main disk to a backup disk. If the main disk suffers a hardware failure or develops a bad sector, the backup disk can save the day. However, it’s worth noting that this strategy provides absolutely no protection against directory damage. That’s because the RAID faithfully records all data to the backup disk, whether or not that data is good, which results in both the main disk and the backup disk containing corrupt data. I may sound like your mother telling you to eat your vegetables, but the best protection really is regular backups.

The most common cause of directory damage is crashing. If the computer crashes while a file is being created or saved, causing only part of the change to be written to disk, the directory will contain inconsistent information. Mac OS X crashes far less often than Mac OS 9, but directory-corrupting crashes can and do still occur. Both Mac OS 9 and Mac OS X automatically check and repair the startup disk after a crash, which reduces the incidence of disk damage dramatically.

A new feature in Mac OS X 10.3 Panther that should reduce directory errors even more is the journalled file system. You can enable it in Disk Utility, and it’s usually turned on for disks onto which you install Panther. Here’s how journalling works. Before the file system changes the directory, it leaves a note on the disk saying, "I’m going to make this change in the directory." Then the file system makes the change, and once it finishes, it clears the note. If the file system ever sees an incomplete change note on the disk during startup, it knows something bad happened and "rolls back" the directory to its previous state. You will lose your last change, but the directory won’t suffer any damage.

Other causes of damaged directories include buggy programs that write bad data to disk, buggy programs that overwrite cached data waiting to be written to disk, and even bugs in the file system itself. The first two are much less likely in Mac OS X than in Mac OS 9 because of its file privileges and memory protection, respectively. Bugs in the file system are extremely rare but have occurred at times in the past.

It’s worth noting that directory damage is not always readily apparent. A damaged disk may appear to operate perfectly, but regular use can cause minor errors to grow into serious problems. Most directory problems are easy to fix if they’re caught early but can be difficult, if not impossible, to fix later. That’s why checking and repairing startup disks automatically after a crash is so important, and why it’s essential to leave the Check Disk option turned on in Mac OS 9’s General Controls control panel (the disk check isn’t optional in Mac OS X).

To test how the disk repair utilities perform with different types of directory errors, I created an HFS+ disk image, copied an assortment of files and folders to it, and then used a low-level disk editor to damage various directory data structures. I then duplicated the damaged disk image, and let each utility try to repair its own copy. Each utility repaired an identical disk image, with identical damage.

Errors Speak Volumes — For my first test, I started with relatively simple damage in the volume bit map, which is also known as the allocation file. The volume bit map tracks which blocks on the disk contain files, and which are unused. All five utilities fixed my damaged volume bit map easily.

Next up was damage to the volume header, which tracks vital information about the disk, such as the amount of used and free space, and the locations of the catalog and allocation file. The volume header is stored at the beginning of the HFS+ partition. I erased the volume header’s signature, which makes the file system assume the volume header is corrupt and refuse to use the disk. Fortunately, the file system keeps a backup copy of the volume header at the end of the disk; it’s imaginatively called the alternate volume header. All five of our utilities successfully repaired the disk, although Drive 10 and DiskGuardian couldn’t figure out the name of the damaged disk.

Catalog Catastrophe — The catalog b-tree tracks all the files and folders on the disk. It’s a vitally important part of the directory, and many of my tests focus on it. The catalog is divided into nodes, and each node is divided into records. Most records track a file or folder on the disk, although some contain threads or indexes, which are used internally by the file system to look up files and folders.

The first node in the catalog is called the header node, which points to other key nodes. I erased the header node. Norton Disk Doctor, DiskWarrior, and Drive 10 recreated the header node properly; Disk Utility and DiskGuardian failed to fix it. Once again, Drive 10 couldn’t figure out the name of the damaged disk.

The header node also contains a map which tracks which nodes are used and which are free. I corrupted this map, but my corruption didn’t faze any of the utilities, all of which successfully fixed the header node map.

The nodes in the catalog are linked together in a precise pattern of connections. Horizontal links connect nodes on the same level, and downward links connect the levels. The file system relies on these links to look up files and folders. I damaged these links. As happened when I erased the catalog header node, Norton Disk Doctor, DiskWarrior, and Drive 10 fixed these links, but Disk Utility and DiskGuardian weren’t able to put the links back together.

File and folder records are stored in alphabetical order in the catalog. I rearranged these records, putting them in random order. All the utilities restored the alphabetical order.

Certain characters, such as a colon, are illegal in file and folder names. Normally, the operating system prevents you from typing an illegal character when saving a file or creating a folder, but it’s not inconceivable that unusual circumstances could cause one to appear. I renamed a folder with a colon by inserting the colon directly into the folder record in the catalog. With this test, the results start to become more interesting. Disk Utility and DiskGuardian didn’t detect any problem. Drive 10 noticed the illegal character, but didn’t fix it. Norton Disk Doctor and DiskWarrior both fixed it properly by replacing the colon with a legal character.

More Catalog Corruption — Each catalog node ends with a map that points back to the records in that node. I damaged the map for one of the nodes, which sounds bad, but it’s still possible to find the records by calculating the size of each record to find the next record. Disk Utility and Norton Disk Doctor realized there was a problem, but they couldn’t fix it. Drive 10 and DiskGuardian both identified and fixed the problem, but in the process lost five and six files, respectively. Partial repair isn’t always better than complete failure, since you may believe the disk was repaired successfully and only later – potentially much later – realize that some files have been lost. DiskWarrior fixed the catalog node map properly.

Next, I changed a thread record to be an unknown type of record, which creates two problems. A thread record that the file system relied upon was missing, and it was confronted by a record with an illegal type. Disk Utility and DiskGuardian detected the corruption, but couldn’t fix it. DiskWarrior fixed the problem but lost some of the data in one file. Only Drive 10 and Norton Disk Doctor managed to repair my damage properly.

Note that DiskWarrior moves any files it suspects may have problems into a folder called Rescued Items. In my tests, most of these files turned out to be fine. This approach has the advantage that it’s clear which files may be damaged. But if the Rescued Items folder contains many files, checking them and putting them away can be tedious. Norton Disk Doctor can optionally put aliases to damaged files in a folder, a potentially more helpful feature. But in my tests it didn’t work. Norton Disk Doctor also lists the names of damaged files in its report.

The most important aspect of a file record is the location of the file’s data on the disk. One of the worst sorts of directory damage that you may see happens when two files try to occupy the same physical space at the same time. Different utilities refer to this problem as "overlapping extents" or "cross-linked files." In the best case, one file has entirely overwritten the other, since then one file has valid data, while the other’s data is completely gone. In the worst case, the two files somehow manage to interleave their data, which results in both being damaged beyond repair. I cross-linked two files, thus damaging the files’ catalog records, as well as the volume header and volume bit map. Disk Utility and DiskGuardian repaired the catalog records, the volume header, and the volume bit map, but they didn’t actually separate the two files. In contrast, Drive 10, DiskWarrior and Norton Disk Doctor fixed the damage and separated the files. It’s important to realize that the data in the overwritten file couldn’t be recovered, but not through any failing of these repair programs. When one file overwrites another, the unlucky file has no chance of surviving the encounter.

I See Fragged People — Some months ago, I wrote an article for TidBITS explaining why defragmenting disks generally isn’t worthwhile (see "Optimizing Disks Is a Waste of Time" in TidBITS-686). Although fragmentation is totally normal and acceptable, serious fragmentation requires additional directory structures, and they too can become corrupted and require repair.

<https://tidbits.com/getbits.acgi?tbart=07254>

Using another proprietary tool, I fragmented a disk very badly, which breaks files into so many pieces that the file record in the catalog b-tree can’t track them all. The file system responds by creating new records in the extents b-tree to help track all the pieces. The extents b-tree is like the catalog b-tree, but exists solely to help track highly fragmented files. Again, severe fragmentation is not inherently a problem, but I had put the disk into a precarious, if legal, state. I then damaged two extent records so the file system couldn’t find all the pieces to two files, but I damaged each in a different way. Disk Utility, DiskGuardian, and Drive 10 all detected my damaged extent records, but weren’t able to repair the damage. DiskWarrior and Norton Disk Doctor performed better, fixing the problem but losing part of the data in one file. Considering the type of damage I inflicted, they did as well as could be expected.

HFS+ volumes are enclosed in a "wrapper," which is actually a plain old HFS volume. The reason for the wrapper is historical. Apple first released HFS+ with Mac OS 8.1. If you connected an HFS+-formatted disk to a Mac running Mac OS 8.0 or earlier – in other words, one that understood only the older HFS format, the wrapper kept the older system from deciding the HFS+ disk was damaged and offering to initialize it. For my next test, I damaged the catalog b-tree header node in the wrapper. Disk Utility and DiskGuardian didn’t notice anything wrong. Drive 10 and DiskWarrior detected the corrupt wrapper, but didn’t fix it. Norton Disk Doctor identified the damage and fixed it properly.

Disks can contain multiple partitions, which are listed in a partition map at the beginning of the disk. Since disk images don’t have partition maps, I used an external FireWire hard disk for this test, in which I damaged the partition map, making the disk driver’s partition overlap the HFS partition. Of all these disk repair utilities, only Norton Disk Doctor claims to check partition maps, and indeed it was the only one to detect the problem, although even it proved incapable of fixing the overlapping partition map. Luckily, damaged partition maps are extremely rare, which may be why none of the other utilities bother to check them.

The Grand Finale — Finally, I decided to recreate the worst damage I’ve ever seen on a Macintosh hard disk. Starting with the badly fragmented disk above, I corrupted and overwrote various parts of the catalog and extent b-trees. In some nodes I corrupted the node header (not to be confused with the header node), in some I munged the data records, and in others I zapped the record offset map. A few lucky nodes suffered all three types of damage at the hands of my disk editor. Only DiskWarrior was able to bring the disk back to a usable state, although 35 files were either lost or partly damaged. That 35 files were lost or damaged is not an indictment of DiskWarrior; the program couldn’t have done any better, considering how much vital information had been destroyed. None of the other utilities managed to repair the disk successfully.

And the Winner Is… Of my 15 tests, DiskWarrior fixed 12 successfully, Norton Disk Doctor fixed 11, Drive 10 fixed 9, DiskGuardian fixed 5, and Disk Utility fixed 4. "Fixed" includes cases where recovery may not have been perfect, but was good enough.

So what, in my professional opinion, should you do if your disk starts acting up? First, try Apple’s free Disk Utility. It may fix only a limited set of problems, but when Disk Utility finds a problem, it’s invariably correct, and it applies fixes only when it’s absolutely certain it knows the correct fix. I’ve never seen Disk Utility accidentally make a problem worse, something the other utilities can do, even if only very occasionally.

If Disk Utility doesn’t succeed, let DiskWarrior do battle with your damaged directory. It was our overall winner, and it deserves its excellent reputation. DiskWarrior can also show you a preview of the repairs before you accept them, which lets you check that a damaged file or folder really was fixed before DiskWarrior makes the fix permanent.

If DiskWarrior fails, give Norton Disk Doctor a try, since it can address some problems that DiskWarrior misses. After that, try sacrificing chickens. Seriously, if the combination of Disk Utility, DiskWarrior, and Norton Disk Doctor can’t repair your disk, you can either restore your data from backup, or, if that’s not possible, decide if the data is sufficiently important to pay DriveSavers for recovery.

I still think the most important data protection utility you should own is a backup program. But sometimes a good disk repair program can save the day by repairing minor damage quickly so you don’t have to run through the time-consuming process of reinitializing your hard disk and restoring from backup.

[David Shayer was a senior engineer on Norton Utilities for Macintosh 3.0, 4.0, and 5.0. Before that he worked on Public Utilities, a disk repair program that won the MacUser Magazine Editor’s Choice Award, and on Sedit, a low-level disk editor.]

PayBITS: If David’s detailed and expert testing told you which

disk utility you should rely on, reward his efforts via PayBITS!

<http://www.amazon.com/paypage/P12NE4WQ7K8ODD>

Read more about PayBITS: <http://www.tidbits.com/paybits/>

Jeff’s video cropping odyssey — Jeff Porten follows up his article about EyeTV with this description of how he stores programs on video. (1 message)

<https://tidbits.com/getbits.acgi?tlkthrd=2118>

Faxing in Panther — TidBITS Talk readers report their experiences using Panther’s built-in fax software. (2 messages)

<https://tidbits.com/getbits.acgi?tlkthrd=2117>

Indie Digital Music — As MP3.com goes dark (including the vaporization of its library), readers point out the quality music that existed amongst the sludge. (2 messages)

<https://tidbits.com/getbits.acgi?tlkthrd=2116>

Stopping spam at the server — Charles Arthur’s review of PostArmor sparks discussion about other sever-level spam-fighting tools. (8 messages)

<https://tidbits.com/getbits.acgi?tlkthrd=2114>

2003 Holiday Gift Ideas: Hardware — It’s time for gizmos, gadgets, and honkin’ big Power Macs. What hardware gifts do you suggest? (1 message)

<https://tidbits.com/getbits.acgi?tlkthrd=2123>

2003 Holiday Gift Ideas: Software — Hand your niece or nephew some source code… or, be traditional and get some pre-compiled software for the holidays. Which titles do you favor? (1 message)

<https://tidbits.com/getbits.acgi?tlkthrd=2122>

2003 Holiday Gift Ideas: Gamwa — Whether you prefer to save the universe or puzzle through some good brain-benders, which game titles do you recommend? (1 message)

<https://tidbits.com/getbits.acgi?tlkthrd=2120>

2003 Holiday Gift Ideas: Computer Miscellaneous — Have a gift idea that doesn’t fit into an easy category? Suggest it here! (1 message)

<https://tidbits.com/getbits.acgi?tlkthrd=2119>

2003 Holiday Gift Ideas: For the Macintosh-minded — Mac users do have other interests, you know. What would your favorite Apple enthusiast appreciate? (1 message)

<https://tidbits.com/getbits.acgi?tlkthrd=2121>