Auditing Free Drive Space: Where Have All the Gigabytes Gone?
Howard Oakley’s explanations of the underpinnings of macOS are always interesting, but his latest post on his Eclectic Light Company blog, “How to get the Trash working properly,” particularly drew my attention. In it, he points out that deleting a very large file to free up space doesn’t necessarily have the desired effect, at least right away.
The reason, he says, is that Time Machine works by making APFS snapshots of your entire drive on every backup and retaining those for 24 hours. Because one of those snapshots contains the very large file, you may not get the space back until a full day later, when Time Machine has deleted those snapshots to make way for new ones.
Oakley says that some backup apps use a similar approach—he calls out Carbon Copy Cloner—but unlike Time Machine, such apps generally provide options to disable the feature, change how long snapshots are retained, and delete the snapshots if you need the space right away. Time Machine has no such options, but you can delete the snapshots manually in Disk Utility. (Still other backup apps, such as Arq and Backblaze, don’t use APFS snapshots but consume significant amounts of space for caching purposes—that’s getting some discussion in TidBITS Talk right now.)
I’ve been pondering just how difficult—perhaps impossible—it is to know how much free space you actually have on a Mac’s drive these days (see “Ensure Sufficient Free Space before Upgrading to Ventura,” 15 November 2022 and “iPhones and iPads Now Require a Passcode on Every Backup/Sync,” 11 January 2023). The complexity underlying APFS and macOS in general creates a situation where the amount of free space isn’t entirely deterministic. Howard Oakley examines the variables in “Explainer: Disk free space,” and Jeff Carlson explains APFS in his Take Control of Your Digital Storage, Second Edition ebook.
I decided to see if I could replicate some of these issues, at least in macOS 12 Monterey, which I’m still running on my 2020 27-inch iMac until I find the time to do a much-needed clean installation of macOS 13 Ventura. I recently upgraded to V2 of the Affinity suite (see “Consider Switching from Creative Cloud to Affinity V2,” 5 December 2022), but I hadn’t gotten around to trashing the earlier versions of the Affinity apps. They’re about 2.5 GB each, and I put them in the Trash yesterday. This morning, Get Info on my boot drive reported 137 GB free, and the Trash contained 7.6 GB.
I figured that if I emptied the Trash, I’d either see no change due to the Time Machine snapshot issue or have almost exactly 145 GB free. Surprisingly, neither was true. After a minute or so, the free space number did rise—as expected—but first to 148 GB (not shown) before settling down at 146 GB. I can’t explain why it was a full gigabyte higher than would seem possible.
So, for whatever reason, I was not seeing what Howard Oakley was describing. That’s good, I suppose, since his scenario was going to cause confusion for users. Nonetheless, I wanted to see what role those Time Machine snapshots played in the free space available on my drive. When I looked in Disk Utility, I had 24 snapshots, nominally consuming 36 GB. (By the time I took the screenshot below, I had already deleted two individual snapshots without making any difference in the amount of free space.)
Regardless, when I selected all the snapshots and clicked the – button below the list, my free space number rose again, first to 163 GB (below left) and then, five minutes later, to 167 GB (below right). I’m somewhat perturbed that it changed so significantly over five minutes, but it may take that long for APFS to recalculate. However, since I had been at 146 GB free before, I would have expected it to rise to 182 GB. I can’t even begin to explain why it rose by only 21 GB.
Grasping at straws, I decided to restart, just in case that somehow helped the filesystem come to a better conclusion about the amount of free space. Unfortunately, after my Mac came back up (with the same apps running), the free space number had dropped to 159 GB. Inexplicable!
Unless things have changed significantly in Ventura—my M1 MacBook Air isn’t a good testbed for this scenario—I no longer believe it’s possible to audit the free space on a Mac or to explain precisely how much space should be freed up by a particular action. Call it the APFS Uncertainty Principle (in which APFS expands to Absolutely Perplexing Fluctuating Space). Therefore, my advice is:
- Unless you’re simply curious, don’t waste time and mental energy trying to determine why deleting files or snapshots doesn’t free up the amount of space you think it should. macOS has become more complex than in past years, and the numbers no longer add up.
- More so than ever before, do not let your boot drive get low on free space. You may be prevented from working until you jump through space-clearing hoops that might be time-consuming and inconvenient.
- When buying a new Mac, choose an SSD with sufficient headroom for your data to grow. For instance, if you’re pushing the limits of a 512 GB SSD, get the 1 TB SSD on your next Mac, even if doubling your storage seems like overkill. Data never shrinks. (Besides, the smallest SSDs may be half the speed of larger ones due to using a single NAND chip instead of two NAND chips.)
Have you had to go to great lengths to free up space on your Mac? Let us know in the comments what you tried and how it worked.
I can’t explain what you saw, but I’ve seen enough in the past to believe that Apple’s GUI tools (including the Finder) are incapable of showing you the complete picture.
Old-school Unix utilities like the df command can give you a better image. On a modern Mac with Time Machine snapshots, there are a lot of nearly-identical volumes listed, since each snapshot on a Time Machine volume is mounted, but it’s not too bad if you just make the Terminal window very wide and scroll up to see the lines you care about:
Of note:
There are some lines that are not actually file systems, but are ways to access device drivers or kernel data. In the above example these are the
devfs
file system (mounted at/dev
, and contains filenames that represent access points for device drivers) and themap auto_home
file system (mounted at/System/Volumes/Data/home
and, I think, is used to represent certain kinds of remote network file systems that may auto-mount in an on-demand fashion).The
-h
option tells thedf
command to display sizes in human-understandable units (e.g.Mi
,Gi
,Ti
, etc.) instead of counts of bytes.Notice that all the volumes on disk1 are identical size and have the same available space. This is because they all belong to a single APFS container and APFS volumes sharing a single container all share the same free space.
All of the time machine snapshot volumes show the same size and available space. This is because, like all other volumes in an APFS container, they share the same free space. Even though snapshots are read-only, they still have “free space”.
Note, however, what is not present:
We know that some parts of the GUI present more information than this in a count of “free” space. For example, the “About this Mac” window includes “purgeable” storage - space that the system can make free, should it need to (probably including caches, temporary files, trashed files and local snapshots). Which is why on my system, it reports 934 GB free, even though
df
reports864 GiB
(which is about 928 GB, not 934 GB):The Finder also reports this 934 GB size, and says that there is about 6 GB (the difference) as “purgeable”:
Disk Utility, on the other hand, is reporting the actual free space (928 GB), since it doesn’t know about purgeable content):
It’s worth noting that in your screen captures, there is a lot of purgeable data, and it varies as you go:
It’s still hard to understand, and I completely agree with your conclusion, but hopefully the numbers will make a little more sense.
Great details, and I agree that the purgeable number is one of the big wildcards in all this. I was hoping that, by working fairly quickly, the changes would reflect only what I was doing explicitly and not some background activity.
But at this point, who knows! If you sit and watch the Available and Used lines in the Get Info window for a drive, they’ll change constantly.
FWIW, some six hours later in the day, Get Info is now reporting 149 GB available, 34 GB of which is purgeable). So the purgeable number has barely changed, but 10 GB of space has disappeared since my final screenshot in the article, even though I’ve done nothing significant. Small amounts of email have come into Mimestream, and I’ve been writing in Google Docs, but certainly not 10 GB worth in either case.
It also occurred to me that you might be seeing the effect of duplicate (or partial-duplicate) files sharing blocks.
If some of those files were duplicated on an APFS volume, then they would be sharing the same blocks. So deleting one wouldn’t create any new free space and deleting both would only free half of the number you’d get by adding up their file sizes.
Thank you for this. I’ve been trying to understand why the available space on my M2 Mac mini seems to jump around unpredictably by many Gb. I’ll try to stop thinking about it now.
By the way, is there any way safely to purge ‘purgeable’ space?
My understanding is that purgeable space is space that macOS can take back if it needs to, not something you can control manually.
Understood. My HD shows 25Gb of purgeable space. I’ll wait and see how much is given back as free space.
It won’t be freed up over time. It will be freed when you start running low/out of free space and macOS needs to make room for something.
IMO, if you are running so low on space that macOS starts purging content, then you probably should have started deleting stuff quite a while ago.
There’s plenty of space, hundreds of Gb. Just like to keep track of what’s happening.
Having deprived us of progress bars, Apple now nominates “free space” as the last true source of randomness.
Instead of “Absolutely Perplexing Fluctuating Space” I’d uncertainly suggest you call it “Approximately Probable Fluctuating Space”.
I didn’t actually assume the F really stood for Fluctuating, but that is me.
These issues are in play for me constantly, not necessary because of snapshots, though those too, but because I’m more than happy to use and abuse my favorite APFS capability, copy-on-write, which enables me to, say, think nothing about duplicating a VM or a sparse bundle or an iMovie library or a Photos library before making changes to it, since those copies take up no additional space. It’s so convenient! I can operate safely without doubling the space I’m using! But it also means that disk math is impossible, because of course it appears that I have hundreds more GB in use than I actually do.
The part that feels frustrating to me is that, in theory, a utility could analyze APFS data structures and theoretically figure this stuff out – I mean, after all, the computer has to know at some level what’s a lightweight duplicate or snapshot versus a block not otherwise represented. But I’m unaware of any tool, even seven years later, that works beneath the APFS file level, other than, of course, fsck_apfs. I don’t know if this is because Apple hasn’t released sufficient technical detail about APFS to make it possible, or whether it’s just because it’s so complex that no one wants to make it, but it’s a bummer. We could use it.
Also problematic is that if you make use of these APFS features, they’re part of the largely non-copyable container they’re in, meaning what fits on one computer may well not fit on another drive of the same size. This was a problem for me when cloning one computer that had about 1.9 TB used on a 2 TB drive – Migration Assistant just refused. I ended up using the Legacy Clone feature of Carbon Copy Cloner, which utilizes a somewhat unreliable Apple-provided underlying container copying mechanism, but it’s hardly something I’d want to depend on. I’d at least think Migration Assistant ought to have the capability of figuring out what is real space occupied when moving from machine to machine, but, no.
As for whether it’s possible to intentionally purge purgeable space – DaisyDisk does this, and in general has long been a tremendously excellent tool for managing disk space. It’s right up there with Carbon Copy Cloner as far as essential Mac tools go for me. I don’t think it has any way of knowing about what’s an APFS copy-on-write clone and what isn’t, but it at least helps me figure out how much space is hidden (meaning, snapshots and other stuff outside the immediate file system), and what the big fish are – or at least big fish candidates – when I’m looking to clear some space up.
Yeah, as soon as you get into any kind of behavior like this, the APFS Uncertainty Principle kicks into play big time. I was just revisiting an article Howard Oakley wrote about disk images, and how you can create a non-sparse image only to have it converted to a sparse image, but depending on what you do with it, it can revert to being non-sparse again.
I imagine that an APFS audit tool could be created, but it might be of relatively limited utility, given that the Fluctuating part of APFS would render whatever number it reports quite variable. It might also take a fair amount of time to run, as it evaluates every sparse image and duplicated block and snapshot to see how much data is really in use, such that it might not be accurate even by the time it finished.
Ah, I missed this one from Howard. Thanks for invoking it – fascinating reading.
YMMV, but I’d be more than happy to let something like this run all night to yield only ballpark accuracy! My question is whether Apple has even released enough information about APFS to make such an audit possible. Or to allow a program like Carbon Copy Cloner to theoretically recreate the internal APFS relationships on a target volume, so that that it ends up with roughly the same space used as the source volume, without having to unreliably block-copy the entire container. (And if not, couldn’t at least Migration Assistant do that? Or an expansion of “diskutil apfs” or “cp”?)
What about a tool to analyze your drive to utterly maximize disk space by identifying redundant true copies of things and turning them into lightweight APFS clones, or sparseify anything that can be sparseified? It would just be cool if the doors were open for devs to give power users tools to do powerful things, rather than one’s file storage being an opaque, unknowable, dynamic organism. Though, of course, Apple has been trending away from that for quite some time in most domains. (Sometimes you get something new and surprising though, like Shortcuts.)
I don’t know if I really am capable of going full zen on my disk space, as much as I’d like to; regardless, I don’t have much choice but to accept the mystery.
Yeah, this drives me nuts! After being a Mac Consultant for over 33 years, I’ve become increasingly critical toward Apple’s continual disregard for ease of use and understanding for their non tech users.
This got critical for me on a previous Mac that had a lot of media files on it, when I tried to install an OS update and was told “not enough free space.” That caused me to do a dive (not as deep as Adam or David C.) and learn about local Time Machine backups, and how to purge them. So this can be more than just an ‘academic question’
And to The Mac Doctor: I’ve always been very pissed off at Apple’s “an error has occurred” (and there’s not a damn thing you can do about it) attitude towards error logging, etc. I’m not sure which is worse, “An error has occurred” or “Error 4231” with NO WAY to find the meaning for that particular error code. Sometimes I have been able to spend some time (hours) digging through console logs, etc, to figure out what went wrong. As Mac OS gets increasingly more complex (a lot of that due to paranoid security), things are much more likely to break in strange (and often not repeatable/Heisenbug) ways.
Thanks for this coverage! It’s enough to validate that I’m not crazy or an idiot since I can’t figure out what appears to be simple math… calculating free drive space.
Besides COW, another way that storage is oversubscribed is Time Machine’s use of hard links, if I recall correctly. Hard links are additional directory entries in a filesystem pointing to the same inode. The data will only free up when its link count drops to zero. Until then, some ways of counting consumed space may double report such storage.
But I’m probably off topic now
I think that the use of APFS snapshots in Time Machine is instead of hard links. Can’t remember where I read that, but I’m pretty sure hard links were the best that could be done on HFS+, but snapshots are a more robust and elegant solution for Time Machine incremental backups.
Aha thanks for that! Yea I learned what I learned from a white paper when it first came out I think.
More off topic, but I’m not a fan of APFS. It seems to be nothing but trouble. Drives I cannot boot from or backup, all kinds of ghost images floating around in utilities that display drives, corrupt file systems that cannot be repaired…
We explored ZFS ages ago and abandoned it, right? I wonder why that was… ZFS has emerged as the standard elsewhere, such as for Proxmox virtualization.
Correct. Time Machine on HFS+ (often called “TMH”) is based on massive amounts of hard-links in order to avoid duplicating content that is unmodified from one backup to the next.
Time Machine on APFS (aka “TMA”) is based on snapshots. So the underlying file system handles the deduplication of content. In general, it is faster and more robust than TMH, but has the disadvantage that there is no way to purge a single file from a set of backups (which you could do with TMH by deleting a file from every backup).
I recently reinstalled Ventura and all my apps and data on a new Mac mini. While I am confident that there is around 300Gb of free space left I cannot get consistent figures. About This Mac, Finder, Disk Utility, and System Settings all give different results varying by as much as 250Gb. Further the breakdown by app type within System Settings produces nonsensical numbers (e.g., it tells me I have 13Gb in TV but the folder is empty).
Another example. Finder tells me Xcode is 22Gb but it also tells me that the package contents amount to 8Gb while System Settings > General > Storage > Applications says it’s 12Gb.
This is ludicrous.
I did some similar experiments on my Ventura system, and offer the following observations:
If I wanted to figure out how much space I had available on my disk, I would take the Available figure in the Get Info box and subtract out the Purgeable space. I shouldn’t have to do that - the Finder needs to do a better job of making this known.
In dumbing down the Finder experience, Apple has masked over valuable information that they shouldn’t have, IMO.
If I subtract “purgeable” from “available” space as shown in Get Info for Macintosh HD I also get the Free space shown in Disk Utility.
I suspect that Ventura classifies my audiobooks in the Books app as purgeable as it is possible to shift this to Apple Cloud. However, I have over 250Gb of audiobooks and have no intention of paying Apple for that much storage when I have already paid them for the SSD on my Mac mini.
Curiously the free space shown by Finder flips during the day between the high “Free” number shown in Get Info and a number closer to that shown in Disk Utility. However, it returns to the high number in the morning.
By the way, I asked a Genius Bar advisor about this earlier this week and they said they do not use Finder but use DaisyDisk instead. Another vote of confidence.
Media you purchased from Apple’s on-line stores (Music, Movies, Books, etc.) is purgeable because (in theory) you can always re-download it. And this content doesn’t count toward your iCloud storage, because any re-downloading comes from the store’s servers.
Your own content (not purchased from Apple), on the other hand should not be purgeable unless you are already storing it in iCloud somewhere, since you would need that storage to re-download it.
I wonder if that’s official or just what that particular tech (or his store manager) uses.
FWIW, if I want to track down what’s consuming all my space, my go-to app is Disk Inventory X, based on the open source kDirStat for Linux and its Windows port WinDirStat.
The audiobooks are from my own sources not Apple so should not be classified as purgeable. However, there’s no other combination of folders that adds up anywhere close to the purgeable number.
DaisyDisk was a personal choice of the advisor.
I’m prepared to leave this entire topic as an Apple mystery. Maybe it will be cleaned up in a future update or macOS 14 and maybe pigs will fly.