Carbon Copy Cloner Backs Up Cloud-Only Content
When I wrote “Apple’s File Provider Forces Mac Cloud Storage Changes” (10 March 2023), I closed the article by noting that backing up data in cloud storage services was potentially fraught because online-only files wouldn’t be backed up. I recommended downloading your entire cloud storage data store, at least temporarily, so it would all be included in your backup. New files you subsequently created would be local and thus backed up, but there are likely situations where files created or modified by collaborators would not be properly reflected in your backup.
That is, unless you’re using the most recent version of Carbon Copy Cloner, which can temporarily download cloud-only content, back it up locally, and evict it again to avoid consuming too much local space. Agen Schmitz mentioned this feature in his Watchlist item (see “Carbon Copy Cloner 6.1.7,” 11 September 2023), but I wanted to call it out more explicitly because it’s unusual and clever. Bombich Software explains:
When a file stored by one of these storage services is flagged to reside only online, the local copy of the file is deleted from your Mac and replaced with a 0-byte placeholder file. While this is a convenient feature that allows you to free up some space on your Mac, this feature imposes a logistical challenge to creating a local backup of those files. If you want to have a local backup of these cloud-only files, CCC must temporarily download these files to your startup disk. CCC can do this, but because this involves downloading a potentially large amount of data from the Internet, this functionality is disabled by default. Likewise, allowing this data to co-mingle with your startup disk’s backup could lead to a situation where it is impossible to restore your entire backup to the original disk due to space constraints. To avoid that, we recommend making backups of your cloud-only storage to a separate volume on your backup disk.
The page continues on to provide complete instructions and explain why some iCloud-only content still won’t be temporarily downloaded (because Apple isn’t yet using its own File Provider technology for iCloud, ironically), among other interesting technical details. It’s worth a read if you’re interested in quirky backup topics or the integration of local and cloud storage.
And, of course, if you’re highly concerned about maintaining local backups of cloud data, add Carbon Copy Cloner to your backup strategy.
Could this feature be used to migrate iCloud data to a different account?
I don’t think so because Carbon Copy Cloner is managing the existence of the local files. I’d need to know more about what you were trying to achieve to be able to make a better suggestion.
A relative has two Apple ID’s, one w/ all the purchases, the other w/ iCloud. The iCloud account is @me.com which should explain a bit. Ideally, they would like to consolidate iCloud to the account w/ the purchases, but reassign the @.me address to that one, and either close or just abandon the other. The reassignment of the addresses seems doable, but I have no idea about transferring the iCloud data. (I just say “NO” to iCloud.)
Apple is apparently adamant about not transferring purchases.
Thanks, Adam!
I personally own a laptop with a large enough drive to maintain all of my files. About a year and a half ago I made the mistake of turning on iCloud, and it removed quite a number of files from this laptop drive so that I do not have access to these files when I am off-line. This can happen quite often.
I have been using Carbon Copy Conner for years and I have backed up all the data on this laptop drive onto a separate drive on my network.
Is there a way that I may use Carbon Copy Cloner (or some other method) to maintain originals of all of these iCloud files on my laptop so my complete copy of files exists there, exists on iCloud, exists on my Carbon Copy Cloner back up, and is available for utilization with my iPhone iPad, and other devices, which do not have large drives so they can grab them as needed?
Paul
In theory, Apple fixed this bug in Sonoma. If you have Optimized Mac Storage turned off, all iCloud Drive files should exist on your Mac, but apparently, that wasn’t always true before.
There shouldn’t be any issue with doing this—it’s just work. They might look at these Take Control books for advice:
And no, purchases can’t be transferred.
That’s what I thought, Adam. I will recommend the TCO’s.
Thanks.
PS.
Do you know if FastSpring is sorted? My relatives patience is already thin.
Saints be praised! I open the file which I haven’t opened since 2000 and lo and behold. The information was there. So now my question is what does that little cloud icon with the “!" In the middle of it mean?
Paul.
FWIW, I just purchased the TCO Apple ID book. It went through Paddle.
As you saw, Joe’s working on switching to Paddle—sounds like that’s just happened.
According to this page, you’re out of iCloud space.
Now that is odd. What am I missing?
I was afraid you were going to say that.
I wonder if the ! icon also means that there’s some sort of sync error. What happens if you move the file out of iCloud and back in to force a resync?
Sorry for taking so long getting back to you. I was in the hospital for a tuneup and I’m doing well.
It appears most of these documents are really old ones. I moved one out of the documents folder into the downloads folder, which is not stored on iCloud, and then put it back into the documents folder, using command Z so it went back to exactly where it was. The cloud icon with the “!" blinked on and off for a bit however, after about five minutes, it stopped blinking and the cloud with the "!” remains.
In the left-hand column of the Finder, the iCloud Drive appears to be doing a synchronization. By that I mean I see a pie with a slice of it next to the title with the cut out of the pie getting larger oh, so slowly. My net work appears to be uploading anywhere from 15 to 20 KB per second. No wonder this is taking quite a long time.
Paul.
Glad the tuneup went well—always good to get your oil changed every so often.
Doesn’t sound like an easy fix, but @glennf just wrote about a number of things that can unstick iCloud, so perhaps run through his collection.
Is there any other backup app that will backup the files stored directly in iCloud Drive?
Similarly, do any of the cloud backup companies like BackBlaze actually backup files from your Desktop if you have “Optimized Mac Storage” and some of the files are removed from your local storage?
Edit: I ask because my dad uses iDrive backup, and he’s running into errors on his automatic backup. Their support tells him the problem is files he’s marked to backup are not actually on his local disk because of this iCloud feature.
This will solve the problem of my daughter backing up her Dropbox Pro files which are online-only.
What I don’t see is a solution to the other half of her problem: backing up her iCloud Photo Library using optimized storage. That is hundreds of GB. Without that, this is yet another partial solution at best.
I have opened a ticket with Bombich today. I’ll let you know what I hear.
Mike from Bombich offered a nice, detailed reply which effectively said it’s not plausible, as I expected:
So this is too bad because it means the CCC solution doesn’t solve the problem of backing up an optimized Photo Library. A half-backup is not a backup. I love the solution he has, though, and I might try it for Dropbox. But the fault is Apple’s that Photos has no easy solution, and I’m not happy about it.
The thing about cloud storage is that the vendors will assure you there is no need for you to have a local backup–they have extensive backups to guard against loss. Of course, if you want quick local access to some files, or you will temporarily lose access, you can mark files also to be stored offsite.
In the real world, you may be skeptical of those claims, and Carbon Copy Cloner has provided a method for backing up most cloud contents. However, as I understand it, you must have enough local space on your main drive to temporarily store the backup files until they can be copied to the backup disk. The issue with photo and media libraries is that you may need more space to store the files temporarily.
Specifically with iCloud Photos, Apple provides an option to store a copy locally. You can even put it on an external drive. Carbon Copy Cloner would have no problem backing that up. However, if you can’t do that, you are in your daughter’s situation.
I have an iPhone, iPad, laptop, and desktop Mac, all using iCloud Photos. My collection of about 16,500 photos and videos takes a bit under 200GB. I have optimized on all devices except my desktop, which is backed up nightly.
So, you could use the following process to get a backup of the library:
This procedure is time-consuming, and you may only want to do it infrequently.
So, the surest way to ensure your photo library is easily backed up is to have one regularly backed-up computer with enough storage to contain the whole library comfortably so that you don’t need to optimize photo storage for that computer.
At one point, I experimented with moving my photo library to an external HDD but found processing slowed significantly. So, I moved it back. I’ll put it on an external SSD if I run out of space.
Yes, and this is exactly what I do. I requires me to maintain a “server” (iMac) that has external drives to store my 1.5 TB photo library. This is not something people should have to do. But I put up with it.
But I have a family of 5, and my kids all have their own photo libraries. And what you described is not scalable to multiple people. I discussed this in another thread where I showed the problems: the other user needs an account with their own Apple ID; that account must always be logged in, even after reboots; that account must have an external drive, since Apple refused to allow iCloud Photo Libraries to reside on NAS devices like my Synology. At every step, Apple makes this implausible.
Not much. 2 GB max.
https://bombich.com/kb/ccc6/limitations-online-only-placeholder-files
CCCs solution is really perfect, except for 2 big gotchas:
I speculated to Mike Bombich that really only Apple can provide a solution to making a low-footprint local backup of if your iCloud Photos, and the place they would do that is in Time Machine, where it should be. While they’re at it, they should also provide a backup for online-only files, as CCC has done. Apple created this problem; they might have to fix it.
Yes, I absolutely am skeptical of cloud provider claims that they can’t lose our files. Witness Google Cloud Drive a couple of days ago…
Besides, syncing is not backup! If I accidentally delete a photo, I want to be able to recover it. That’s what having a backup allows.
Let’s be clear. Apple has intentionally designed the system so as to make it nigh impossible to use Photos without paying them monthly for storage. I won’t bother to speculate why they did this, but one could be forgiven for thinking that their commitment to growing the “Services” line of their income might have something to do with it.
Correction. They made it nigh impossible to use Photos with iCloud storage without paying them for a storage upgrade.
There is absolutely no problem using it with local-only photo libraries. Which is exactly how I have always used it.
Yes, the libraries can get very large. If mine ends up getting too big for comfort, then I’ll probably create a new library and move the old one to external (and probably slower) storage.
You might be able to create a work-around for the Photos backup issue by using this method: In Photos, select a number of photos (I’ve done this with up to 200 at a time, but you could probably select more). From the Share icon, choose “Copy iCloud Link.” It takes a few seconds to create the link, but when it is finished, paste the link into a browser and then you can download all the photos you selected. Save them to a folder that you have labelled appropriately (because the file names of the photos are generic), and run CCC and/or Time Machine. Once you’ve downloaded them and backed them up, you can delete them because the originals are still in Photos. The biggest downside is that you probably can’t download and back them all up in one go; you have to get copies from iCloud in batches.
How does this differ from just using Photos’ export menu (or drag/drop from the app to a folder)?
Probably not at all. I always use the iCloud link because I can save the URL for later use.
Doesn’t it expire after 30 days?
That’s a good question. I don’t know. I have never tried to use one after 30 days. I don’t personally use iCloud links for backup purposes, I just thought it might be a useful answer to the problem of downloading photos from iCloud for backup. I take hundreds of photos a day and send them around to various people who need local copies, and that’s how I do it. Nobody has ever complained that a link expired, but there’s a good chance nobody has ever tried to download any of my photos after 30 days (also I usually delete most of those photos by 30 days so, for me the question is largely moot).
I do not optimize storage by having Apple move files to iCloud Drive, but I do use iCloud Drive.
Thank you for the link to the Apple support page. I had been wondering what some of those icons (mainly the Download status icon and the Waiting to Upload status icon) meant.
I had a file that had the Waiting to Upload status icon. It had been been that way for over a week. I confirmed it was not available from another Mac logged into my Apple account. I moved it from iCloud Drive to Documents, then moved it back. (I did not use cmd-Z.) After a moment, the status icon changed to Downloaded (shouldn’t that be Uploaded?) and it was available on the other Mac. Note that the vast majority of my files in iCloud Drive do not have any status icon shown.
Aw, nuts. Now I need to read and try to understand another article. (Howard Oakley’s article taxed me. I hope to do better with Glenn’s.) And probably get a couple TC books.
Thanks for the links.
I just noticed that backup software Arq, which I also use, appears to have support for “materializing” cloud-only files that are encountered:
It appears that was added quietly back in April:
https://www.arqbackup.com/download/arqbackup/arq7_release_notes.html
What I can’t find in the Help or online is any explanation about how it handles this “materialization”. The magic of CCC is that it pulls down no more than 2GB at a time, then releases those files back to the cloud, ensuring your disk space doesn’t fill. If Arq doesn’t do that, then it will basically be undermining people who (often pay extra) to have smart-sync logic that frees up local disk space in favor of pulling things down only as needed.
I’ll ping the dev for info.
From the dev:
I don’t think this will play well for people with limited local storage. Currently, I only have Arq running on my Mac “server”. This is basically a Mac I maintain with a ton of storage just to solve these inadequacies in backup solutions. Very expensive workaround to a critical problem.
So here’s the roundup:
CCC manages local storage in the face of cloud-only files well, but doesn’t support cloud destinations for its backups. You can setup a remote Mac, but that’s more hassle. So this makes it hard to have good off site backup support.
Arq handles cloud destinations well, but doesn’t play well for people using cloud-only files because they have limited local storage. (Does Backblaze solve this one?)
Neither one knows how to solve the problem of backing up an Optimized iCloud Photo Library
As I see it, users have no good backup solutions at this time without either 1) trusting the cloud, or 2) ditching all cloud-only and optimized features.
Am I missing something?