Apple’s File Provider Forces Mac Cloud Storage Changes
Over the last year, cloud storage services Box, Dropbox, Google Drive, and Microsoft OneDrive—and probably others—have migrated from custom kernel extensions to Apple’s new-ish File Provider extension. It provides an Apple-approved framework for integrating remote files into macOS and displaying them in the Finder. I touched on this move a year ago in “Cloud Storage Forecast Unsettled, with Possible Storms” (4 February 2022).
The File Provider extension first appeared in macOS 12.1 Monterey, though Apple’s Developer site suggests that it may have subsequently been made available in macOS 10.15 Catalina. Earlier versions of macOS don’t support it, so nothing changes for those running older systems for now. The cloud storage services will likely pull support for older systems at some point, but there’s no telling when that will happen. (Everything I say below I’ve tested in macOS 13.2.1 Ventura—it’s possible, even likely, that some details are different in Monterey.)
The cloud storage companies undoubtedly felt they had little choice about adopting the File Provider extension because Apple said it would be deprecating kernel extensions at some point. That hasn’t happened yet, but Apple often gives developers several years to get with the program.
My understanding is that Box, Google, and Microsoft have migrated their Mac users to the File Provider approach, whereas Dropbox—probably the most popular among everyday Mac users—has only recently started to encourage those outside its beta program to switch (while others are still being asked to join the beta). It can be hard to tell since cloud storage providers often roll out changes over time or to subsets of customers to test user response, identify concerns, and reduce support loads. Some Dropbox users even report one Mac being upgraded to the new File Provider approach, while others continue to use the kernel extension.
Although it’s good to have cloud storage integrated into macOS in a coherent fashion, the switch to Apple’s File Provider extension brings with it two key changes that will require adjustments in how you work.
The first change is where local files are stored. All cloud-storage services must now store their files in ~/Library/CloudStorage. (That’s in your home folder’s Library folder, which is non-trivial to access for most users since the Library folder is hidden by default.) From a consistency and coherency point of view, the new standard location is a good thing, since users don’t have to keep track of where each individual service might have its files. However, just as when Mac OS X introduced the home folder and its now-canonical subfolders (Desktop, Documents, Library, Movies, Music, and so on), the loss of flexibility will create pain points for some users.
The second change revolves around whether the files exist both locally and online (“offline” or “pinned”) or just online (“online-only”). Both have advantages and disadvantages:
- Offline files are stored both in the cloud and on the local drive for quick access. However, they consume local storage space, which may be a problem if you’re low on free space. There’s no visible indication that a file is offline.
- Online-only files exist locally only as icon placeholders until they’re opened—they have little cloud icons next to their names in the Finder. This approach saves space, but may suffer from loading delays with large files or slow connections. Of course, online-only files aren’t available if you lack connectivity.
After you first migrate a cloud storage service to the File Provider approach, assume that all your files and folders are online-only. That might not be the case—you can tell by looking for the little cloud icons—but such files, as I’ll discuss below, can’t be viewed using Quick Look, indexed by Spotlight, or backed up by Time Machine and other backup apps.
You can control the state of any file or folder. If you click the cloud icon next to a file or folder name, that initiates a download, turning it from online-only to offline. Control-click an offline file and you’ll find commands for downloading files (making them offline) or removing their downloads (making them online-only). Dropbox (below left) relies on Make Available Offline and Make Online-Only commands in the Quick Actions section at the bottom of the pop-up menu. Google Drive (below right) puts Download Now and Remove Download commands at the top of the menu; it has Quick Actions as well.
Of course, opening an online-only file in an app also downloads it. It’s possible that a third-party app would be unhappy when fed a file that hasn’t yet been downloaded; if you get an error, make sure the file is available offline and try again.
To be clear, this distinction between offline and online-only files isn’t new or unique to the File Provider extension. All the cloud storage services have provided ways to specify which files or folders were kept locally and which existed only online. The details and implications varied somewhat, but the concept was the same.
Let’s look at some of the impacts of these File Provider changes. I focus on Dropbox and Google Drive below because those are the services I use regularly, but Box and OneDrive should be similar in most cases.
Sidebar Locations May Change
The File Provider approach brings cloud storage services into parity with one another and with other forms of storage in macOS. As a result, Apple has decreed that they appear by default in the Finder window sidebar under Locations. You can even turn them all on and off in Finder > Settings/Preferences > Sidebar.
The new spot may cause some minor confusion, if you previously kept the Locations item closed in the sidebar or regularly accessed Dropbox or Google Drive, for instance, from sidebar items under Favorites. However, you can drag any cloud storage service’s sidebar item from Locations to Favorites or even have them in both Locations and Favorites.
This change is minor and easily tweaked however you prefer, but I mention it first because you should click any sidebar items you have and make sure they still work and, more importantly, still work properly.
Look Out for Disconnected Local Folders
Why might a sidebar item not go to the right place? When a cloud storage service switches from its custom kernel extension to the File Provider extension, the location of its local folder moves from wherever you had it to ~/Library/CloudStorage. Or at least that’s the goal.
(As a quick reminder, if you don’t see the Library folder in your home folder, hold Option and click the Go menu, which makes Library visible in the menu. Or, with the home folder open, choose View > Show View Options and select Show Library Folder to make it always visible.)
Some users, including us, have experienced situations where the local folder doesn’t move but becomes disconnected from the service. Since the cloud storage service will immediately sync to the new location, you can end up with two Dropbox folders, for instance, one at ~/Dropbox and another at ~/Library/CloudStorage/Dropbox.
The cloud storage services somewhat muddy the waters by creating what looks like an alias (really a symlink) in their old locations to maintain continuity for users. But is that Dropbox folder in your home folder a symlink or a disconnected local folder? Check by opening Terminal and typing ls -l (if you’re not already in your home folder, use the cd command first). As you can see below, I now have Dropbox and Google Drive folder symlinks in my home folder. Notice how the symlinks have an l in the leftmost column, whereas directories like Downloads and Library start with a d and the Dropbox Demo Alias I made for illustration has a -.
Verify your findings by copying a new file into the local folder and then checking on the cloud storage service’s website to see if it was uploaded.
If you do have a disconnected local folder, you’ll have to do some spot checks to make sure everything in it also exists in the online version. I do that by using List View in the Finder, sorting by Date Modified, and then expanding the folders and checking that the most recent files are in both places. When I ran into this with Google Drive, I had only a handful of files that were out of sync, but if you see a lot of differences, consider using synchronization software like ChronoSync to figure out which files need to be moved.
Be sure to delete the disconnected local folder once you’re done to avoid accidentally using it later. If the service doesn’t subsequently create a symlink automatically for you, I recommend making one with these steps:
- In Terminal, type
cdand press Return to ensure you’re in your home folder. - Type
ln -s(leave a space after thes). - In the Finder, choose Go > Go to Folder, type
~/Library/CloudStorage, and press Return to open that folder.
- Drag the cloud storage service folder—Dropbox or Google Drive, say—from the Finder to the Terminal window to insert its path. This approach is necessary because at least Google Drive actually has a different name than what you see. Press Return to create the symlink.
References to Local Files and Folders May Break
In an ideal world, if the cloud storage service created an alias of its relocated local folder from ~/Library/CloudStorage to wherever you had it previously, any aliases or other references to files or folders in that folder should continue to work.
However, that’s not always the case, especially if you had to clean up a disconnected local folder. If you deleted your disconnected local folder, as I recommend, any aliases that pointed to files or folders in it will break. That’s good, since then you know to recreate them.
Similarly, some apps like BBEdit, Keyboard Maestro, and Transmit can optionally store support files in cloud storage as a way of syncing settings and other data between Macs. In the ideal world described above, everything should keep working, and that was my experience with my most recent Dropbox migration with a few Keyboard Maestro macros that reference Automator workflows and BBEdit text filters that I had stored in Dropbox.
However, there are no guarantees that references to files will all resolve to the new locations, and in the case of apps that use cloud storage for syncing settings, you may not even notice immediately. For hints about which apps might be leveraging cloud storage for syncing, look through your various cloud storage folders, Dropbox in particular.
External Drive Support Disappears, Except for OneDrive
The most significant limitation of the new File Provider approach is that ~/Library/CloudStorage is located on your internal drive. For those with terabytes of files in a cloud storage service, that’s a huge problem.
Previously, you could specify where you wanted your cloud storage service’s local folder to live, making it easy to locate it on a large external drive. With the File Provider approach, that’s no longer possible. Or so Box, Dropbox, and Google are saying at the moment, although Microsoft has worked around the limitation with OneDrive by separating the “sync root” (which has to remain in ~/Library/CloudStorage) from the cache, which can live on an external drive.
Users are up in arms about this change, particularly those who use Dropbox to keep audio or video processing workstations updated with very large shared files.
The news isn’t entirely bad. On 17 February 2023, Dropbox posted a note in a long and vitriolic thread about the loss of external drive support, saying that the company was working on a solution and users who have their Dropbox folder on an external drive wouldn’t be upgraded in the meantime.
If waiting for Dropbox isn’t an option, consider the open-source Dropbox client Maestral. It allows you to locate your Dropbox folder anywhere you want, including on an external drive. Maestral relies on the public Dropbox API, which prevents it from transferring only the parts of a file that change from save to save—a binary diff. Instead, Maestral has to transfer entire files, which will be slower and use more bandwidth. Maestral doesn’t count toward Dropbox’s three-device limit for basic accounts (see “Dropbox Limits Free Accounts to Three Devices,” 14 March 2019).
There might be another workaround, which is to relocate your entire home folder to an external drive. Control-click a user in System Settings/Preferences > Users & Groups to access the Home Directory field. Don’t even think about doing this—even with a test account—unless you’re technically adept and have made multiple backups first.
Even assuming that relocating the home folder to an external drive works as desired, there are negatives. It won’t work well for laptops, of course, and if you don’t wish to shell out for an expensive SSD, you’ll suffer from the slower performance of a hard drive. In addition, there’s some online discussion about how installing macOS updates while logged into an account located on an external drive can cause problems; the advice seems to be to update macOS while using an administrator account on the internal drive. If you try this, let us know in the comments how it works.
For thrill-seekers who have the necessary technical chops and excellent backups, there are other options you could try—again, let us know in the comments if either of these works:
- Dropbox: If you’ve already migrated Dropbox to the File Provider approach, a user posted a possible workaround in another Dropbox Community thread. It involves using a symlink to point at the Dropbox folder on an external drive, similar to what Microsoft does with OneDrive. In the recent past, Dropbox hasn’t supported symlinks to items on external drives, but perhaps that’s changed.
- Google Drive: Although Google Drive claims you cannot change its location, a May 2022 discussion in the Google Drive Help forum recommends using the Unix
touchcommand to create a file in a particular location. The suggestion is that doing so causes the Google Drive app to revert to pre-File Provider behavior.
Dragging Files Moves Rather Than Copies
Some of the cloud storage services used their kernel extensions to treat their local folder as a separate volume. The practical upshot was that dragging a file to or from the service’s local folder resulted in a copy action, leaving the original in place and making a new version in the destination location. Reasonable people can differ about whether this was the right or wrong thing to do, but regardless, it’s what users became accustomed to.
With the File Provider approach, all the cloud storage services are essentially folders on the internal drive, not separate volumes, so dragging a file from your Desktop to your Dropbox folder moves the file rather than copying it, as does dragging a file from Google Drive to the Desktop.
That’s an important change because the services also have to maintain an online copy of the file, so at least Dropbox and Google Drive interpret moving a file to another folder in the Finder as a deletion of the online version, putting the moved file in the service’s online trash. Both also alert you to what’s happening, and you can turn the alerts off.
Trashing Retains Online Copies, but Local Behavior May Vary
So what happens when you use the Finder to delete a file from a cloud storage service? It varies, and I can speak only to what happens with Dropbox and Google Drive, but I suspect Box and OneDrive will be similar to one or the other of these approaches. If that’s not the case, please contribute additional details in the comments.
With Dropbox, you’ll see different behavior based on whether the file is offline or online-only. When you trash an offline file, it simply moves to the Trash like any other file in the Finder. You can put it back, move it to another location, or delete it permanently by emptying the Trash. However, when you delete an online-only file, Dropbox alerts you that the file will be deleted immediately (below left). When Dropbox says that you can’t undo the action, it means that it’s not putting the file in the Mac’s Trash, so Command-Z won’t work. However, deleting either type of file moves it to the Deleted Files collection on the Dropbox website, where it stays for 30 days before being deleted permanently (below right).
Google Drive has no such distinction. When you delete any file in Google Drive, offline or online-only, it moves into the Mac’s Trash like any other file. You can press Command-Z to undo the deletion. Online-only files retain their cloud icons while in the Trash. As with Dropbox, Google Drive also puts deleted files in its online trash, as you can see with the two ChartOfAccounts files I’ve deleted in the screenshots below. Again, deleted files remain available online for 30 days.
Searches May Work Poorly for Cloud-Based Content
The offline/online-only state of files has implications for searching. Unsurprisingly, if a file is online-only, there’s no way Spotlight can index its content. However, filename searches in Finder windows should work, as should searches in third-party utilities like EasyFind or Find Any File.
Google Drive offers an easy workaround for content searches. You can bring up a custom Google Drive search dialog in the Finder; the default keyboard shortcut is Command-Option-G. It runs the search on Google Drive on the Web, which can easily search the full content of your files stored in Google Drive. It also finds content in Google Docs and Google Sheets, which wouldn’t ever show up in Spotlight, since they exist on local drives only as pointers to the online data.
Backups Work, But…
By now, you can see that online-only files don’t really exist on your Mac. That means Time Machine and other backup solutions cannot back them up. It appears as though all Dropbox folders appear in Time Machine, though only offline files show up in them. In contrast, Google Drive’s folders only appear if they contain offline files. Similarly, Backblaze sees and backs up offline files in ~/Library/CloudStorage, but it doesn’t know about online-only files at all. That’s why there are only two files in the folder shown below.
As I noted at the top, you should assume that everything in a cloud storage service is online-only to start and thus isn’t being backed up separately. That might not concern you—after all, the data all exists in the cloud. However, if you’re sharing folders with other people, it’s possible for them to delete files or entire folders accidentally (or maliciously), thus removing them from everyone in the share group. In theory, they should be available in an online trash or Deleted Files collection, but you may not want to bet on that. A local backup made with Time Machine, Backblaze, Super Duper, or any other app can protect you from such mistakes.
Such things happen—Tonya recently ran into a situation where someone in her workgroup inadvertently deleted an important Box folder containing weeks of work. There might have been other ways of recovering it, but copying from her Time Machine backup was quick and didn’t require opening a ticket with IT support.
Therefore, assuming you have sufficient disk space, I recommend downloading your entire cloud storage data store, at least temporarily. Bring everything down and let Time Machine and your other backup systems make a copy. Then, if you want to recover the disk space, reset folders you don’t need to online-only. New files you create will be offline because you’re working on them, so they’ll automatically get into the backups.
Phew! When I started this article, I thought it would be a fairly short explanation of how the Dropbox folder had moved after migrating to the new File Provider extension. Three days of research and testing later, I think I’ve wrapped my head around all the details. But perhaps not! If you still have questions, ask them in the comments.
Adam, brilliant article. The shortcomings of Dropbox—the cost of more than two devices apart—are becoming more & more obvious. And its new interface—fairly awful. Where Dropbox is concerned, I’m leaving.
Oh, the complexity. I just moved everything from Dropbox to iCloud myself. I bought a copy of one of the take control books on iCloud to help me understand it. Now here’s Adam, telling me everything I wanted to know about iCloud, but only in the context of how it works with Dropbox and the other third party apps, leaving me with only hints of how any of this affects iCloud itself. I’m getting too old to scrutinize these articles in depth to decipher how to help me understand iCloud. I was hoping iCloud might help me deal with the tiny hard drive in my new MacBook Air, and I can see that there is no hope! Will iCloud eventually fill my drive with local copies of online files and leave me no way to make them online only? I haven’t found a way to mark files as online only in the Finder. I remember the ease of the Dropbox extension via the contextual menu with despair.
The problem with iCloud has always been that it decided itself to evict files to make them online only. Even when there was enough space on the hard disk.
And now Dropbox is starting with similar stupidity. I add a file to Dropbox on my iMac. Then I have to download the file manually on the MacBook Air with Ventura.
These TidBits articles are one of the many reasons why I gladly subscribe to Tidbits.
Dropbox has been notifying me for a week now that “Dropbox for MacOS is now ready”, but I have held off, waiting for just the sort of information you have presented so well!
Thank you Adam.
So many things about these changes are going to break my existing use of these tools, such as using Google Drive to synchronise my Documents and source code folders for an instant “backup” (yes, I know it’s not really a backup).
Is the offline file access only integrated with Finder, or does it go deeper like the hierarchical filesystems of old, where a recursive
grepcould trigger a retrieval of every file from nearline storage? Or does it have the opposite issue, that regular applications cannot access online-only files without support for requesting they be fetched?What happens to Dropbox files if you drop the service? Is there a way to check that the files have been deleted or do you delete before you cancel the service?
You should be fine using iCloud Drive this way. Making a file online-only is just the same as with Google Drive—Control-click the file or folder in question and choose Remove Download. I don’t know why Dropbox doesn’t present those commands.
I haven’t done the necessary testing to verify this either for iCloud or other services, but it’s an interesting question.
One thing that does seem to be lacking in the File Provider world is a setting to make everything offline all the time, disk space permitting.
That’s a good question, and I don’t know the answer. I have verified that the Unix
morecommand will force a download, buttouchdoesn’t seem to. Perhaps someone with actual Unix chops can weigh in.Thanks to @peternlewis for nudging me to fix what I was saying about an alias of the cloud storage folder in your home folder. It is of course a symlink, not an alias, so I’ve updated the article to clarify how to tell and how to make one if you don’t get one automatically.
I’m getting increasingly annoyed at Apple’s insistence I can only organize files/directories/information -their way-. And I fear the further convergence of iOS and Mac OS will make this much more annoying.
(For me, the serious annoyance started with the changes to the Files dialog box in the Apple built-in applications, particularly Preview, several years/versions ago. In particular, removing “save as” as a choice without a modifier key pisses me off each time I want to put a file into a -specific location of MY CHOOSING- within a file hierarchy.)
I think I can explain this. “touch” only updates the file attributes, not the contents. “more” or (or “less” :-) ) access the file contents.
A big fat
to this. I’d like a big master switch somewhere that allows me to turn everything to offline-only all the time. Make Dropbox and iCloud work like Dropbox used to when it was new. I get that some folks are faced with undersized internal storage and want to use cloud services to offload and that’s fine. But for those of us with ample storage just looking for a tool to keep our various devices in sync, it would be nice to not have to deal with these online/offline shenanigans and all the heartache that comes with when stuff expected to exist locally actually doesn’t.
OneDrive does have this. As does iCloud Drive.
(See the button at the bottom.)
(Turn off Optimize Mac Storage to download all iCloud Drive files.)
For Dropbox, you can download files automatically. For a file or folder, select “Make available offline” from the contextual menu.
For iCloud Drive, I have “Optimize Mac Storage” unchecked. Sometimes there is a delay, but everything gets downloaded automatically.
… In the still silence, resonant, derisive, cackling laughter can be distantly discerned. It sounds eerily like Apple’s former CEO avenging the rise in popularity of Dropbox from beyond the grave, for a daring slight of technical engineering, committed a lifetime ago …
I now use iCloud exclusively for the small stuff that needs to be everywhere, basically because of the always-download feature. However, if you are happy to get your hands dirty, I can highly recommend the very excellent rclone for syncing from anywhere to anywhere and accessing pretty much anything with a storage API (including serving local WebDAV that can be browsed directly in macOS), and Syncthing for syncing point-to-point between computers whether on-net or off (using a bag of tricks including NAT hole punching and relays). These are command line tools, but once running, infinitely configurable, so you can decide exactly what to keep and where. They are not going to be as fast as native clients for cloud providers, no getting around that. But, your options are much wider: you can deploy syncing to a NAS or network volume, and you can use S3 or Wasabi (very cheap) as easily as Dropbox. It might be worth thinking about.
I’m a big fan of Syncthing. It’s what I use to sync my desktop and download and local documents folders between all of my Macs. Sure, it has a command line interface if you want, but you can do everything easily from a local webpage on each client. It does take a while to reconnect on Intel clients sometimes, but on Apple Silicon it’s up and running in no time.
I was a bit confused, wasn’t sure if we were talking about the same thing. Then I realised that just because I used MacPorts and hadn’t visited the SyncThing site in ages didn’t mean there wasn’t now a non-command-line way. And, sure enough, there’s now a really nice native Mac app. That’s awesome. Now I can get more people using this, no more excuses.
Two points.
However after four days “Get Info” shows only 11GB of it is on line only and 57GB is on disk. It has been stuck at this for three of the four days. I repeated the exercise and it stopped at the same point 57.21GB on disk:
Can anyone else who thinks they have on-line folders in iCloud Drive, please do “Get info” to see how much is still on disk?
A test on a much smaller folder worked correctly. Get Info showed it was iCloud only.
Incidentally Making the same 68GB test folder to be on-line only in Dropbox worked correctly within a few hours.
Thanks
I feel your pain. I mentioned in another thread that this is why I ditched iCloud Drive for mass storage in favor of Dropbox. iCloud can get stuck for days. Worse, it can make your only copy of the file inaccessible even for reading when it’s in this stuck state.
Now, if the File Provider API is the cause of this stuckness, we’re all screwed. Apple is replacing what works with what doesn’t work. And when it comes to reliable protection of and access to your files, nothing in computing is more important.
Google Drive’s switch from FUSE based functionality to Apple File Providor has completely destroyed our shared video workflow costing us hundreds of thousands of dollars in lost time and attempts to find new solutions. Our entire company is run off of Google Drive including hundreds of TB of video footage. Suddenly all of our video timelines are broken, design files are broken, DVD & Blu-ray projects are failing to load due to paths changing. Basically links everywhere are messed up. In the past we could all use the same paths, now every path is unique to the user’s computer due to being buried inside everyone’s unique ~/Library folder.
And to make matters worse, even when I do fix all the paths, opening a video project in Davinci Resolve results in sometimes HOURS of waiting while all the related files cache locally, assuming their boot drive doesn’t fill up in the mean time and completely crash the system.
With the FUSE base system Google Drive could just load the frames of exactly where the playhead was sitting rather than forcing a full download of each and every video and audio layer touched by the playhead. Utterly unusable now.
I’m diehard Apple fan going back to the 1970s, my License Plate on my car was MCNTOSH for over a decade including the dark days of the late '90s, but I swear the software division of Apple isn’t even the same company anymore. Their priorities are all messed up. They don’t give a &^# about true video professionals who actually push their amazing hardware in the way the hardware designers intended.
Offline-only is easy: just move everything into your Documents folder and cancel your account with whatever cloud sync provider you were using
You can easily restore the old behaviour and get
Save Asback in all File menus (in fact, it’s so seamless I forgot that Apple had changed this, as once you make the change it sticks through upgrades, etc.):Saving Cloud Files to External Drives
What if you change the location directory of
$HOME/Library/CloudStorage/xxxxxwherexxxxxis a folder to a symlink to an external drive?And if it all works…
And if it doesn’t work…
It’s been a while since I did Unix shell scripting magic, so there might be syntax errors. But it seems to me this might work. Directory symlinks are normally treated as directories they’re pointing to in Unix except if the process verifies it’s a symlink and not a file.
cpcommand can detect symlinks depending upon the option and so canfind.I did this a few years ago to change around my Desktop Folder. My Desktop Folder was a symlink to point to various Desktop folders I could switch around. The idea is when I was working on a particular project, I could run a shell script to switch my desktop to be the files in that project. I had some app that allowed me to create a desktop menu of shell scripts, and used that to change around the Desktop folder.
Unfortunately, MacOS found the setup too confusing whenever I switched the desktop folder, and I ended up giving it up.
Well, that’s the question.
The theory is good, but someone needs to test it for real. Since Dropbox has been iffy with symlinks in the past, that’s not something for the faint of heart.
Another View of "Apple … forces … storage changes. " This is not a new problem for user who depend on external drives to hold media files. Apple has done this to several apps they supply. Here is my take on use of an external volume to provide enough storage for media in ~/Library. It could also be used to accommodate a large Dropbox folder within ~/Library.
iTunes managed multiple media types from a single interface with excellent metadata management and location independence for the iTunes folder. (soft links). When Apple dismembered iTunes, it moved Audiobooks, epubs, and pdfs to ~/Library/… This meant, as my library grew, I ran out of space on my DATA volume. My solution was to move my Home directory to an external volume on an external drive. This requires only one edit to System Settings(Preferences) to change the location of the Home directory. This must be done after the old Home directory has been cloned to the new location. I used Carbon Copy Cloner because it preserves essentially everything, including UID, permissions, and softlinks to external Movies, Music, and Pictures folders.
Caveats below do assume modest competence in Terminal and macOS user management. If you lack this, find some who has the competence or solve the space problem by throwing more money at Apple.
Caveats include:
After some time, Delete old Home directory (rm -rf /Users/oldhomediretory)
Regarding updates while main user account is on external volume:
In my experience, major upgrades are almost guaranteed to mess up if this sequence is not followed. Application updates have been no problem.
You need a slash instead of a space between CloudStorage and Dropbox.old, or else you’ll wipe out the whole CloudStorage folder.
@ace ,
What’s your opinion (this is also open for comments from the readers) about a professional design agency storing and actively working on Photoshop, Illustrator, and InDesign files exclusively on Google Drive? Yes – each designer would, ahead of time, first choose to make their file offline / downloaded locally, then remember to change the file’s status to online-only upon the completion of the project.
I’m interested in feedback about the pitfalls and possible troubles of depending on such an approach. Does anyone know of a design firm that has chosen to operate in this fashion?
Also, I’ve heard that the app entitled Arq will allow for syncing files & folders out on Google Drive with an in-house file server running macOS Server software. Anyone have experience with Arq and Google Drive?
I can’t comment on the File Provider API implementation since it hasn’t rolled out to me yet. But traditional Dropbox (which I’m running) certainly makes that quite easy. Just select “Make available Offline”:
at which point it downloads, you get a solid green circle badge, and it stays offline until you change your mind.
Is this functionality gone under File Provider API?
Yes, that’s the same in the new world order, as I note in the article. But there’s no setting in Dropbox to say that everything should be offline, all the time. You’ll have to make things offline manually. It should stick, but I’m not sure that’s guaranteed.
I’m confused. Here below you can see that you can go all the way up to the top folder and say Make Available Offline (all the time), and you are saying it’s the same in the new world order, but you’re also saying there’s no such setting:
So are you saying that is a recursive command that applied once will then affect any ~/Dropbox/* contents from here on out?
It’s absolutely recursive. You can open the subfolders and their subfolders and see that they all have the green circle badge. And when you “force offline” like this, I don’t think it will ever remove them locally without your permission.
Yes, but when you add new files from another source (not that Mac) will those files automatically be on disk or only in-cloud? I suspect the latter, so you keep having to manually tell new files to download.
What bothers me about this kind of thing is I use cloud-syncing like Dropbox and One Drive as a way to move files around between devices. It’s frustrating when I go to the other device and the files aren’t actually there and I have to do extra steps to force them to download and then wait for that download to happen. I want it to be automatic like in the old days.
I do some photo editing for a community newspaper. Photos that they want me to look at are placed in a folder shared via Dropbox. I set the folder to “Make available offline”. Yes, anything they put in that folder is automatically downloaded to my computer.
One result of all this has changed my own workflow significantly. It used to be that if you changed a file on a Google drive (via Finder overwrite or application save), the new contents became a new version of the old Google drive file. Crucially, the link to the file on Google Drive didn’t change.
Now, such overwrites are new files with new Drive links, meaning any references to the old file (from other documents or web pages or emails or …) give the old contents. Not pleasant.
The workaround is (queue the irony here) to update files only by dropping them into Drive in a web browser.
It took me a while to figure out that this was going on. Hope I save someone else some time.
Thank you for this article. I ran into problems with Dropbox due to a crash of my iMac (Catalina) & so an upgrade to a Mac mini (Ventura) via my Time Machine back-up. Saved this article for future reference
Interesting, but it didn’t work for me in Ventura. Duplicate remains stubbornly attached to shift- command-s
Note that ‘Save As…’ uses an ellipsis (option-;) not 3 periods.
Excellent thanks. Also found there’s some inconsistency between apps. Some use Save As and some Save as.
Ohhh, new files
. Sorry, now I get it.
I was about to reply that it definitely doesn’t apply to new files. And a test shows that moving an online-only DROPBOX file into an offline Dropbox folder doesn’t make it download and become offline.
But strangely, moving a NON-DROPBOX (and therefore offline) file into this same folder doesn’t appear to change anything. The file DOES sync to my other devices, but it doesn’t get a Dropbox badge of any kind. (It has a non-zero file size and Quick View works).
To round out the tests, moving the same NON-DROPBOX (offline) file into an otherwise online-only Dropbox folder again does nothing to the file: no badge, and it stays offline.
So, I’m not sure exactly how to answer the question :-) but I can confidently say it’s not the behavior you’re looking for. Dropbox doesn’t appear to maintain any directory-level rules that it enforces; it simply applies offline/online preferences upon request. Think of it as a lazy parent
At the risk of being talkative (too late), let me reply to:
Yes, but as we can see, neither is there any such setting in the “old Dropbox”. I think the reason I was confused is because this whole discussion (or so I thought) is about regressions created by the New World Order, and this would not be a regression, at least not for Dropbox.
So if this all behaves the same, then there’s “nothing to see here”, and the discussion should revert back to the pain of forcing all your files under
~/Library, right?Am I missing something? On my system (Monterey) Cmd+Option+G does nothing. It’s supposed to open a search in Google Drive. I can’t see any option to set this. Thanks for any help.
Shared folders do get deleted, yes. It happened to me, and it was a bit of a mess to recover from. But that’s only one of the ways that cloud files can be lost.
I had local Time Machine backups of my computer, and that’s the only way we recovered the files. They weren’t in any online deleted items folder.
The downsides of cloud files seem substantial to me:
These are things that I take for granted as basic things I need to be able to do with my files.
I guess I’ll keep buying large internal drives.
But I’m left wondering how people handle backing up cloud files?
I think you make a couple very good points.
My concern would be that backing up cloud-stored data is perhaps not taken seriously because “it’s in the cloud” and “that means it’s backed up, right?”.
Like you, I subscribe to large internal storage (I refuse to externalize my data storage needs). I rely on TM and clones for backup. Fortunately, I don’t have much critical work on cloud-based storage, but I do essentially rely on my existing backup strategy to secure that data too. If this were more critical data, I’d be giving that a closer look right about now. I’d also be curious to hear what others do, especially those that rely on cloud storage for really critical files.
Strangely enough…on my M1 MBP up to date Ventura and DropBox 169.4.5684…and I’ve already done the ‘update to latest version’ process as prompted by the red flag on the menubar icon…my DropBox folder is still located at ~/DropBox and not in CloudStorage…and under Sync in DB Prefs I can select all folders to be synced to my computer…and it shows the correct size for what I have in there so everything is syncing locally for me. And even more strangely…my Studio that has the same up to date Ventura and DB versions has a bunch of stuff shown not local and it is in CloudStorage.
I’m thinking that between Apple and DB they haven’t quite worked all the bugs out of this new arrangement.
I am on a local school board. We are in the process of installing “Hyperbackup” on a Synology NAS box. It will back up the files in the cloud, even Google Docs files (it stores those as Word or Excel or PowerPoint.) We haven’t gotten it running yet, so I can’t report any experience, but it purports to do the right thing.
I know that many people have found that Spotlight will not index files in the CloudStorage directory without forcing a rebuild of the Spotlight index. I have rebuilt the Spotlight index successfully and now have the following two different behaviors:
If I use spotlight to search for any “offline” file that is stored on my mac in the directory ~/Library/CloudStorage/DropBox/tmp/ , Spotlight search is SUCCESSFUL, finding the file with a full text search or a search for the name of the file. It also reports the file’s full path correctly.
But, if I navigate to that directory using the Finder and search for that file using the search box in that Finder window, that search FAILS, whether I search for the file name or the text contents of the file.
Any ideas why that would be or how to make the folder-specific search work?
This change broke my WXPO menu bar app for opening frequently used Office files. This stinks.
James, this explains that Finder search not working is a known limitation while Spotlight search will work. Expected changes with Dropbox for macOS - Dropbox Help
There is also a team update (business accounts with multiple users) rolling out and I’ve been documenting that as well as the ins and outs of this app update in a Twitter thread. https://twitter.com/ridogi/status/1585380467339104265
Dave K, there is a downgrade in ability to keep files offline in the new version vs the old. If you have 2 computers and you add a file it will be offline on the other computer. If a different Dropbox user adds a file in a shared folder then that file will be online only for you until you manually set it offline. If you are collaborating with others this is a downgrade. The “Storage of new files” option in Sync preferences is gone as you no longer have the option to exert that control over newly added files. This is the old view:
Thanks Eric! I think that description is out of date? I find that the description that “Searching through Finder will only find online-only files or folders that have been previously accessed on your device running Dropbox for macOS” does not correspond to the behavior I see: Finder cannot find files that are online-only or files or folders that have “been previously accessed…”
Still looking for an explanation, but this may hint at a difference between Finder search and Spotlight search. is there any documentation on what that is?
You’re welcome. Finder and Spotlight searches are just 2 interfaces to the same search engine as far as I know so the restriction doesn’t make much sense, but as you say it does seem like the difference between them does impact everyone and is not a problem specific to your computer.
My dropbox has been bugging me about that too. I dug into it enough to learn that it’s going to put my local copies in the hidden ~Library folder instead of in my Home folder where I prefer. I haven’t read Adam’s article yet, but I guess I better, in case there are other changes to.
I purposefully ordered my new MacBook Air with a 2 TB drive because where I live, the Internet is kind of sketchy and we have frequent outages. I like having online copies for access from multiple devices, but I really must have local mirroring, so I’m not dead in the water if there’s a power outage or an Internet outage.
I’m sure we will be forced to accept the update at some point in the coming months, but since there is no upside to doing it (it enables no new functionality) and there is a list of a dozen downsides you may as well wait as long as possible before doing it. You just have to put up with the red badge and a once a day reminder notification.
I tried the simlink hack for Google Drive and for the entire CloudStorage folder, and it didn’t work. It’s going to be rough going over here for a bit, managing almost 100TB of cloud storage and not being able to keep the index files externally. I hope Google offers a workaround soon. That beta touch hack was a lifesaver for a while but now it’s patched.
I don’t use Google Drive or Dropbox or iCloud or Box, which I suppose makes me perfectly Internet Qualified™ to comment on this topic, but the thought occurred to me that if one wished to dedicate an external drive for cloud storage purposes, one might be able to get away with mounting the drive itself (which I’ve called /dev/disk99 here) as the cloud storage directory, for example:
mount -t apfs /dev/disk99 ~/Library/CloudStorage/Don’t try this at home, folks, unless you’ve spent enough time with Unix filesystems to understand the implications. For instance, you would almost certainly want to unmount the volume from the /Volumes/ directory first. There are opportunities for data loss! But it is much more likely to work than symlinks.
I will try it if I get some time, but I’m on a product release push so it might be a month or more.
That seems like a reasonable thing to try. It’s hard to imagine in advance if there would be gotchas.
Whoa, that’s an interesting quirk. I ran into something similar in Dropbox some years ago. I haven’t retested recently to see if it has been fixed.
The hotkey is configurable if you click the Google Drive icon in the menu bar, click the gear icon in the window that appears, click Preferences in the menu that appears, click the gear icon in the next window that appears, and scroll down.
100 terabytes, seriously?
Thanks Adam. On my newish system it was there and clicked, but it was CMD + Option + Control + G !! One more key. So I’ll leave it that way, now I know where it is. And that was quite some drilling down to find it, thanks.
Having trouble finding context for this message. I’m interested because I’ve long complained that aliases created on the Mac within iCloud Drive don’t work on my iPhone, so interested in this topic. What article?
OK, I got a reply back from the folks at Sync regarding CloudStorage…for those of you not familiar with Sync it’s similar to what DropBox does…except it encompasses the two rules PIE (Pre Internet Encryption) and TNO (Trust No One) that Steve Gibson espouses and he switched from DropBox to Sync a year or two back because of those features. The drawback of these rules is that there is no syncing of just the changed portions of a document since it’s already fully encrypted on your devices…so a change means the whole thing gets synced again…if you have large files that get synced and slow connectivity that might be a consideration.
Sync offers a free tier which gives you 6 GB of space and that allows you to put the Sync folder anywhere you want…and all files are synced to all devices so there’s no online only option as DB has.
In addition…they offer paid tiers starting at 60 bucks a year for 200GB and 96 for 2 TB and then up from there. The paid tiers offer the option of having online only files to save space on your local drive…but that option requires that the Sync folder be located in CloudStorage. Paid tiers also offer the option to put the Sync folder anywhere…but if it’s not in CloudStorage and say on an external disk then the online option only is not available. However…paid accounts offer the option per computer to use CloudStorage or another local drive so one could use CloudStorage on your small disk laptop for instance and an external drive on your desktop at home so you could backup the whole Sync folder separately as desired.
That approach really seems like it’s a good idea to me…offering both options to the user…and I also like the PIE/TNO options it offers as well. I’ll likely stay with DropBox and all files both local and cloud until it becomes an issue for me which isn’t that likely IMO. However…if it does then Sync is a perfectly viable option to replace it.
I have not used the Dropbox app because I have no interest in synchronizing files. You cannot use Dropbox simply as an external drive, at least the last time I checked. Even if you designate a file as “online”, it gets downloaded to your computer when you open it.
I currently use Dropbox as an external drive by running CloudMounter and Mountain Duck.
These look like fascinating apps! Thanks for sharing.
Do you use both of them? If so, why isn’t just one of them sufficient?
Thanks again.
I find CloundMounter to be more reliable than Mountain Duck. I had problems MD when updating/deleting files. But that was a couple years ago. They may have fixed it since then.
CM is quicker but it only displays generic icons. I prefer to see thumbnails when I’m working with photographs. I use MD then.
Great article. Made an account just to ask this question: why are iCloud files no longer stored in the cloud storage folder? It would seem logical to put all cloud storage under the same umbrella, but iCloud files now get their own folder, which means they need to be special-cased in every script/automation.
It’s a good question, but I doubt anyone outside the iCloud team knows. ;-) My best guess would be that Apple needs to do special things with iCloud given how deeply it’s integrated into macOS, and some of those preclude being located in
~Library/CloudStorage.Can someone do a sanity check for me? I’m trying to use the Finder window’s search field to find a file that I know exists in Dropbox. (The same problem applies with Google Drive.) When I select Dropbox in the sidebar and then limit the search to Dropbox, it never completes. However, if I change the scope to This Mac, the file appears instantly. So it is indexed!
I’m seeing the same problem on my M1 MacBook Air, and both it and my iMac are running Ventura.
Do others see this behavior too? If so, it could be a problem with the File Provider architecture.
I don’t see that problem with either of my Google Drive folders. When I did the search, the process FinderHelper did ask for access to the files managed by Google Drive. Revoking that access didn’t make any difference with the search results.
I do see the problem, but not necessarily limited to cloud storage. Seems to me like a problem with finding in a particular Finder folder.
As shown below, the web interface to Google Drive has three categories of storage, namely:
But, Finder shows only:
How do I get “Shared with me” to be accessible from the Finder?
Huh! Curiouser and curiouser. I’m using Ventura—what versions of macOS are you two using?
Anyone else?
I’m running Ventura.
Adam, I found exactly the same behavior (see Apple’s File Provider Forces Mac Cloud Storage Changes - #46 by jamesajacobs). from following posts over in dropbox’s own discussion site, i’m hoping macos 13.3 will fix this?
iCloud is not involved with Dropbox. DropBox Storage is a cloud.
I’ve been acquainted with the structure of DropBox (DB) for many years. Let’s start with the high-level view.
There are four components of DB, the cloud storage database, a local work area in macOS, the macOS DB application, and the DB website.
The Dropbox.com website (the cloud site) does the usual things - sets up an account, prepares the cloud storage and Oops areas,
and downloads the DropBox application.
When the DropBox application is installed, it sets up the local work area storage (the Apple File Provider ~/Library/CloudStorage) and the Oops areas - a mirror of the DB cloud structure.
The work area storage contains a simple file structure of folders and files and cannot be found in Finder. The head of this structure is the DropBox icon inserted into the User folder which shows up in the Apple Finder. Clicking the icon opens DBFinder which looks exactly like Apple Finder. (Calling this another SSD drive would be incorrect.) Here the user copies/moves folders and files between the two Finder structures.
In to early DB days, the DB work area was in the System folder, and the mirror image was that folders and files were synced and identical. This meant that there were multiple copies of the same file - a minimum of 4 (cloud/work areas/two Oops areas). One more if the original file was from macOS, and 2 x the number of times a file was opened and closed during the thirty-day Oops area copies.
It was no wonder there were many unhappy users about how much macOS disk space was being gobbled up.
There’s more to write here but I wanted to get this out first.
E.g., The Oops is a 30-day retention area for Restoring previous versions of a file. i.e. an archival area. …
Before getting into syncing, I need to be clear about “online/offline”. It’s probably the worst pair of words one could choose. The DropBox the cloud storage database is always online except when selecting Sleep or Shut Down. The start-up file contains the file that starts up the DropBox app which immediately looks for work to do, i.e. connected and online. Always, unless the user suspends and restarts syncing manually.
The underlying method employed by DB is called “store and forward”. This method solves problems like this. You’ve finished your wellness exam and the doctor sets up authorizations for pharmacy, blood test, and an x-ray exam. The goal is the make sure that individual is registered in each department to avoid waiting to be registered at each department before they can arrive to get treatment in any order of arrival. And one of the department computers may be down.
The “store and forward” design consists of two forms - hub and spoke or husband wheel. “Hub and wheel” is where everyone is connected to everyone. This is a huge mess when you want to add another member of the wheel (and beyond this discussion).
The hub (the DropBox cloub) becomes the focus of hub and spoke design and is under very heavy activity. For example, if DB had 1,000,000 accounts, with 5 employees sharing all files, and its 5 PM Friday night in a single time zone, and 5 million users closed their application after submitting 1 update or new file, the download would be 5M syncs and 20M uploads to sync the other 4 devices on the account. A perhaps extreme example but mathematically correct. In this “hub and spoke” example, there is no “I” or “me”. It’s an environment of queuing requests. It is a fantasy to expect a half-second response time.
Syncing
Many conditions prevent syncing. As an example, The Fri sync request (the updated file is moved into a DBFinder folder) may be ignored if the hub cannot respond to start or finish a download request because the security protocol requires that a machine must be shut down before leaving the office. That sync will begin syncing Monday when the computer is started up. The file will be stale if looked at over the weekend.
Stay tuned. Mike
To be continued.
Mike
Didn’t find this option mentioned, but you can have Dropbox backup your Documents folder to their cloud. It won’t synch between other computers/phones, but the files are available offline and doesn’t screw up files as sometimes happens in the Dropbox folder. If you need to share between device or provide a link to someone, just transfer or copy the specific file(s) to the Dropbox folder. You may ask, why not put everything as downloaded/offline in Dropbox. My experience was the Dropbox was just not up to it for some reason. Just not consistent with what was happening there.
Hi Albert, Welcome to Tidbits-Talk.
It was unfortunate that you have an bad experience with DropBox. In my message above (though incomplete) I intentionally made up specific names of the four main areas to not be bogged down in Dropbox terminology. I mentioned this was the first version of DropBox as the first version.
(I’ve corrected/rewritten some text in this section)
Hi Mike,
oops, I wasn’t replying directly to you!
I just was putting out the option of using the Documents folder as it is an always-available offline folder that is synched to their cloud when online. As I mentioned, this only suffers from not synching between other devices automatically, as far as I can tell. You can access the Documents folder via a browser from other devices, though.
I use the “hidden” Dropbox folder for cross-device synchronization and just move files in and out as occasionally needed.
Thanks for the welcome!
Scott
installed macos 13.3 and spotlight is working again, including within dropbox folders!
With today’s Ventura update to 13.3 I noticed, on my M1 MacBook Pro, that “fileproviderd” is using an unfluctuating 110-125% CPU. I didn’t notice this with macOS 13.2 but after updating to 13.3, all file activity slowed down drastically (saving a 20 character BBEdit file took over half a minute while Activity Monitor reporting BBEdit “unresponsive” several times) and I started to look for a reason.
I see the same greedy “fileproviderd” phenomenon reported multiple times in both Dropbox and Apple support forums. Since the new Dropbox still describes itself as a “beta” it’s time to click the “leave beta” button!
Replying to myself before I confuse anyone else … with Dropbox disabled, “fileproviderd” still uses over 100% CPU. Back to the drawing board …
@mjtsai just posted about Google Drive implementing a 5-million-item limit. This is the first I’m hearing about it, but it sounds like Google has rolled back the change for now and promises to alert people if it reinstates a limit.
https://mjtsai.com/blog/2023/04/05/google-drive-adds-file-limit/
I was struck by one person’s complaint about not being able to know how many files were in Google Drive, but I think you can do this in Terminal if you’re using Drive for Desktop. Both Bard and Bing (via GPT, call it Bingbot?) claim this command works and it seems to for me:
find . -type f | wc -lYou have to navigate into the Google Drive folder first, which is easiest done with a
cdcommand followed by dragging the Google Drive folder into the Terminal window.I apparently only have 9375 files in Google Drive, so I’m not stressing about any limits.
I think the filesize calculation is cached and therefor shows the wrong size. You could check filesize with terminal or perhaps after a restart.
We have big problems when our Google Drive without notice changed all Macs to fileprovider a few months ago. About 50TB of data. The automigration on each computer was awful with old favorites still linking to data that wasn’t synced.
Our biggest problem now is however the new way Google Drive prioritizes sync jobs. It can only do one syncjob at the time and when an offline folder has added content by someone it will start download but the user can’t open files until Drive has downloaded all updated content of the less important folder. Opening a file should always be prioritized.
This is probably even lower than a beginner 101 iCloud and Dropbox question but…
I note in Adam’s screen shot of his Finder window that it shows Documents and Desktop.
I use Dropbox for online syncing of my Macs and iPhone and yes I guess it does provide another level of backup to my CCC backups to my external HDs.
So, in light of this change in Dropbox…
Is signing up for iCloud Drive and moving all of my Dropbox files, which is close to 100% of my files, from Dropbox to Documents a good option for online backup and syncing between devices?
Thank you.