Beware Mail Data Loss in Catalina

Using Mail with Gmail, IMAP always behaves a bit strangely; and Mail under Catalina does add a few more quirks. The main one I’m seeing is that a mail item moved from one mailbox to another copies rather than moving, leaving the original to be manually deleted. Annoying but not fatal— I’m not losing any mail bodies, I’m glad to say.

It’s worse than that, because you should be able to specifically target the Data volume to fix whatever problem led you to fsck in the first place, and the Mac-Enterprise list indicates that doesn’t work either. (And in theory, the read-only volume should have no such problems, so it’s there or something else entirely.)

I recommend Mail Steward for archiving emails before a major OS update. It stores emails in a MySQL database that cant be touched by the OS installation. I have emails going back over 15 years.
https://mailsteward.com/

No data loss here, but a different problem: out of application memory crashes when Catalina was upgrading its Mail from my Mojave installation. Running 16 GB RAM, that shouldn’t happen but with Catalina it did. Reason: I have nearly 20 years of old emails stored in mailboxes “On My Mac.” This is the first upgrade when I’ve run into the problem. I did find a work-around that I’d be glad to share if anyone wants to know, but I’m also looking for a solution that would keep some of those emails outside the Mail program without turning them into textfiles while having them ready to bring back into Mail in Catalina and future Mac OS as well.

PS: Mail Steward could be the longer-term solution I’m looking for, of course. I’ll check it out.

Wait… is this the opposite of Sherlocked when a OS upgrade makes a 3rd party app more necessary?

This app has been stewarded by Apple.

Thanks Apple.

Catalina has also introduced the issue where 3rd party programs (such as HoudahSpot or BusyContacts cannot search Apple Mail. (I have registered my view on https://www.apple.com/feedback/) For this reason I am researching 3rd party mail programs such as Mailmate or Postbox. Any opinions or comments on what 3rd party mail apps are worth looking at… Thanks

I swear by InfoClick. Use it multiple times every day.

I have to admit, I’ve never had a problem searching for mail and my email, stored in mail boxes on “on my mac” goes back decades.

What problems occur that require a 3rd party search engine for mail?

David

Al Varnell

    October 16

garethmcclay:
Any opinions or comments on what 3rd party mail apps are worth looking at…

I swear by InfoClick. Use it multiple times every day.

David, I agree, Mail search is very good. But I like HoudahSpot as it allows me to create a search then save it to the Finder (like a smart mailbox). I have lots of these, which I can then search for. I don’t think I can do this with Smart Mailboxes in Apple Mail (also when you have 100s of smart mailboxes it gets a bit messy!)

Thanks Al for InfoClick I’ll check this out as it appears to build its own index (so independent of Catalina’s restriction). I could not tell if it allows me to save a search to the Finder & then run the search by launching from the Finder (I’ll download the trial & test this…)

DEVONthink Pro will archive individual emails, entire mailboxes, or multiple mailboxes with a single menu command. They remain in EML format, as originally organized, with all metadata including MailTags. You can update the archive whenever you wish and DTP will not import duplicate UUIDs. Anything indexed in DTP is also indexed in Spotlight, unless you opt out. Of course, DTP will also file your other documents and perform all sorts of tricks. 15% discount for TIDbits members.

Infoclick won’t work with Catalina Mail, since as noted elsewhere here Spotlight searching access has been disabled by Apple. That’s a problem for other non-Apple programs like Houdah Spot as well. Really stupid; I don’t see the point of that “feature.”

Mail in Catalina has been awful for me. Tons of random re-downloads. And it lost and continues to lose all local “On My Mac” folders, even after I re-imported them.

So I’m looking for a replacement; tired of Apple’s buggy junk. Looking at old dependable Thunderbird right now.

I’m having no issues whatsoever with InfoClick in Catalina. I don’t believe it has ever used the Spotlight database, rather it builds it’s own which is far faster with multiple means of search. It doesn’t appear to have anything to do with Catalina Mail. It’s a separate app, not a Mail extension of any kind.

-Al-

I’ve seen several reports ‘round the net that Catalina Mail has some issues that sometimes delete mail in the Inbox. Googling tells me that it’s mostly for those that use gmail although not in so many words.

Is there any reliable info regarding the issue, what’s affected and the like or are we mostly waiting for Catalina.2 to be issued and hope that fixes it? I’ve held off so far on upgrading.

I do have a gmail account but it’s not primary…heck, it’s not even tertiary. My main mail accounts are my original account at his.com, iCloud, and at FastMail.

Thanks.

I’m waiting for Apple to mention something in release notes or Michael Tsai to say more.

Hello,

When I updated to Catalina, I started having problems with Apple Mail on my MacBook Pro.

I’m now retired from the University of Illinois, but still use their email service. They utilize an Exchange server, and I started getting error messages for some of my incoming emails. The dialog error message box was:

The message “[xxx]” could not be moved to the mailbox “Exchange”
The specified object was not found in the store., The process failed to get the correct properties.

The error needed to be cleared before any additional emails could be downloaded from the Exchange server, and sometimes there were quite a few of them. None of the messages showed up anywhere after clicking “OK”, either in Mail, or if I opened my email in Outlook on Safari. Fortunately, they were all messages I didn’t care about, and I think they all were targeted for deletion by my Mail filters. (I have at least a couple hundred filters.)

After a back and forth with IT at the University, they pointed me to a possible solution at this website:

https://appletoolbox.com/mail-not-working-in-macos-catalina-how-to-fix/

Yesterday I went through the process of removing and rebuilding the Library files related to Mail as per the first fix described on that website. Quite a slow process for me, probably because my local Apple Mail folders go back to 1997. After some reboots and repeated rebuilds of various mailboxes, things seem to now be working ok.

Too soon for me to know if things are permanently fixed, but so far, so good. I should note, however, that all my filters were turned off in the process (but they still were there) and I’m gradually reinstituting them as the myriad of spam emails now flood in. I’m also taking the opportunity to unsubscribe from most of the mail lists that I had previously just automatically deleted through my filters.

Interesting article, thanks! But the comments following it are extremely frightening!

On my MacBook running Catalina, an IMAP account in Mail has become very problematic. Messages no longer indicate they’ve been read after I open and read them. If I mark a message as Read, it switches back to Unread immediately. If I delete a message, it vanishes for an instant, then pops back. If I mark it as Junk, pretty much the same thing.

I’ve run utilities, deleted the account from the computer and recreated it. No change.

The account is through GoDaddy and accounts from other services don’t exhibit the problem. But when I run Mail on other computers with older OS’s, it works as it should. So I’m inclined to think the issue is in Mail and Catalina.

If I turn off wifi, all of those issues disappear but as soon as wifi is on again, deleted messages re-appear and so on. But is that a failure of Mail to properly interface with GoDaddy’s servers or is it an otherwise unreported issue with GoDaddy?

So Mail in Catalina works correctly as long as you’re using Ethernet? That’s… freaky. I can’t come up with any way of explaining that behavior.

Have you checked with GoDaddy support about this? If it were generally a problem with Mail and Catalina, I imagine they would have heard about it by now.

Sorry if I wasn’t clear about turning off the wifi. I’m off line altogether at that point; no ethernet, no nothing.

So it clearly relates to being able to talk to the mail server, but there’s a failure in that communication that seems to be exclusive to the computer running Catalina.

I have been checking with GoDaddy and can’t find anything and they claim there’s no issue on their end.

Can you get them to say if other people are having similar problems?

James, does you MacBook exhibit the same behavior on multiple (unrelated) wifi networks? Do other Macs on the same wifi network work correctly?

In Catalina, my Mail app displays most messages correctly, but some messages have content replaced by the question-mark-in-a-box character and others have parts of the content missing entirely. Some of these message errors occur in a corporate Gmail account that I read in Mail (and I can see those messages just fine at gmail.com), but other messages occur in an IMAP account (through DreamHost, and these appear fine in the webmail client).

Seriously frustrating. FWIW, I love InfoClick, and that’s displaying things funny, too.

Still not safe in 10.15.2…

That normally indicates a font issue. I would start by using Font Book to Validate all your fonts and eliminate any duplicates it finds.

Thanks! I actually tried that already, on someone else’s recommendation, but no luck.

I need to buy a new MacBook Pro (which will have Catalina installed) and migrate my data to it, including a large amount of POP email in On My Mac.

My current MBP is a 2017 model, running Mojave (10.14.6).

What is the easiest way of making sure I don’t lose my email?

If it’s on a backup clone, will I later be able to import it?

Thank you,

David

It’s likely to be OK, but if you wanted to be extra careful, you could archive it using EagleFiler or MailArchiver X or MailSteward or such an app. That way no matter what Mail did to it, you’d be fine.

In your case, however, since it’s all POP, it should just be .mbox files on your backup, which means that bringing it back in again shouldn’t be terrible.

It’s likely to be OK, but if you wanted to be extra careful, you could archive it using EagleFiler or MailArchiver X or MailSteward or such an app. That way no matter what Mail did to it, you’d be fine.

I see now that 10.15.3 supposedly avoids the problem, but I couldn’t wait any longer and finally migrated to the new machine (running 10.15.2) last week.

As it turned out, I did lose massive numbers of messages, but luckily I had taken your advice and archived everything in EagleFiler beforehand, which I highly recommend.

(The advantage over importing from a backup is that message status is preserved, plus of course you offload inactive messages from Mail.)

In many ways this was a blessing in disguise, since I had been wanting to create an archive for a long time and had just been putting it off.

I’m sorry to hear that you lost email, of course, but glad my advice was spot on. And EagleFiler is a good solution there. And doubly appropriate that it worked for you given that the author, @mjtsai, has been the primary conduit for warnings about Mail in Catalina.

FWIW, this morning I saw what appeared to this problem in Mail Version 12.4 (3445.104.14) on macOS 10.14.6 (18G5033) Mojave.

In Trash, I have an On My Mac folder. In that folder, I have messages from Lending Tree. (Aside: I never even visited Lending Tree’s web site and I don’t know where it got my email address, but I get a message or two each month. I move said messages to this folder.) There are three messages from entities other than Lending Tree in this On My Mac folder. This morning, I received a Lending Tree message, and for no reason in particular, I opened an earlier message. The body was missing.

The three other messages, all older than the oldest Lending Tree message, still have information in the body. All Lending Tree messages from January (that’s when they started arriving) have empty bodies. All Lending Tree messages from February and later still have the body contents.

For one message, I chose View > Message > All Headers; all header information disappeared. I then chose View > Message > Raw Source; the new window was empty. I was able to copy a message that still displayed body contents back to a folder on the original server. Attempts to copy a message that had the body missing back to the original server failed.

In this case, I don’t really care, but it will make me more cautious about using On My Mac, even on Mojave.

If you scroll to the bottom of nearly any commercial email message, there will be an Unsubscribe link that you can click to remove yourself from their list. It’s absolutely worth doing—don’t put up with unwanted email, however you started getting it.

My only question is if the Lending Tree message could have been loading most or all of its content remotely, such that after a certain time, they remove the source material from wherever it’s being loaded from.

That’s good advice if the unsubscribe is legit.

Problem is, I’ve read that some spammers use such unsubscribe links to verify that their spam has been read by an actual human instead of just ending up filtered out to /dev/null. That in turn makes the email address of somebody who actually hit that link much more valuable and spam volume increases accordingly.

It would be nice to know if there’s any hints as to which unsubscribe links are legit and which essentially mean you’re signing up for even more spam. I’d like to think if it’s from a reputable company the link is likely legit, but then again, I’d think reputable companies don’t start spamming people in the first place.

I simply decide based on whether the email actually originates from the a legit organization indicated on the “From:” line using SpamCop. I also hover over the “unsubscribe” link to see if it matches that organization. One problem with that is I occasionally see that it’s from a third party the organization used for bulk mail purposes. I can usually recognize those, as well.

I’ve forgotten who it was that recommended a few years back that they started using those unsubscribe links again with great success, but I’ve certainly found the same to be true.

I assume Lending Tree is a reputable outfit and would honor an unsubscribe request, but I remember being cautioned never to click on unsubscribe because it just confirms that the target address is valid. (I see @Simon made this point, and @alvarnell addressed it.) However, the primary reason I haven’t unsubscribed is because I wonder who Lending Tree thinks is the recipient and why, and maybe I’ll get a clue. And it’s only 1.5 messages per month.

One strike against Lending Tree is that the first message it sent asked me to confirm my email address; I did not reply in any fashion, yet I continue to receive messages.

I have Mail > Preferences > Viewing > Load remote content in messages unchecked, so I believe that should not be a factor. Also, the body is completely blank, so even non-remote content (which did exist earlier) is failing to be displayed. (I assume my preference regarding remote content is being honored because I often have gaps in other commercial email that I receive.)

If this isn’t a scam, then that mail is probably the result of someone providing your e-mail address and Lending Tree is asking for confirmation. But normally, such confirmation messages mean the address will later be dropped if there is no reply - which doesn’t seem to have happened here.

This reminds me a bit of the same thing from Netflix. I got several e-mails from them claiming that I subscribed and need to go create a password and register payment. We have a subscription, but it’s using my wife’s e-mail address, not mine.

I called Netflix customer service. They confirmed that the mail is real and that “someone” must have signed up using my e-mail address, and that I should just ignore it because the account doesn’t have any payment mechanism on file. I demanded that they delete the account (Netflix doesn’t actually give you a way to do that - you can only stop the service and payment) and they did.

A month later, the cycle repeated.

Should I start treating Netflix as a criminal scam organization? It’s really tempting and I would except for the fact that the rest of my family really likes the service.

First off, that worry about somehow “confirming” your email address to spammers hasn’t been true for many years. Spam is about sending as much as possible, not careful targeting. Billions of email addresses have been revealed in security breaches, so it’s trivial for spammers to get your email address along with a few billion of your closest carbon-based life forms. No one is special when it comes to spam. Use an email service with good spam filtering or rely on a tool like SpamSieve and move on. Thinking about spam is a waste of time.

Second, I would hope that everyone here would be able to tell the difference between legitimate (if unwanted) commercial email and spam. If you’ve ever heard of the company, it’s legit. If you can find the company’s Web site with a quick Google search, it’s legit. Just look at your actual spam folder and scan through some of the messages. They’re obviously spam. Don’t click any links in spam.

And yes, I know there are some phishing messages that can fool people. But 99% of the commercial mail you see in your Inbox is from companies you’ve ordered from once, filled out a form with, or otherwise been connected to in some way. Just unsubscribe and move on. If you’re asked to provide a password before unsubscribing, it’s phishing and you shouldn’t do it.

Finally, speaking as someone who sends hundreds of thousands of messages per month, it’s really annoying (and damaging) when people mark legitimate mail as spam just because they don’t want to get it anymore. That’s what the %^^$&* Unsubscribe links are for.

Since you’re using iCloud, the next question is what happens when you look at those messages on iCloud.com?

Um, but the messages are in the On My Mac mailbox, so they wouldn’t appear in iCloud.com. (And I know, because I logged into iCloud.com without thinking about it, and they weren’t there.) So, while I’m using iCloud, not in this case.

On replying to unwanted commercial email:

Confirm that the message is from the actual organization by checking the domain of the sender (and any Reply to header). In almost all cases, if it’s not the actual domain of the the organization, it’s probably a scam. Also, most good email programs let you preview the URL for any link in the message. Again, be sure that the URL is not suspicious before you click on it.

One exception: Many political campaigns use a service associated with their political party to deal with donations (for example, the Democrats in the USA use actblue.com). The sites involved with such a campaign may have links to the service rather than the campaign.

I get that. If it’s something I signed up for I won’t report it as spam.

But let me show you the other side of the coin. When I publish an article my email address gets listed as a footnote. Companies that sell equipment for use in lab environments like one I work in harvest email addresses from these journals (even non-open source journals usually list the authors, affiliations, and email in the free preview) and then send me marketing email about their latest spectrometer or whatever. Is that spam? It sure is in my book. I never asked for their email, I left my address obviously for interested readers of that journal, not for marketing. I don’t care if their marketing comes with an unsubscribe link. I will add them to my blacklist and I will report their email as spam. How a business thinks they can get new customers by sending them obviously unwanted ads is to me baffling. The fact that they keep on doing it, to me indicates though that at least at some point they have been successful. Puzzling to me.

Maybe, but it may also be a legitimate third-party mailing-list company. For example, I get a lot of stuff from Mailchimp domains, most of which have no indication that they are Mailchimp unless you perform a whois lookup on them.

Yes, if you have no relationship with the company in question, it’s totally fine to mark their email as spam and they can suffer the collateral damage. But particularly if their email will regularly get through your spam filter in the future, it’s also worth trying to unsubscribe.

The main thing is that marking something as spam is not the way to unsubscribe from something you signed up for and no longer want. That’s just obnoxious.

I am suddenly finding that messages are going missing in Mojave. I’ve delayed upgrading to Catalina because of reported Mail problems, and now I’m seeing them anyway. I’ve kept all email messages of any substance for years now and suddenly my history has holes in it. I’ve switched from Verizon to Gmail, but I think the missing messages go back to before the switch. Do I need to abandon Mail and find another app? If so, what do you suggest?

If you put your saved messages in folders on your mac, you can back up using time machine.
I found whole folders empty and had to back up some - but it also got me to do some housecleaning.

David

Have the email problems been fixed in Big Sur? I just bought a slightly used laptop for my wife and am debating upgrading her from High Sierra to Catalina or Big Sur. She has email coming in from Outlook, Gmail, POP and IMAP.

–Michael

@david_tuma and others: I’m embarrassed to say that as a long term Mac user I’m not sure how to look for my email folders that hold all the many, many, many e-mails I still have - I too keep important mails for a lifetime and it is amazing how often an old email comes back as important.

An example: I once performed a marriage for a couple and over 15 years later they needed a confirmation that I had sent them an item from the wedding! I was able to recover the email and the attachment title so I could search through tons of old saved PDF documents and locate the original! They were very happy and so was I!

So what is the best way to archive? I tried Mail Archiver X but it took forever, the file was massive and took up my hard drive, and it kept breaking. Suggestions most welcome! Thanks.

I have been very satisfied with EagleFiler – not only robust but great features and excellent support.

@davbro Thanks for the suggestion. I have used the Tidbits discount and the Black Friday discount and purchased it. Hope it works well because i was very disappointed in my results with Mail Archiver - not that the program was not good, just I think all my e-mails were a crusher - my bad. Thanks again.

I’m still curious if there is a way I can just copy all the e-mails to a separate folder that I can keep on some external hard drives although that sort of defeats the incredible value of another program, InfoClick, which has been absolutely invaluable to me and rock solid as a program. Doesn’t surprise me that InfoClick is made by the same folk who make Nisus Writer Pro, my constant companion lol.

Thanks again, David.