When Are Summaries Valuable?
Apple Intelligence has introduced uncertainty into formerly verbatim news article notifications, sometimes producing blatantly erroneous summaries. The company’s response to a formal complaint from the BBC and widespread negative media coverage? It will update the feature to perform better. Jason Snell of Six Colors thinks that’s insufficient. As it stands, apps can’t opt out of having their notifications summarized by Apple Intelligence; Jason is calling on Apple to allow individual apps or similar classes of apps to opt out of notification summarizations. I’m with him on this topic—it’s problematic for Apple to put words in the mouths of others. The Verge’s collection of notification summarization mistakes is reminiscent of auto-correct fails, but at least with those, the user can revert to their original text. With news notifications, Apple Intelligence summarizes a collection of unrelated content, often providing actively unhelpful results.
These AI summarization mishaps prompted me to think about summaries in general. I’ll admit to a knee-jerk negative reaction whenever I have been offered an option to summarize, whether AI-generated or not. As a fast reader, I was never intimidated by long books in school, and I picked up on my teachers’ disdain for CliffsNotes summaries of classic works of literature.
Upon reflection, though, my reaction is unfair. While summarization certainly has its problems, dismissing it overlooks something fundamental: summarization isn’t just an overhyped AI feature—it’s core to the human experience.
Think of summarization as a form of lossy compression, similar to how digital photos are compressed to save space. Both attempt to reduce the amount of data required from the original to convey its meaning. Damage is always done in the process—a JPEG-compressed image loses fine details from the original, and text summaries lose detail and nuance. Romeo and Juliet is more than a tragedy about two young lovers whose devotion to each other defies their families’ bitter feud and ultimately ends in their untimely deaths. Thanks, ChatGPT, for getting it right.
We accept the loss of detail because one or more constraints often make summaries more practical or useful for specific purposes. The most common constraint is time—you can read that one-sentence summary of Romeo and Juliet in a few seconds, but watching the play or reading the text would take several hours.
Another constraint is background. Without a solid grounding in physics, you may not get much from reading “On the Electrodynamics of Moving Bodies,” the paper in which Albert Einstein introduced his special theory of relativity. Those of us who lack that background or a desire to achieve such a state—life is finite, and we can only learn so much—are better off knowing that the paper demonstrates that the laws of physics are the same for all non-accelerating observers and establishes the relationship between space and time, fundamentally altering concepts of simultaneity and motion. I hope that’s a reasonable one-sentence summary.
Physical display space is a third type of constraint. When you look at the list of messages in Mail, that’s a form of summary—reading your email as a single scrolling document would be insane. One of Apple Intelligence’s features enhances the message list to replace snippets from the first few lines of an email message with a summary. These summaries will be more descriptive than the snippet preview, as the preview is just the first part of a message instead of its meat. However, they can suffer from the same sort of errors as the news notifications.
The value of a summary is, within limits, proportional to the difference in length between the source and the summary. The more compressed the summary, the better—again, within limits.
Those limits vary by situation—I needed a single sentence for the examples above, but such short summaries lose so much of the originals that they aren’t otherwise all that useful. Asking ChatGPT for longer summaries provides significantly richer results. In other words, there’s always a sweet spot between how tightly the summary compresses the original and how much of the original’s information is retained.
That value explains my discomfort with Apple Intelligence’s summarization options. Because I read quickly, I see no reason to ask Apple Intelligence to generate a summary of a Web page or a conversation in Mail. The downside of losing detail and nuance—and of possible errors—outweighs the upside of saving a few minutes of reading time. Notification summaries are even worse; for me, they save seconds at most and often introduce confusion by summarizing unrelated news articles or information that has changed multiple times within the summary period. The main utility I see for notification summaries is to reduce the irritation of too many notifications from chatty conversations or overactive apps, but Apple has already addressed that by grouping notifications.
While AI-generated summaries raise valid concerns, it’s essential to recognize that human-created summaries permeate nearly everything we read. For instance, every email message and discussion forum post has a subject line that’s supposed to summarize the message’s intent. People often write poor subject lines, but they remain an essential form of summary—one that AI could actually help improve.
That’s just the start. Nearly every article or non-fiction book has a title that is, most of the time, the shortest possible summary the author or editor can think of that is both attractive to a potential reader and accurate to its contents. Many articles, including ours, have short summaries that serve as teasers in a list. All academic papers have built-in summaries in the form of abstracts—I rely heavily on those when researching topics outside my sphere of expertise.
The need to summarize goes even deeper. Most news articles are themselves summaries of the events they cover. Wikipedia may contain 6.9 million articles, but the average length of an article is a mere 690 words—it’s a collection of summaries. While few people would consider a book to be a summary, most non-fiction titles are distillations of the author’s more extensive research.
I would even argue that human language is itself a form of summary. There’s a reason we say that we “choose our words”—we’re summarizing the rich, complex, and chaotic thoughts and feelings in our minds into a limited but hopefully understandable collection of words. Just as summaries lose nuance and detail, language often falls short of conveying precisely what we’re thinking. Without full-bandwidth telepathy, it’s the best we have for sharing ideas. Summaries are intrinsic to human expression.
To summarize—I had to!—summaries offer a different value proposition for everyone. Reading speed, language fluency, topical understanding, display space, and other factors play into how valuable a summary of a particular length will be in any given situation. You should ask for AI-generated summaries only when they will provide actual value and you can verify their accuracy when it matters. Finally, remember that just because something can be summarized doesn’t mean it should be.
I love the summaries, even if there is something blatantly erroneous, for the simple fact that I can delete all of the, say, NY Times notifications after simply reading the summary rather than having to expand the notification to see what each one was. The same goes for Nest camera notifications, sports event notifications that I get, etc.
I didn’t read Jason Snell’s article (I hope that you summarized his suggestion well
) but - Apple Intelligence remains in what Apple calls beta, so I’m fine giving any errors like that a pass for now as long as Apple commits to making it better. And in the case of these erroneous summaries - to me they are a really small matter. And, summaries are under user control - they can be turned off on an app-by-app basis, or even altogether, if users don’t like them.
In fact, I think I’m against the notion of a developer having control over this on my behalf. If I want BBC notifications summarized, I should have control over that - not them. If this ever happened and a developer had this sort of control, I might decided that the app just isn’t worth keeping, or I’d turn off notifications altogether. I think summaries of notifications are a good thing.
However, perhaps a compromise is allowing apps a one time first-use dialog notification allowing users to disable them for that app.
Needs work, in my opinion.
I have recently turned it off after using it since the feature debuted. I found it suffered from the same hallucinations as the news summary gaffes referred to in the article. For example, I was falsely told that a friend had died (from an email where that friend’s name was mentioned, and the topic of death appeared), that a meeting was requested on the wrong date (from an email where several alternate dates were discussed), and that someone was not able to fulfill a request I made (from an email where that person said they’d be happy to, but not on the day I suggested).
To be fair, the error incidence was probably around 1-2%, but the remainder rarely told me anything useful, or told me less than the two or three line preview did. Which is now what I’m back to using.
A key point is that summarizing a collection of notifications provides a summary of a collection of summaries. The problem is that the AI doing the final summary works only with the summary of each article in the notification. If AI accessed each article in the collection being summarized, a more usable summary would be produced. It’s like doing a 2nd lossy compression on a set that has already been compressed.
The question of what “beta” means in this context has come up quite a lot, such as from Nick Heer and John Gruber. I tend to think the term is a meaningless weasel word when applied to a heavily marketed online service that has no announced final date. I have no issue with betas of macOS and iOS, for instance, because they iterate through releases and exit beta in a reasonable time frame. At that point, Apple is basically saying that the features they contain are fully complete. What beta version of Apple Intelligence are we at now? When will it exit beta? How will we know? Without answers to those questions, Apple Intelligence is just another online service that gets constant improvement behind the scenes.
Ouch. I haven’t used it that much, so I haven’t run across any of these errors. But as you say, they could be seriously problematic if they display the wrong date for a meeting or some other fact where the user would think that the summary encapsulates all the necessary information. (At least when it says your friend died, you’d read that message and discover the mistake immediately.)
I’ve recast the article slightly to tone down the description and mention the possibility of error.
My $.02: Summaries have significant value when they’re context sensitive. By that, I mean that the person doing the summary has knowledge of what the target audience knows and is interested in. AI won’t have that knowledge!
(My former boss told me last month, "I was in awe at your ability to summarize a long technical paper with the relevant information I needed. It was like having my own ‘ChatGPT’ before that was a thing.
)
I think another factor that makes email summaries even more difficult is that with a news story you start with content that in theory should be professionally written, with proper subject-verb-object construction, and a straight line narrative. Casual emails between friends? Not so much.
While I haven’t turned them off yet, I’m with Jeff in that they are problematic.
For instance, one (older) person I text with has a tendency to send long texts with multiple subjects. AppleIntel will give me a summary that confuses the topics and often produces strange results.
Like my mom had been ill and I was telling this person she was feeling better, and when this person responded about my mom’s health, AppleIntel said something along the lines of “Mom not well” – which was exactly the opposite of what we were discussing (my mom was getting over her cold and doing better).
The other problem I’m having with these summaries is that they are often so abbreviated they are useless, not really telling me what I need to know. I have to read the full text anyway, so why bother with the summary? I have noticed that both in CarPlay and when I’m using my AirPods, Siri will read me the summary version first. This is frustrating because then I have to tell it to read me the full text. I would have saved time just listening to the “long” text in the first place.
(It’s even worse in CarPlay, but because there it forces me to listen to the summary twice. It will interrupt my audiobook with a summary of the incoming text, but to read the full text I have to switch to Messages, tap “read,” and then listen to the summary again. Only then can I tell it to read me the full text!)
That said, I have found some summaries really nice. Things like notifications from your dentist about an appointment, emailed receipts, UPS delivery notices, etc. are usually summarized succinctly and avoid all the surrounding marketing crap.
I also like it when I’m working and I get an email from someone important – that would usually make me stop and read the message – but the summary says something routine like confirming a future meeting or approving something I asked about and I know it’s not important enough to bother disrupting my rhythm and I can continue working.
I don’t do news alerts or notifications from very many apps, so I could care less about summarizing those.
Adam, I don’t know if you’ve used the AI summaries in the Strava app or not. They’ve gotten a little better recently by being more specific, but they’re still mostly just telling you what you can immediately see for yourself in the data presented. It’s not just Apple. I don’t find Apple’s summaries to be that useful either.
Amazon AI summaries of product reviews are extremely useful.
No, I haven’t—this is the Athlete Intelligence system? I only very recently resubscribed to Strava Premium since I hadn’t been using Strava nearly as much while rehabbing from injury.
What don’t you like about it?
Whenever this kind of topic comes up I like to remind us of the quote from newspaperman Sid Harris:
“Any philosophy that can be put in a nutshell belongs there.”
I also like the early Simpson’s joke blurb about the Reader’s Digest version of Shakespeare:
“Brevity is … wit.”
While I’m at it, there was a MS PowerPoint tutorial which said you should have no more than 6 bullet points on a slide – if you have more, you should consider presenting about something else.
If you’re giving a presentation, ideally, the bulk of your content should be what you say, with the slides only being quick hints.
But that’s if you’re presenting to an audience.
Very often, PowerPoint is used in lieu of a word processor document, and is e-mailed around without anyone ever actually presenting it. In which case, all the rules change, because the file needs to contain all your information and the audience is expected to read it, not sit and listen to you present it.
Where do we see these summaries? I have a MPB M1 running the latest Sequoia. Should I be seeing summaries somewhere?
When I first started using it, I started calling it “artificial banality”, because it was just telling me things I already knew or could easily see from the graphs and bar plots the app shows. They’ve improved it a little, but it’s still pretty useless. For example, today I rode stage 2 of the Tour de Zwift, a route called “Snowman”, and the first AI box tells me “Solid group ride with a personal best on multiple segments, pushing your distance beyond recent averages and maintaining consistent power output.” That’s all true, but I was there and I already knew that. It also tells me things about power, heart rate, and (virtual) altitude gain, all of which can easily been seen from the provided graphs/bar charts. I doubt that it will ever be actually useful.
In Sequoia I see them most often in Mail in the list of mail. In iOS I also see them in notifications. I don’t remember if they have to be turned on or not; I believe you can turn them off.
Summaries can be turned off: Settings> Notifications> Summarize Notifications> all can be turned off at once, or by specific app.
Now that you mention it, I am seeing them. In some cases it can be useful. For example:
The actual mail message itself doesn’t include that much detail!
I think I’ll leave it on for now. Thanks.
The ultimate in summarization can perhaps be described by a quote from Douglas Hofstadter: “Words are labels for categories of experience.”
Charles Darwin’s Origin of Species was actually a summary of the much larger work he had been writing on for years, when someone else (Alfred Russel Wallace) wrote him a letter about the same idea of natural selection. This forced him to forge ahead and he wrote his 500-page ‘summary’ in just over a year, which was then published in November 1859.
Darwin had a tendency to dither, and evolution was a very controversial topic in the mid-19th century when the vast majority in Britain believed in divine creation, so he wanted to present all the evidence he could bring to bear from his travels, his experiments and the work of others. Scholars have a tendency to do that, but in the days of printed books, publishers stepped in to scale the volumes down to something they could sell. On the other hand, textbook publishers often wanted new editions of popular textbooks to be bulked up to justify a more expensive new edition. (My Understanding Fiber Optics bulked up from 280 pages in the first edition in 1988 to 800 pages in the 2006 fifth edition.)
That came much later (or at least “what they could sell” was much longer in the 19th century). In Darwin’s time, without TV, radio, Internet, etc, lengthy books were prized entertainment. The books of Charles Dickens were massive. Word count for his top-10:
At about 90k words / 300 pages, most of those are around 1000 pages each.
Anthony Trollope wrote similarly long works.
My experience with the anti-summary was with the first edition of my Internet Starter Kit for Macintosh, which the publisher wanted to bulk up so it was thicker and thus competed better on the shelves next to other books. We added lists of Usenet newsgroups in an appendix.
Of course Dickens’ ’novels’ were serialized and he was paid based on length so both his and his publisher’s motivations were to string readers along.
Which is the opposite of publishers asking their authors to scale things down.
(And Trollope’s novels were not serialized)
Wow. Look at all those words!
It wasn’t just England. In France Victor Hugo, Théophile Gautier, Balzac, et.al. all produced (delightful) monster books. And then there’s Proust. . . . It’s my impression it was the same in the rest of the continent.
Also, when I was at Contemporary Books in the mid-eighties, first as a typesetter then editor, we all noticed that manuscripts seemed to be getting strikingly larger. The answer, of course, was word processors on personal computers.
Dave
Looking back at Adam’s concluding summary, he makes the crucial point that all summaries do not do the same thing and will not serve all people I have seen upper-level physics textbooks that open with Maxwell’s equations because the author thinks of them as an elegant summary of electromagnetic physics. So it may be to a professor, but what a first-year student needs a much longer explanation of what the equations mean both mathematically and physically.
All we can realistically expect from AI is a list of names and subjects mentioned, which can be helpful if you’re trying to find information on specific people, things or topics, but won’t help you understand the subject if you don’t already know something about it.
If it wasn’t for the “Internet Starter Kit”, my husband and I would probably never been able to get online. In my humble opinion, at the time it was the digital equivalent of Gutenberg’s movable type and printing press.
Couldn’t agree more. It’s my device and I have the settings for it. It shouldn’t be up to the developer what happens after they send the notification to me.
Excellent article Adam. Thought provoking for this guy. Thanks.
PS Now provide a speed reading silver bullet for those of us in need. ;) LOL
Looks like Apple is tweaking the notifications summaries a bit.
Great article. I had to login to say so. My first experience with anti-summarization was in High School English class when the teacher said my paper wasn’t long enough. My first experience with the idea of summarization was the scene in A River Runs Through It when the dad keeps telling the son to write his article half as long. I thought, “people value that?”