Why AI Agents Fill Me with Dread
If you’ve been reading any media coverage of AI, you’ve probably gotten the idea that Silicon Valley has decided that we’re all going to be using AI agents to help with the tedium of our everyday lives.
I’m pretty technical, generally game for trying new things, and reasonably enthusiastic about the benefits of AI in general. I think AI chatbots backed by Web searches significantly outperform classic search engines, AI editing tools like Grammarly do an excellent job of eliminating mistakes and infelicitous wording, and AI coding is the most fun I’ve had building something on a Mac since HyperCard.
But I’m baffled by the talk of autonomous AI-powered agents like OpenClaw. The breathless examples that are bandied about seem either trivial, impossible for an AI to complete without reading my mind, or something I wouldn’t delegate to hardly anyone. Among much else, AI agents are supposedly going to:
- Schedule meetings and appointments
- Make restaurant reservations
- Buy event tickets
- Book flights and hotel reservations
- Research and buy products
- Respond to email, messages, and social media posts
- Plan group events with friends
- Deal with customer service disputes
Even setting aside my skepticism that an AI agent could actually accomplish any of these tasks, I’m horrified at the idea of one doing them for me. I may be a bit of a control freak, but the only person I’d trust to do most of these things is Tonya, and even then, she’d confirm with me before committing to anything. (As I would with her.) That trust was built over decades of living together, but AI agent proponents seem to expect the same level of trust on day one.
I’ve identified a handful of reasons why all this AI agent talk fills me with dread. See if my reticence resonates with you or if you think I’m merely yelling at the AI agents to get off my lawn:
- Minimal labor savings: Some of the supposed ways an AI agent will save me time are simply spurious. There’s minimal effort in clicking an Unsubscribe link in a newsletter I no longer want (and only I know when I no longer want it). I rarely want to track a package, but when I do, it’s merely a matter of clicking a link if I haven’t already been inundated with delivery status emails.
- Missing my internal context: No matter how much context an AI thinks it will have, it can’t know everything I know, particularly my internal goals and desires. How will it reply to emails or texts for me without knowing how I want to respond, especially since each response will be tailored to the individual and situation? How could it guess where I’d want to eat dinner without knowing what I’m in the mood to eat and how far I’m willing to drive that particular night?
- Lost information I need: Information that comes through my devices falls into three categories: important details that I have to know, information that interests me but I don’t need to retain (see “Reading Doesn’t Fill a Database, It Trains Your Internal LLM,” 28 February 2026), and stuff that’s neither important nor interesting… unless it becomes one or the other for some reason. For instance, I don’t care about email from Tompkins County about roadwork unless I need to drive on one of those roads that day. The relevance of those emails is contextual and changes daily, and only I know which bucket a given piece of information falls into on a given day. An AI agent would constantly have to ask me to recalibrate, which defeats the point of delegating.
- Unacceptably high stakes: For many tasks, the risks of having them done badly are too high to even consider handing them off to an AI agent. I always double and triple-check my own travel arrangements because getting it wrong could mean missing part of a conference. If I have to check an AI’s work that carefully, I might as well do it myself.
- Discomfort with mimicry: Even if an AI could respond to email or text messages accurately and at least roughly in my voice, I’m uncomfortable having someone or something speak for me. It’s not just my comfort at stake—the person on the other end thinks they’re hearing from me, not an AI, and that’s a deception even if the content is accurate. If they later find out an AI wrote that text, they’ll reassess how they read other things I’ve written. That’s why my Driving Focus auto-reply is upfront about who’s actually talking: “Good day, human! This is Adam’s iPhone. He’s driving right now, and while he swears he’s an Above Average™ driver, it’s best not to interrupt him unless it’s urgent. I’ll inform him of your message once he parks the car.”
- Problematic spending: When I do manage to overcome my innate frugality, I want to be certain I’m getting precisely what I want. I can’t imagine giving an AI agent permission to spend money on my behalf.
- No net benefit: I’m a great believer in automation, but as Randall Munroe has pointed out in xkcd (more than once), it’s important to ensure the effort saved outweighs the effort invested. Training a human assistant is time-consuming; training and maintaining an AI assistant feels like it would be vastly more work than just doing the tasks myself. Even getting a chatbot to respond to simple queries in ways I want has taken non-trivial amounts of training.
Ultimately, I wonder if some of the enthusiasm for AI agents comes from people who are sufficiently busy, wealthy, and senior that they’re accustomed to at least the idea of having human assistants.
Years ago, when Tonya and I were struggling with an overwhelming amount of work on TidBITS, Take Control, and parenthood, advice pundits were recommending offloading and outsourcing mundane tasks. We tried hiring a personal assistant for a while, but the experiment wasn’t a success, for many of the same reasons as I’ve outlined above. Too many tasks required personal knowledge, needed a personal touch, or were beyond our assistant’s skill set.
So I’m sure some people benefit from having an AI agent manage aspects of their lives, but I can’t imagine how it would work for me or for most people. And since even one hesitation about what an autonomous AI agent might do is to stop someone from trying one, it’s hard to see this field taking off.
But how about you? If you’ve tried an autonomous AI agent, how has it worked out for you? Have you worked with a human assistant, and if so, what lessons from that would you apply to an AI agent?
It all seems so small scale. So we’re destroying the environment and economy so I can have automated restaurant reservations?
Adam, many thanks for making explicit the doubts I’ve had about this entire topic. AI agents have not filled me with dread because I have had a great deal of difficulty working out what possible use I might put them to. Of course, as I am now retired I no longer schedule meetings, research vendors, competitors, or providers, make travel arrangements, etc. etc. When I was an IT executive I had staff for these tasks. I can see that if AI agents were as easy to work with, as responsive, as trustworthy, and as reliable as my staff were they might be useful. But AI has none of these qualities and I share your disbelief that any serious person would let them loose on their personal data or allow them to make decisions on their behalf.
Haven’t tried any AI agents and thank you for saving me the time of finding they’re a waste of time and energy.
It’s the shiny new thing that marketing departments are talking about in order to secure VC funding and pump their stock prices.
And in a year or two, when the tech proves to be less than expected, they’ll quickly pivot to the next shiny new thing and pretend that the current one never existed at all.
Although I don’t bring any firsthand experience to the topic, much of Adam’s perspective seems reasonable—verging on plain common sense. An essential element called for in all these actions, which AI lacks, is emotional intelligence.
Adam wrote; “Training a human assistant is time-consuming; training and maintaining an AI assistant feels like it would be vastly more work than just doing the tasks myself.” That will certainly be true if one expects equivalent results. Call me cynical—I expect the AI developers know their LLM/agents are inadequate to the task all the while they are pitching the glories of agentic AI. The paramount interest here is to have as many users spend as much time, data, effort and money as possible on further training their LLMs.
Back around 1993 or so, a friend did a back-of-the-envelope calculation. He said, “It’s not worth deleting any file less than 10kb, because the time it takes you to look at the file and decide to delete it is, at our salary, more costly than the cost/kb of disks.” And that’s back when 100mb was a big disk drive…
For the foreseeable future, I expect AI to take no actions on my behalf. “Dread” applies when I think of being unable to avoid others’ AI agents.
Joy of Tech had a nice commentary: The Joy of Tech comic... Human communication is disappearing!.
To me, the dread of AI agents lies in their capacity to run amok with potentially horrible consequences either on your machine or on the internet. And who is responsible for their disastrous actions?
Nasty, nasty stuff.
Dave
I concur. None of the tasks suggested sound like anything I’d want an AI to do, nor are they things I do often enough where any time savings would be significant.
I do restaurant reservations maybe a few times a year and flights/hotels maybe once or twice a year; and for the latter I am extremely picky about getting the flight times, ticket restrictions, and pricing just right. I have no idea how AI would know to do what I want and by the time I teach it, I might as well do it myself!
The only scenario where I can think of this kind of thing working is for repetitive stuff: like if you frequently book certain flights or stay at certain hotels, maybe the AI could do that for you. But you certainly wouldn’t use it for a vacation to Europe where there are a million preferences you’d want just right.
All tasks you listed do not benefit at all from AI. Anyone who lets AI have full access to a live email account is just crazy.
As a developer some of my workflows have been improved by using AI. The workflows are now sorta agentic. Handling translations was a manual process taking at least half a day for each release. Instead of manually editing testimonials it’s now a simple 3 step process where I only need to check the result.
The movie Fifth Element was ahead of its time:
Also “Anticipation denotes intelligence”
Right on, @ace!
I have no experience with any “AI” and hope I never do, and was never a big enough fish to have any staff, so it’s hard to imagine needing one. A self employed family member could use an assistant but as @ace noted, it may be harder to train the person than do it yourself, as tough as that may be. I’m perfectly happy with the asistance my digital devices provide now, I don’t need any more “help” or “suggestions” or “top hits”.
I was trying to formulate a reply in my head while reading comments and I think @Shamino nailed it better than I could have.
This is absolutely not about the users, im(small and)ho, and even more for we sceptical curmudgeons:
there is also the “All the Presidents Men” movie, where a famous quote, possibly from the original reporters, was that to find what’s really going on, “follow the money”.
Bingo! The sales pitch is an ‘assistant’ who will do whatever you ask without complaint… or pay. (Though the latter never actually went way, it just got externalised. And now the cost part is starting to come back to bite users, either directly (pricing) or indirectly (ads)).
Yep. The only one I use is the free version of ChatGPT…and for me it is just a better Google search that can be a 2 way interaction rather than editing my query to get more detailed info. It gives suggested follows ideas and a one word answer goes there. I haven’t decided if it is worth paying for a subscription or even know what extra that gets me because the free version suits what I want it to do.
The fun part is, AI agents are being sold by the likes of Microsoft as productivity aids for work. That motive doesn’t necessarily align with the priorities of the
users, ahem, employees, on whose behalf they will be deployed.“I’m sorry, Dave, but the boss wants you to meet with an important client. I have cancelled your dinner reservation with your wife, and scheduled a Teams meeting with Frank in California instead. This meeting is too important for me to allow you to jeopardize it.”
That one I wouldn’t mind. Since companies are already using chatbots to respond to customer concerns, this would just be two bots botting at each other.
Love this. I’ve spent a lot of time trying to convince people to stop worrying about text files for this reason.
I’m picturing the algorithm (with weights) I’d have to write for the flight reservations. Not before 8 AM and an aisle seat unless the exit row is available but maybe I’d pay for premium economy and it better be non-stop and it should be one of these two airlines…
That’s largely why I don’t delete email or label it manually (I have lots of filters that do appropriate labeling). It’s just not worth my personal CPU cycles when Gmail’s search is so fast. And the very few times I have had trouble finding something recently, an AI-informed search has helped.
And if you’re traveling with your spouse, adding in their preferences as well. It’s a massive multivariable calculation.
@podfeet and I were chatting more about this in Messages in preparation for discussing the topic on her podcast this afternoon, and we were pondering what’s different for those people who have gone down the AI agent path. Perhaps less of a deep-seated belief that we can do whatever it is better than any other person or system, such that they’re willing to accept sub-standard results in exchange for less upfront effort.
Adam, I (and apparently a lot of others) are totally with you, on this issue. Thanks for explicitly stating the case for everyone.
The notion of AI-Agentics is simply ridiculous (at least, at this point in time). We’re still in the age of “AI slop”, and online cautions/disclaimers like “AI information may not be accurate.” So if a person is to trust their everyday, important-to-get-it-right items, to AI, seems pretty foolish to me.
And please, don’t tell me AI can “write code”, like I have heard so many times recently from the general news media. I was a software engineer for years, and I still don’t believe this claim. Maybe AI can generate code that compiles, but does this code actually accomplish anything useful? As I have stated before on TidBits threads, in order to develop good, quality software, one needs to have a deep understanding of the problem to be solved. AI is not nearly capable of that understanding (at least yet).
I’d say something that could be behind the “difference” is, as has been seen throughout history, an individual’s openness to change and interest in change. Scepticism and resistance to newly-popular technologies seems to be a deeply embedded human trait. There isn’t anything inherently wrong about this, in my view, as long as enough people who are curious and risk-takers—let’s call them optimists—exist to balance the pessimists. After all, none of us would be discussing this topic on TBT today if those who felt “the telephone is good enough for me!” had prevailed.
Put another way, it’s similar to the difference between venture capital investments (which, by the way, are usually put into private companies so there isn’t any stock price to pump up), buying shares in a publicly listed company, and purchasing US Treasury bonds. Each has a different level of risk and may or may not be appropriate for a given investor.
:-)
I was definitely in the skeptical camp myself, but I’ve seen with my own eyes that AI can indeed write useful code. At first I used it like an enhanced Google search through Apple’s docs (I write primarily in Swift/SwiftUI). Then I started giving it (ChatGPT at first, now Claude Code) actual coding tasks, but within a more narrow set of parameters. My latest project is a macOS receipt management app which Claude designed from my specifications, and wrote all the code.
From my perspective, I feel that having a software development background is a tremendous advantage in that I’m able to provide very specific guidance and guardrails, like I would to a team of human programmers I might have led. But Adam @ace has stated he does not have that background, and yet Claude has built his SetShot app based on his specifications. So maybe I’m just flattering myself to believe such a background is needed.
I routinely use Perplexity Pro over a general google search or DuckDuckGo search, both of which also proffer A.I. supported answers to queries. I value that. No agentic work there without buying tokens. I was given access today to A.I. Siri on my iPad Pro and am curious and probably positive about exploring Apple’s approach. The relationship is too facile to entrust my life and work just yet to it, I asked it what was I heading into these coming weeks and it simply summarised my Calendar.
One example (that I use) is delegating the management of my investments. Most of my investments are managed by a UBS person (not an AI and I don’t think he uses AI) and he contacts me for major decisions about investment changes. I trust him and that has been rewarded over many years. I would never hand off my portfolio to an AI agent to manage. (I have some investments in a personal account that I “manage” but that consists mostly of dividend stocks and funds that I never touch.)
These kinds of relationships rest on a foundation of trust and I can’t imagine that extending to AI models (at least at this time).
David
I am in agreement with everything that Adam has pointed out. I am also very uneasy about the possible exponential growth of inaccurate, perhaps deliberately biased, information that will result from A.I. “learning” from the incredibly massive amounts of information it has vacuumed up from around the Internet. If some of that information is deliberately skewed or woefully wrong (or fabricated), it will still end up in A.I. “knowledge,” and then consider that the flawed output from A.I. will likely also end up being ingested, more errors added to the same “knowledge” base, and so on, and so on….. Yikes.
I recently started a new job as a software engineer. Management is very excited about being “AI forward.” We recently had training on using the AI Google built into its suite. One of the touted features was being able to point it a collection of documents, have it generate a summary and if you want, have it turned into a podcast so you can listen to the summary. A couple times during the presentation we got the safe-harbor statement that we should be sure to check what the AI produces for accuracy. So, should I check the podcast for accuracy before or after I listen to it? And why would I have it do all that if I have to summarize the documents myself sufficiently to determine if the machine summary is accurate?
Also, the cutesy words that Claude uses when it’s doing its thing annoy me to no end. Just show me a spinning wheel or something until the output is ready to be viewed.
Lastly, I’m curious how “AI forward” we’re going to be when OpenAI, Anthropic, et al start charging what it costs to do the queries. Lighting other people’s money on fire is all well and good, but our company’s money seems like an entirely different thing.
You can now listen to my conversation about this topic with @podfeet. Lots of fun!
Sure—check out SetShot. It’s not done or perfect by any stretch, but I think it’s pretty good.
brief take
Seems to me AI Agent is the slippery slope thing that all the movies are about.
vs general AI assist?: I don’t want AI interfering with my creative writing, so I turn those platforms off (even when Google sneaks it back on)
vs ChatBots? … One thing I notice is that online AI can actually help one create more thoughtful questions. As I was researching a subject the other day, reading the responses helped me conjure up more sophisticated complex questions … resulting in helpful prompts to further and better-informed inquiry.
Although I can see the trap, so to speak, of how students can use it to, basically, do their writing for them.
1 1/2 cents
p.s. I can’t help but laugh though, when it’s obvious that a chatbot, that’s not a real entity, feigns flattery
“what an excellent and nuanced perspective on the subject”, “How incisive of you” LOL
No wonder some folks fall in love with their Bot
Agent is useful as two other a-words: Advisor and Actor.
You need to purchase a replacement for an aging security camera, subscribe to a better video generation model, and book a hotel room.
An Advisor provides a quick compare-and-contrast.
An Actor commits your wallet.
Both save time and effort. Only one can damage your lawn.
The AI pushers like Altman, Musk, Thiel, … knew to get users addicted with prompts. Then add costs/credits per prompt. Then sell the AI to support and corporate help desks, hospitals, etc. Remove the human from humanity and its AI.
Watched a funny but scary youtube video and its maker asks the AI to order him a car, and a robot, and have the robot drive him around. Not like a self-driving car, but to show the next level of fear!
We have every right to fear AI, to hate AI, because we didn’t need it, and it needs regulation.
Meanwhile, I’m on perchance.org, making some renderings for an idea of friend to paint for me. (the AI generated art would be a reference but not the original). So AI can be useful. If I had to hire a designer for this… I would be broke.
I was thinking that it takes alot of effort to command an AI “assistant” for your needs. And if it learns your nuances and subtleties, then it could be useful. And you would be dependent, and you would have to pay, in the end. I can’t even get Carplay to shuffle without repeating songs… or telling Siri to set alarm for forty minutes and she(it) replies, “setting alarm for 15 minutes”. No No NO!
I keep looking back at Do robots dream of electric sheep, PKDick, and wonder, are we going to make slaves of AI, robots and end humanity?
Adam, I know this is an area where your expertise is higher than mine, but an AI agent seems like it could be ideal for endurance training. It could account for your work schedule, the weather forecasts and use biometric data to come up with work out advice to improve or maintain endurance with lowest injury risk. For example - if high heat indexes are forecast - it will prompt for an early morning work out the next day. I know that I would need to plan that out a day or two in advance since morning workouts are not in my routine. The AI agent could even ‘close the loop’ so to speak as it could evaluate results of your workout data and continue to adjust advice. Along the same lines, there are a whole bunch of ‘Life Coach’ functions that will certainly go to AI Agents - I can see them as a big part of chronic disease management like diabetes, hypertension, heart failure and many more conditions. A lot of the management is protocol driven already and AI agents do a pretty good job of taking a general protocol and personalizing it. So like a lot of this tech, you may not be the customer of the AI agent, you will be the target product.
I have a minority opinion. While I agree that AI agents now aren’t up to what we want, I hope and expect that in the future they will be.
Imagine you’re rich or a company bigwig with an executive assistant. It is the executive assistant’s job to do all they can to take care of the drudge work. You fully expect that the assistant will be trusted, know your goals and desires, know what’s important and what’s not, be able to filter contacts and stand in for you in communication, and so on.
So why shouldn’t we all be able to have an AI agent than can do the same? Everything except drop off clothes at the dry cleaner?
Science Fiction predicts it will happen. For example, Rainbows End by Vernor Vinge (2006).
I don’t think it is that far off. As Bill Gates said:
Today I will not use AI for any of the tasks you list @ace Maybe in 10 years @mschmitt if I am still alive
I think it should be possible to start with something simple though. I have just begun thinking about having an AI system that is trained only on my data. It should know nothing about the external world and all about my world. It will not be allowed to contact external sources. The goal is to be able to ask for information that is only mine. I have 1677 text files that document a lot about what and how I do things. I have a lot of videos showing how things are done and PDF manuals. I have all my mail back to the 1990s. Checklists for packaging to go fishing. Digital insurance documents. Work-out checklists. And a lot more. I want the AI to answer simple questions like how do I prepare the toilet in my wife’s cabin for the winter? Where did I buy my TV? What is the tyre pressure I use on my studded winter tyres? Things I do only once a year I have been documenting for many years. I always find my documentation today, but I would love to have a fast answer instead of the slow process it is now since it often is information that is found in the combination of a video, a text file and a PDF manual. I will probably work with Claude to make this system.
This is what I’d like an AI agent to do for me. I want to do a brain dump every weekend:
Ride my bike as many days as possible (needs to check the weather)
Here is a list of random things I need to buy at some point, some have priorities, some do not.
Here’s a list of things I need to get done sometime this week.
I plan on billing about 20 hours and like to do most of my work in the afternoon.
Here are the current meetings on my calendar (or access my calendar for meetings scheduled)
If it’s summer I need to mow the lawn. If it’s snowed I need to shovel.
In a perfect world, with read access to my calendar, the list of things I want to buy and the knowledge that I want to ride my bike daily, my agent would, every Monday say:
I’ve checked the weather and it looks like Monday, Wednesday and Thursday will be good days for you to ride in the morning.
How is your bank balance? Oh you have $225 not assigned? Consider buying Items 3 and 7 on your list. If/when you do, I will cross them off
Since it’s supposed to rain all day Tuesday, I suggest you do “this" batch of errands since they are all near each other. When you get back you can start on the prep for Wednesday’s meeting.
Don’t forgot to start a load of laundry this morning so it can dry on the line today.
Alas, that does not exist yet. So I continue to manage my tasks with a combination of “in my head” and sticky notes for random daily things (because the notes go with me then get tossed), all while cursing the weatherman.
In a nutshell, I need a personal assistant. But I would not want it to do the things you listed in the article.
I have played with the free tiers of Gemini and ChatGPT. It is interesting to find that ChatGPT will pull knowledge from all threads but Gemini will not. Gemini is better with some product research that I do.
Perplexity sounds interesting for some research that I do but I’m not willing to pay for it.
I will try a local model when I get a chance to install, because I like the idea of keeping everything in a contained room.
Some days I feel like this the way the internet used to be, even for simple searches, before it got so commercialized, all while hating the reality of data centers.
Thank you for writing this. I agree completely. I can’t see ever using this. I don’t like the idea for giving up control of anything.
I can see benefits for ai in medical research. I’ve read that it could speed up creating new drugs for diseases significantly. I just don’t want it in my personal life.
Thanks again for the thoughtful article.
Betty
That’s my thinking. Right now those ideas for agents are generated by AI companies desperately trying to find a use for them (and failing). In the future, I see agents as being useful, but for tasks that we can’t think of right now because the concept is too new and radical.
What I want agents to do is keep my life organized. Sort my photos and files by analyzing the content, remind me of bills and expenses or appointments I might have forgotten about (like annual subscription renewals coming up), find me good shopping deals on things that are low-priority but I might have an interest in, etc.
I definitely don’t want AI doing anything automatically for me, but if it could hang out in the background and watch for things that are important to me, that could be valuable.
I agree with just about everything you said. I also find the chatbots a quicker way of doing web searches and aggregation of information to get a quick understanding of some things I was trying to figure out. And it’s helpful with many technical things I do.
But I can’t imagine letting it run wild and being an agent for me. I would dread checking my monthly bills afterward.
I have to wonder if the human species is heading down the path to become Eloi (H G Wells’ Time Machine, 1895). Will we be using less and less brainpower and become more reliant on machines (run by Morlocks!) ?
Yes, I use a smart watch to set timers and sometimes agree to calendar events suggested by iOS (eg based on the content of emails). But I don’t see a benefit in having an AI assistant that tries to anticipate my needs any further than that.
I suppose my reluctance to accept the technology started with the annoying, intrusive MS Office Paperclip (that app, incidentally, was in a Windows folder called Actors - I found I could disable it by renaming the folder).
But maybe I am getting too grumpy
Unfortunately, this AI business is going to make us all robots. Really sad and disappointing. It started with calculators. Folks got to depend on them more and more, and that led to people not able to do even basic math in their heads. I used to teach Mathematics courses on a part time basis at some community colleges, and it was distressing to see how much the students depended on their calculators. Wonder what would happen if the calculator ran out of battery power? I suspect most of them could not think on their feet.
Then it was the cell phone, and things got worse. In fact, the entire world is wired to cell phones.
Many folks (like myself) stick to the “old fashioned” way of doing many things, and fortunately we can think on our feet. But an incident from the other day indicates what is coming with AI. A woman was driving a Tesla on I-5 north about 30 miles south of Seattle. She passed a cop doing almost 80 mph (speed limit is 60 mph). The cop followed her, and noticed strange behavior from her in the car. Also she was swerving some. So, he pulled her over, and apparently while letting the Tesla do the driving, she fell asleep! Luckily she did not hit anyone, but who knows?
So, once again we see that folks will stop doing the thinking.
I’ve been curious about something similar with only feeding it my DEVONthink databases.
Years ago I lived in Northern California during the Loma Prieta Earthquake. Power was out for 3 days. I went to a hardware store for batteries and bottled water (along with hundreds of others). They couldn’t take credit cards without electricity, so it was cash-only. But teenage clerk couldn’t do basic math to give back change. Like you’d give her a $20 bill and your total was $18.73 and she was completely stumped. It was so sad.
The following is only indirectly related to Adam’s article, but seems to address similar problems.
I found the following article in the Italian newspaper “Corriere della Sera” and had it translated into English by DeepL.
Technology Review (Corriere della Sera)
The initial effects of AI use on university students: they are losing the ability to understand
WALTER QUATTROCIOCCHI
I have just finished one of the exam sessions for the Data Management and Analysis course, one of the courses I teach at university. Every year, this is a time for assessing students, but it also offers a unique insight into how their learning styles are changing. This time, however, I left the lecture theatre with a very bitter feeling that I had never experienced before. What was conspicuously absent was meaning.
Many students passed the exam without difficulty, some with excellent results. But a proportion – far from marginal – seemed to belong to a different world.
I saw students trying to read directly from the ChatGPT chat window during their project presentations, relying on the explanations generated by the model as if they were part of their own reasoning. I saw Pearson correlations used without understanding their meaning, linear regressions interpreted in a way that contradicted what the data actually showed, and even probabilities greater than one (for those not in the field, this is, by definition, impossible). These were not simply errors of preparation. They were errors of a different nature.
Up to now, the students’ errors have reflected an incomplete understanding. A formula applied incorrectly, a concept only partially grasped, a logical step that needed reconstructing. A single question was enough to get that process back on track and, often, the student would arrive at the solution independently. Not this time. The presentations were well-organised, the language fluent, the terminology almost always correct. However, all it took was to deviate from the script – to ask why one technique had been chosen over another, or what the significance of a statistical coefficient was – for the entire structure to crumble. The problem was not the fragility of the reasoning. It was its absence.
What struck me most was the way in which these tools had been used. They were not merely for writing the project. They were for studying. And this is where the issue takes on a completely different character. When a system designed to produce the statistically most plausible answer becomes the primary vehicle for learning, the risk lies not merely in arriving at a few incorrect answers. It lies in internalising a different way of constructing knowledge.
It is no coincidence that the projects thus all ended up looking alike. They followed the same structure, the same vocabulary, the same blatant absence of logic.
Rather than reasoning, they seemed to follow the statistical coherence of language. Plausible combinations of concepts which, at the first attempt at depth, struggled to hold up. In some ways, they were reminiscent of certain contemporary pseudo-academic debates: discourses impeccable in form, rich in the right words, but lacking the conceptual structure necessary to make them truly meaningful and full of gaps.
These tools were sold to us as technologies capable of democratising knowledge, breaking down barriers to learning and making everyone more competent. That promise quickly took hold in the language of tech companies, public debate and even parts of academia. What I had just witnessed, however, suggested a very different question: what happens when a machine designed to optimise statistical plausibility becomes the primary tool through which we learn?
It is precisely from this question that the concept of Epistemia arises. When I proposed it, it represented above all an interpretative framework: the idea that the large-scale adoption of systems based on statistical plausibility was altering the environment in which we construct knowledge, gradually shifting the focus from verification to plausibility.
At the MIT Media Lab, an experimental study has shown that the systematic use of ChatGPT in writing reduces cognitive engagement during the task and leaves measurable traces even in the ability to recall and rework what has just been produced. The researchers refer to ‘cognitive debt’: a cognitive debt that accumulates every time mental work is delegated without constructing an internal representation of the problem.
Another study, conducted by Microsoft Research on professionals who use generative AI systems on a daily basis, describes a complementary phenomenon. Critical thinking is changing its function: it is becoming less and less involved in constructing an answer, and increasingly focused on verifying one that has already been produced. It is a seemingly marginal shift, but it alters the very process of learning: when the generation of the answer is automated, the opportunities in which we build understanding also diminish.
Taken together, these studies tell the story of the same transformation: the environment in which we learn, work and attribute value to knowledge is undergoing a fundamental shift.
LLMs are probably one of the most extraordinary technologies developed in recent years. The misunderstanding does not concern their value, but the way in which we have chosen to describe them. We have built a machine of statistical plausibility and have ended up describing it as a machine of knowledge. We are constantly told that LLMs, almost messianically, democratise knowledge, lower the barriers to learning and make everyone more competent. These expressions have rapidly entered common parlance, featuring in tech companies’ campaigns, at conferences, in books and even in some academic debate. By repeating them so often, we have ended up treating them as descriptions of reality, when they were primarily a promise.
Continuing to portray these tools as something they are not is beginning to take on the dimensions of a historical responsibility. The price of this misunderstanding risks being an entire generation of skills going to waste.
A linguistic model constructs probability distributions based on sequences of words. Every response arises from an estimation of the statistically most plausible continuation given a particular context. It is precisely this ability that makes it extraordinarily effective. Plausibility, however, belongs to a different realm from knowledge. One measures the statistical coherence of a response; the other requires observation, verification, comparison with reality and the construction of explanations.
At first glance, this distinction may seem like a trivial matter. In reality, it concerns the very meaning of knowing. For centuries, words such as ‘understand’, ‘learn’, ‘explain’ and ‘know’ have denoted processes in which an answer was inseparable from the process that made it justifiable. Today, we use the same verbs to describe systems that produce results through a radically different mechanism.
Drastically reducing the cost of producing language is not the same as reducing the cost of knowledge. Quite the opposite, in fact. Billions of plausible texts are generated every day; what is truly scarce is the ability to distinguish between them, interpret them, verify them and attribute the correct meaning to them.
The promise of democratisation thus produces a paradoxical effect. Inequality is growing ever stronger. The new dividing line separates those who retain the tools of judgement from those who are gradually replacing them with a reliance on plausibility.
Meanwhile, the scale of the phenomenon continues to grow. Each new generation of models requires more data, more parameters, more processors, more energy and more investment. There is even talk of the need to build an ‘AI CERN’. It is a proposal that perfectly captures the historical moment: we continue to imagine that the answer lies in scaling up the technology, whilst the crucial question concerns the kind of cognitive environment that this very technology is helping to build.
It is this change that I have proposed to call Epistemia. Even before it is a technological transformation, it is a cultural transformation. A society can produce an almost infinite quantity of flawless texts and, at the same time, progressively lose the ability to distinguish an explanation from a plausible sequence of words. Every time we delegate the construction of an answer, we also delegate part of the process through which that answer acquires meaning.
The history of knowledge coincides with the history of the tools that have expanded our cognitive capacities: writing, the printing press, the telescope, the computer. LLMs belong squarely within this tradition. Every major technology, however, also alters the cognitive environment in which it is adopted. The real question, then, is not about how powerful these machines will become. It concerns the criteria by which we will continue to distinguish what we know from what merely appears plausible. Much more than the success of a technology will depend on that answer. Looking back on that morning, I still have the same impression. What was conspicuously absent was not merely a sense of purpose in a university lecture theatre. It was the very meaning of knowledge itself.
*Walter Quattrociocchi is Professor of Computer Science at La Sapienza University in Rome
Sadly that is nothing new. I gave a cashier $5.25 for a $4.24 charge. She unfortunately predicted I’d just give her a $5 so she plugged that in and hit enter too soon. As one who’s dealt with cash since I was kid, this was an easy one and I wanted the dollar back, not all the change. I explained to her how to figure it out. She had to call the manager who rolled his eyes and said “she gets $1.01 back”
And I said that decades ago when those fancy cash registers came out that calculated the change for you. “Pretty soon we’re going to have people how can’t figure out how to make change”
Didn’t take long.
I used to waitress and did cash out of apron pockets. In the dark.
but it’s true)
(yes the equivalent of “walked 5 miles to school in the snow uphill in both directions”
My favorite experience was at a crowded fast food restaurant where the system the registers ran on was down. They were doing ok with calculators, but while waiting in line I could see they were laboriously totaling the bill, then manually consulting a paper chart for the sales tax, then adding that to the total. When it was my turn, I told the teenage clerk, “Why don’t you just multiply the bill amount by 1.0775 to get the total with tax?” (The sales tax rate was 7.75%.) The clerk just looked at me wide-eyed, but tried it, checked it against the chart and exclaimed out loud “Oh my god…that works!” He even called the manager over to see, and they both thanked me as if I had just given them the formula to turn base metal into gold.
Appreciate that! Yeah, I experienced that a number of times in the past, even with the electricity working! It was also sad that a number of them had problems processing a check for payment,
But it still goes on today. I experienced it the other day cashing in my (few) chips at a casino. I have $23 in chips, and to make it easier for the cashier, I also gave her $2 (total of $25). Man, she had difficulties processing that!
Luckily, where we get gas for our cars, paying with cash works out well. The cashiers seem to know what they are doing.
But yeah, really sad!
That is what I did at the casino. Gave her $2 cash, so along with the $23 in chips, the amount she needed to give me was $25. But STILL had issues processing that! I even said that to her, that I was just trying to make it easier.
One other thing folks are forgetting about this AI business (same with on line shopping (folks even order food!)) is exactly what my wife says: folks will become more and more isolated, having very little social life, and hardly ever going to s store any more. Again, a robotic society.
Great story! If you really want to blow their minds, show them how to back out the tax. ;)
Small business owners do this all the time. Give a nice round number to the customer but they still have to pay the tax to the state so it’s got to be entered on the books the right way.
Great story! And it says it all. I actually have witnessed where folks cannot figure out what 10% of something is! Besides 1, that is the easiest multiplication problem to do. Yet they still go to their phone, start up the calculator, and perform the simple arithmetic. And what’s more ironic is that here in Washington state, the sales tax is around 10%. Even then folks can’t figure out the final cost (it’s just about the original price!). And god forbid figuring out what 15% of some thing is. Actually a simple problem, as 15% = 10% + 5%, and 5% = 1/2 of 10%.
As Tom Hanks said in “Forest Gump”, “Stupid is as stupid does”.
I don’t agree memorizing that 2+2=4 or 8x4=32 is going to save—or not save, in the case of using a calculator to do arithmetic—civilization. I think having the ability to think critically and the maturity to have empathy for others are much more important.
I’m convinced one of the reasons so many of my fellow Americans are financially illiterate is that they cannot do simple math in their heads. If you believe you need a CFP and a spreadsheet to figure out that 22.9% interest on your 5-figure CC debt is absolutely 100% no questions asked going to kill you, you’re doomed. The best way to spend within your means is to know what your means are. And where all your spending goes. If you need Excel and Quicken and TurboTax and all that other mumbo jumbo to figure these basics out, you’re already in trouble.
People who can do basic arithmetic in their heads are people who can always easily estimate if they are being suckered. It’s not about being able to split the check without a calculator. It’s rather that those people who have the ability to do that without a calculator are the ones who don’t get taken advantage of and move onwards and upwards.
thanks for elaborating on this. I was also negative on AI agents but I hadn’t articulated all the reasons. Very useful to have your post and everyone’s comments on it.
My Ph.D. thesis was Goal Processing in Autonomous Agents, many years ago. I don’t see modern AI agents handling what my work pointed to as problems needing solutions – and I didn’t even look at social requirements.
I’m relieved that I don’t need to learn much more about modern AI agents yet.
Here’s a long, more general (dread-generating) critique of LLM-based AI:
(His Longer PDF here worth reading). I do appreciate many uses of AI, however. I.e., I’m not as skeptical as Kingsbury, though I do appreciate a lot of Kingsbury’s skepticism.
And here is David Sparks’ take, published yesterday, on Apple’s AI strategy:
which is a worthy topic in itself. I asked ChatGPT what it thinks of the latter article and it was generally favorable within limits, claiming (in a nutshell) that server-based AI will still be required for the most knowledge-intense queries.
I agree with you Adam.
When I was working, my job had me dealing with people who had trouble expressing themselves and often trouble understanding what other people were saying to them. I had to deal with each individual as an individual. I do not see AI doing that. One size does not fit all.
I also often have had (and still have) to deal with people on an emotional/spiritual level. Again this is something AI can’t do.
AI also scares me because I see over dependence on it, causing people not to develop their own critical thinking (which I believe is needed even more with AI out there). These are just my thoughts.
The problem is currently, that major companies are worried that there are all these things that you might want, and they are worried that they will be sitting there watching the crowded in a different direction.
Not inconceivable, although it depends on how much you want to hand over to it. I see this even with human coaching, where there’s tension between what the coach has programmed and what the runner feels like doing on any given day, or is invited to do by a visiting friend, or suddenly has a conflict.
On further reflection, I think this may be a somewhat different situation. You’re really talking about coaching, not an assistant. A coach directs your behavior, whereas an assistant takes on tasks you don’t wish to do. Well-trained AIs could do pretty well at coaching in a variety of scenarios.
I agree—I’d be interested in having something to backstop me and help me be better at the things I already do and that I don’t want to or can’t delegate. I could also imagine an AI agent watching the traffic in and out of my computer and querying me about things that looked concerning—the volume and sophistication of security attacks makes me believe that most people stand a chance only with local help by a capable agent.
If I ever let an AI agent take my calls, it’s going to have to start any response with, “Hi, I’m Eddy, your friendly purveyor of personalized AI slop,” with apologies to the late Douglas Adams.
Adam writes:
In doing so, they will be negotiating with other AI agents set up to do the same thing, and thus cut us out entirely from our own lives.
AI reminds me of the “saying”, “It’s a solution searching for a problem”.
As I stated above, we’re becoming a robotic society. It’s been slowly building up to that, and AI will just speed that along.
Your article exactly matches my thinking. I couldn’t trust AI agents to do anything for me.
I am also retired. Maybe this is why it is so obvious to me that the deeper problem is our collective obsession with doing more and more things faster and faster. Why are we in such a hurry to get wherever we think we’re going? If this is “progress,” the world could use a whole lot less of it.
This is a very good thread which highlights the dangers of letting others, and machines, do your thinking for you. Of course, this applies to a far wider range of national and international issues than the use of AI agents but TB is not the platform for that discussion.
However, on the narrower issue of AI agents perhaps we are being too limited in our thinking. Adam has listed many of the tasks that we have all seen as examples of agents in operation but if we consider that an agent’s role is to (semi-)automate any task that we might do repetitively could they not have real value? Clerical work, administrative work, bureaucratic work, accounting, record keeping, document preparation, many legal tasks, and much more all involve repetitive aspects. If we applied a computer program to any of these we would call it automation and not think twice. But the prospect of applying a computer program called an AI agent generates a strong response.
As matters stand, in most instances computer programs are deterministic but AI is not. This is a genuine cause for concern. But within well defined application areas we have been using AI for years, e.g., GPS navigation, quite happily. AI agents applied within defined fields of knowledge with proper safeguards (not present today) could just be the next level of automation.
Well stated. Retired also (for almost 20 years). But even when I was working, I would not let machines, devices, etc. take over my life! Again, we are becoming more and more a robotic society. And your last sentence in your post is perfect! Thanks for saying it so clearly.
i don’t mean to sound self-serving, but it’s good to know that not everyone is asleep at the wheel!
At the risk of sounding glib, I’d say because I want to make the most of the time I have on this planet. Less time spent on, say, doing the math on tax returns and manually tracking airfares means I have more time to devote to more rewarding activities.
I also think that maximizing one’s working life will increase the chances of having a comfortable retirement (or being able to retire for that matter).
Different strokes for different folks.
Adam,
I think I’m a bit late to the AI party.
I recently have had three or four really excellent uses for Claude and ChatGPT.
I have reason to finally need a stripped down version of Windows on my 2019 iMac. I employed Claude. First I gave Claude all the information I thought it needed. I did not tell it that I use Logitec wireless keyboard and wireless mouse. That was important.
Claude told me that Apple no longer supports Bootcamp but my computer is so old that there was a copy in my Applications folder which did work.
I thought that Claude would lay out all the steps, I would print them out, that would be that. It was far more complicated. I switched to Claude on my iPhone so that I wouldn’t lose the thread and then went one step at a time. Some worked. Some did not. Many did not. I reported the failures and Claude told me a different way. I finally suceeded. I surely would have quit without Claude to keep helping me recover.
And I still don’t like Windows but I use Claude to tell me how to use it when I have to.
What I would love to do is to use an AI to rewrite the Windows app I need to use into an App which runs on the iMac without using Windows at all. Don’t dare try that yet.
Gil
Exactly! I just hope that does not become a continuing occurance, given how lazy some folks are.
I can understand the examples you gave, but unfortunately many folks will rely on AI for so many things that are actually better if they do it on their own. And time management is one of them. I have no issues managing my time, enjoying life, and not relying on technology 100% of the time. Plus I can maintain some social activities, and not be a robot.
Serious question: how do you know this? My current viewpoint is that AI agents have the potential to allow many more people access to the kind of personal services only accessible to elites today. If this comes to pass, it will not lead to everybody losing their free will or ending their ability to think. Every moment not lost to drudge tasks or unnecessary cognitive load is a moment that can be spent creating, musing, enjoying, or dreaming.
I also believe every person has their own strengths and weaknesses in skills and abilities. This means it is impossible to predict for everybody if a product or service will be better or worse than somebody doing something themselves. In fact, I’d say assuming every human is the same isn’t too far off from a “robotic society”.
Agreed. I mean, I made fun of the restaurant reservation thing, but I’m fine with, eg, not having to do all the annoying little things that come with life at the moment.
There is IMHO a serious idiocracy risk here. We are building huge data centers at considerable cost and with huge impact on the environment, the economy, and the labor market. And all this just so somebody can ask a chat bot what the opening hours are for the Target down the road rather than just firing up Google or Apple Maps? There’s a fine line where “convenience” actually becomes outright wasteful stupidity.
Adam, I agree with you. I don’t see the benefit for me. I appreciate using an AI agent to summarize search queries when I need to research something versus reading several different web pages. But to do all these other things? No thanks.
Which also requires massive data centers with substantial impact to the environment. Just because we’re used to it doesn’t mean it doesn’t happen.
I think it depends on the query and the needs of the person searching. There’s plenty of research showing that ChatGPT query uses a lot more energy than a simple Google search. But if a google search prompts a string of follow up queries, it could end up evening out compared to a perhaps fuller ChatGPT response.
I’m no fan of hurting the environment either, but those data centers are accounting for juuuuust a bit more than allowing chat bots.
I have a developing concern about AI that I freely admit is not fully formed.
Over the decades teaching I’ve noted the broad changes students have faced into. From traditional media with little hooks for digital images or content to hook into until the industries shifted wholesale into digital production and eventually distribution. From a desert to an ocean of content.
As the web developed and as apps came on the scene a shift came too as money was made and their primary engagement became more and more passive, consumers rather than the producers. The French artist Louise Druhl highlighted what she called the ‘slope of the internet’. The early days of the web had a gentle slope, encouraging meandering around, nowadays that slope is much steeper, funnelling users rapidly into fewer and fewer sites which are harder to leave. This passivity is combined now with just how big the digital realm is, a sense of being dwarfed, in something huge, bigger than you. This is the seedbed AI is being planted in.
I think young folks have to find their voice, whatever form that takes, the process of working stuff out, digging for yourself, all feeds into that. AI could become a kind of silencing, as the mental muscles of working things out, the learning of what you are like, how you do things, are underused.
Just want to add that my comments in this thread are about AI Agents, not Generative AI. I draw a distinction between AI-powered task automators (agents, including OpenClaw) and large language model-trained chat bots (generative AI’s, including AI-assisted Google searches).
In any case, AI’s, no matter the “flavor”, have been improving online activities for much longer than generative AI’s have been around. For example, Google Translate used a Deep Learning technique from 2016 to 2020. Frequent users probably remember how the quality of translations greatly increased from the often context-free and unintentionally hilarious output that the non-AI engine spat out.
And don’t forget all the water and electricity they are consuming! Those are at least 2 of the reasons why many “areas” are not allowing them to be built. Even here in Seattle, with all the technology companies, there are laws in place restricting them being built.
Just started re-reading Dune by Frank Herbert and came across this quote:
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
Excellent and very timely post! And yeah, feels like AI technology is trying to take over.
Not sure if “AI technology is trying to take over,” but sure that investors in AI technology are taking over. We keep getting distracted from the real problem.
Not exactly taking over, but I suspect so many folks will use it. Just like the calculator and cell phones, it will make folks more “robotic”, have way less of a social life, and many of them will think less “on their feet”. It’s bad enough as it is right now.
I logged in just to say THANK YOU, well said!!
Also, I can’t AVOID the AI any more. Everything shoves it at me whether I like it or not.
this infinitely underestimates the value of LLM-AI.
Verizon is rolling out a new plan for new customers and those of us with 1 line. (Simplicity Plan). I pay $70 for a single line with 5 gig of data. Just me. As a widow, I don’t need or want another line. So I called Verizon and got the AI Chat Robot. It assured me I didn’t qualify for $30 as a new customer, but could change to the new plan with unlimited data and Hot spot for $45. I asked the robot if all taxes and fees were included and it said yes. But it wouldn’t transfer me to an actual person to talk to! Then the AI robot texted me this same info: $45 with taxes and fees included. I hung up that call, waited a while and called back. I was able to talk to a live person! I said I wanted the new plan and a confirmation that taxes and fees were included. He said that wasn’t true. I read the text the robot had sent. He said he had to check with an admin. THE AI ROBOT WAS WRONG! Taxes and fees are extra! So the plan was going to be $51. I did change to the new plan –but learned a lesson about AI Chat Robots! They don’t always know what they are talking about!
This relates to the “People and Companies Should Be Legally Liable for Their AI Agents” thread: I feel like companies must be liable for whatever their customer support robots say.
If their robots aren’t trustworthy enough for them to do that, then they shouldn’t be using them.
Ain’t it the truth!!!
Well here’s an excellent example of what I have been saying about AI and robots: Tesla’s Safety Camera Lets Woman Cruising at Highway Speed While Completely Passed Out
Definitely scary!
To reply to my own comment, today I had an issue with Alaska Airlines and I used their chatbot. It ended up needing to redirect me to a human, so I didn’t use the AI, but I noticed this disclaimer:
Well that’s baloney. Why should it be the customer’s responsibility to make up for a decision they had no input in? If the company choses bots to cut payroll cost (or whatever other reason), it’s their sole responsibility. If the bot messes up, they should be just as liable as if their human rep messed up. They’re trying to move the goalposts here but I’m confident consumers aren’t going to just let that slide. Not when it involves AI given its current public perception.
I wonder if Frank Herbert felt the same way if women turned their thinking over to machines, etc.
Could not agree more! But I would not be surprised if such companies have some kind of disclaimer buried deep in some “available” documentation that gets them out of such situations. Along these same lines, I read that at banks, their checking and savings accounts (and probably other types of accounts) have of course documentation, and there are some “bizarre” rules and regulations that can severely restrict one’s access to their funds. That really gets me!
I agree completely, but I would also like companies to be liable for what their customer support people say. (If anyone cares, it was Cunard. The purser attendant told me something in person. When it turned out to be false and I asked a different person, I got the original answer, with no hesitation in either case. I am absolutely convinced it was a training issue, and someone decided to go cheap and have incomplete training—similar to AI.)
That is true, too! About a year ago I had to implement my mom’s long-term care insurance. That was incredibly complicated and tedious, involving lots of paperwork, forms, etc. Every time I called the insurance company and was an hold waiting to speak to a person, the hold message said something along the lines of “Anything you’re told on this call does not change the terms of your policy contract.”
Since I was mostly calling to get information and find out the procedures and make sure everything was done according to their requirements, I found this disclaimer outrageous: it basically meant they could lie and tell me whatever they wanted, and it didn’t matter, since only the terms of the policy contract could be enforced. (A contract, BTW, that I didn’t have since my mom had bought into it two decades earlier.)
Sure enough, there were several instances were they told me stuff, I did the stuff, and they later told me something different claiming whatever I’d been told before was wrong. It frequently delayed the process and repayment. “Oh no, that’s not right. You have to do X first. Now we’ll have to refile the paperwork from scratch and we have three weeks to verify before we continue.”
Absurd, but technically within the law.
I had that happen, too, with MetLife. I ascribe no deliberate misleading, just improper training of the front-line rep (the same as Cunard, except that was front-line and second tier reps). Of course, whether the misleading is intentional or not, the effect is the same, and the lack of consequences for the organization.