Episode 10: Is AI Making Us Faster But Dumber?

Brian Milner (00:22) Welcome back in everyone. Welcome. This is the People Over Prompts podcast where we talk about the future of teamwork in the age of AI. I am Brian Milner. I am your host, and I am here as always. And today I I've brought what I think is going to be an interesting topic for you. I titled the episode Is AI Making Us Faster But Dumber? And it's a little clickbaity I know there with the the the title, but let me let me get into it because it's based on some interesting research that I became aware of here recently, and I think I think you'll find this interesting as well. basically there's two kind of sources for this that I want to talk to you about. One is kind of the the source of all of it is is a A research study called the Narrative AI Advantage question mark. it was done by Harvard, the University of Washington, and the MIT SOLV program. And that is also then mentioned and discussed in a Harvard Business Review article called AI is undermining leaders' judgment. Here's what to do about it. that came out August 19th, 2026 here, written by Leonid Sudakov and Nathan Fur. And both of those are kind of interrelated, talking about kind of a similar problem. So let me kind of set the table here for you as far as what are we talking about? I I think we've discussed this in previous episodes, but one of the things I think that is really key about understanding AI and the human interaction with AI is that judgment is the thing that humans need to retain. At least at this moment in time, There may be times when when that that problem is overcome, but as of the time of this recording, that is the inherent flaw in in that working relationship is AI does not make good judgment calls, and any system that we put in place to to engineer how AI and humans work together needs to retain the human judgment element of it. And the the the thing that I think these articles are bringing up is that it's very possible that AI is actually because of its overuse that it it could actually be kind of giving us atrophy of our judgment making skills. And that we we we aren't able to make judgment calls as well because we are offloading so much to ai. So the analogy here is let's say that you need to you decide that you want to get more exercise. And so in order to do that, you decide that you're gonna start biking to work. And while biking to work, you you s you encounter a second problem, which is that you're You're often arriving late at work. It's taking you longer to bike than it did than it took to previously get to work. And so now you have a secondary problem, which is I'm getting to work late. So you investigate and find out that you can get these e bikes or mopeds or something of that nature where if you if you take those, you can use the motor on occasion, but you can still pedal as well. It's it's really up to you. But you start using that e-bike and you you realize that you're using the motor more and more and more. And before you know it, your your leg muscles, which had been something that it was starting to build up now are starting to atrophy because you're using more and more this this onboard motor of your bike, right? That's similar to what I think they're trying to describe here in this research, is that we have this innate judgment skill that humans have developed and that that skill is undergoing atrophy because we are offloading more and more to AI. the The HBR article refers to this and the research refers to this as something they calls original judgment. they they define original judgment as the uniquely human capacity to see beyond the dominant narrative and make choices that represent your values under uncertainty. So put another way, just in my own words. That you are comfortable, or that you you develop the skill of making a judgment call without necessarily needing all the information before you make the judgment call, right? That's something that AI doesn't do as well at. And that's something that we routinely do as humans and and have come to kind of feel comfortable with, right? This quote from the HBR HBR article says as organizations gain more intelligence, their leaders are being trained out of the very capacity that creates competitive advantage, original judgment. It also says sophisticated decision support systems are training leaders to defer, not originate. And the kind of question to wrestle with there is at what point does Assistance actually becomes substitution, right? And I think that's what we're we're having to wrestle with as we work with AI. So, what is judgment here? What are we talking about when we talk about judgment? Well, in the HBR article, they talk about two things: breadth of perception, seeing weak signals, unusual patterns, things outside the the obvious frame. and independence of interpretation, forming your your own view rather than deferring to the model or even to the group if you're you're with a group of people. AI is handling more of that routine sort of analysis. And that's leaving humans with more ambiguity, uncertainty, exceptions, values, and novel situations. So judgment should become more important, not less. And that's kind of the dichotomy I think that we're finding ourselves in is at the very point in time when judgment is more needed than ever, the the use of AI is actually weakening those muscles in us in us and and and and It's the the precise moment in history when we need those skills. We're we're offloading the things that gave us that information, that gave us that ability, like research and and analysis and evaluation of of of data, interpretation of the data. you know, i there there's there's a story in the data, and we're we're more and more relying on AI to tell us. What the story is in the data. So that that's kind of the question for you when as you think about your place in this is how often do you ask AI what it thinks before you decide what you think? To use another word that we use all the time in the Agile world, this is grounding. Right? It's grounding your decision based or anchoring your decision based in someone else's opinion. if you're in a group kind of environment in a in a meeting and someone who you feel is in a position of authority speaks up first when there's a question and says, Well, I think this should be the answer. You're much more likely to just go along and say, Yeah, that's probably right. And what this research is showing us is that That same anchoring is taking place with AI. AI is giving that initial analysis response, and rather than actually approve, rather than actually analyzing it and trying to determine if it's correct, we're just sort of going along with it and saying, yeah, it's probably right. So It's important to kind of look at the data here. There's a there's a an organization called MIT Solve, that tries to it's a program at MIT that that runs global competitions for innovators that are trying to solve really big social problems. And they get lots and lots of submissions for funding. And in in this study they were trying to to we've To examine that evaluation process and really kind of understand how AI might enter into the picture. So there were 228 evaluators of submissions in this study, 48 real innovation submissions. And 3,002 decisions that had to be made based off of those. There were three separate conditions they looked at. One, human only, right? Only people looking at this and making decisions. The second group was humans making a decision with an AI recommendation. And then the third, and this is where it gets interesting: AI recommendation plus an AI explanation. Of the recommendation. So compare it against it kind of the independent bit benchmark then that they had. Here's what they found. The human-only group they hit about 66% with the real evaluators. So evaluators agreed with the expert panel that was evaluating these things on whether the submission should pass or fail about 66% of the time. If the the evaluators had an AI recommendation only, not the explanation, but just the recommendation, that number went up to 69.5%. when when they actually saw the AI pass fail recommendation. They matched that expert opinion 69.5% of the time. Then the third, when they got the AI recommendation plus the AI explanation. 64.3% when evaluators saw the same AI recommendation plus a written rationale, their their final decision matched the experts only 64.3% of the time. So it went down. And that's an important nuance. I think I think that's a a revealing piece of data there. AI wasn't the problem in itself. It wasn't even the recommendation that was the problem. It was the recommendation plus the explanation. And something about that caused the humans to find less wrong answers from the AI. What what they found was that the the the humans the people stopped checking, basically. They they They accepted the AI recommendations with narratives, and and they found that when they accepted the AI recommendations with the narratives, they they viewed eight point three percentage points less source material in the recommendations. They just stopped getting into the guts of it. And they they scrolled eight point six percentage points less of the information. So they they concluded here from this that narrative explanations substitute then for independent evaluation rather than enhancing it. And that's kind of the the e-bike analogy there again. The problem isn't using the motor, it's it's that we stop pedaling altogether. so examples of that maybe that that might hit closer to home for you, AI summarizes of reports. And you never actually read the report. AI evaluates a candidate and you just skim the resume afterward. AI analyzes customer feedback and you accept its interpretation. Or AI recommends product direction and discussion begins from AI's answer. That's the danger point. So We become worse than catching AI's mistakes, is kind of the end result. there's a term that the research team came up with for this. They called it productive override. What they mean is that this is productive override is when when AI gets something wrong, the the human in the loop spots it and says, Yeah, there's a mistake. The human disagrees with what the AI comes up with, and then the final decision actually improves. That's what they refer to as productive override. The researchers describe it as disagreement that improves decisions. And with the narrative explanations in the in the equation, productive overrides dropped 11.9 percentage points. And dropped 14.7 points when AI incorrectly recommended rejection. So that's an interesting point there as well, right? Let that sink in. Just overall 11.9% drop. But it's it's especially egregious when it made a negative recommendation. So think about that in the kind of things that you have AI recommend for you. you know, I know one of the examples I just used was skimming a resume. Well, AI is being used all the time to help with candidate submissions. And I think what this research is showing is that it we're more likely to accept that negative rejection when there's a narrative attached to it. And it it's even more so if it's negative than it than if it's positive. So that that that's important to kind of recognize that distinction that maybe our greatest value in this loop, in this chain, isn't agreeing with AI, but it's it's knowing when AI is wrong. And that's really the important skill or muscle to train is to be able to catch and find when AI is wrong. And I think that is, I know software teams are finding this. That's where practices are being developed, enhanced. we're we're trying to find those places and and and and and means to identify the AI error more consistently right so w why is this the case? Why did why does the explanation make things so much worse? Here's what the research kind of said: that large language model explanations sound like human reasoning. This sound familiar. So the explanations sound like they come from a human. And so, guess what we do as humans? We we give them trust, right? But the the papers here, the the the research paper, it stresses that a a fluent narrative is not necessarily a faithful account of how the model reached the decision. It's it's interacting with us in a way that makes us feel comfortable, but it's not necessarily accurate. So the explanation is not necessarily matching reality, right? The the the most dangerous AI answer isn't the obviously stupid one, right? It's not the obviously wrong answer. It's the wrong answer, but the wrong answer with a brilliant explanation. That's the most dangerous AI response. And unfortunately, the systems we're using are engineered to that. They are engineered to give brilliant responses or br brilliant explanations and not necessarily engineered to find the right answer. Right? So that gets us to what do we do about it? How do we protect ourselves? How do we protect judgment in our organizations and in our own work? A couple things here to to kind of leave you with from the the article from what HBR kind of identified here. Make an initial judgment first, then compare it with AI. Don't let AI establish the frame before you have one. Don't anchor in what the AI already has decided. Right? You make your judgment and then you can have the argument back and forth with AI about whether you agree or disagree with what it's saying. So that means that you you have you have to put your own thinking first. Think before you ask AI. Then then use AI to challenge that thinking, not to replace that thinking. Instead of asking AI, what should I do in the scenario? Or which option is the best option. Ask it things like, what am I overlooking here? What assumption am I making that might be wrong? Make the strongest case against my view. what what evidence what evidence might change my mind about this? what would someone with a very different perspective than mine see in this? You want AI to be more like a sparring partner, not not some mystical oracle that's dulling out. wisdom right the third thing they say is preserve the disagreement better questions here that that organizations can ask is not are employees following the ai but but can employees still recognize when ai is wrong so what's the what's the systems we have in place to try to determine When AI is incorrect. How do we catch the errors? Do we have practices in place to help the employees find when AI is incorrect? Do we counterbalance or counter-check in any way? I know that's something I'm constantly doing if I use AI for something, is to check multiple sources. make sure that it's not hallucinating in some way. Build also build intentional descent. The HBR article recommends what it calls structured curiosity and intentional dissent. So it talks about useful practices here that it describes as things like seeking unusual sources. Look for weak signals, not just the strong signals. Delay premature closure. For any decision. Do pre-mortems. I love that. I think that's a great practice. If we were, one of the great things I love to do is to say, if we were going to royally screw this up and get it wrong, what would we have to do to ensure that we did everything wrong, right? That got this the worst possible result out of this decision. Then you can compare that and say, is there anything that we're doing, or is there anything that we're planning to do that aligns with maybe some of that worst possible way that we could actually screw this up? So that pre-mortem is is a good structured way to kind of go about that. invite more independent perspectives, widen the circle. Especially the more important the decision, the more voices you need. The bring in people that you know might be contrarian to your opinion. Seek out those opinions, right? Because that's the danger with AI, is you you kind of get locked in that echo chamber. And since AI is primarily trying to give you a response that you expect. And that's what the prediction is trying to do is predict what you expect to hear, then it might think that you expect to hear the wrong answer. And you get locked in that echo chamber because it will endlessly try to prop you up. And you see these cases pop up all the time in the news, with people who who have been been been led astray from from AI. And then the la the last thing they say here is dependently pressure test your conclusions. or I'm sorry, not dil dependently, deliberately pressure test your conclusions. maybe like I said, this is the sparring partner part. Rather than asking AI for an answer, say, here's my here's what I think is the right answer. Tell me all the ways this could be wrong. Help me find the the ways that this could crack. What are the things I should consider that could really fail in this approach? In the HBR article, they warn they they give a warning here. They say AI outputs can arrive with a veneer of objectivity that makes disagreement appear irrational. I love that. It's very academic language, but I I I love that that that thought. A veneer of objectivity that makes disagreement appear irrational. Yeah. Heck yeah. That's what it feels like when you're talking to AI sometimes is boy, there's this veneer of certainty about the response that you're getting from AI, and that you just sort of feel like, well, gosh, it would be foolish of me to think that's wrong. It'd be foolish of me to go another direction. it has all the this wealth of knowledge at its fingertips. Certainly it's it's knows more than I do about this. Mmm, not so much. Right. I think you you have to preserve that human judgment. so you know, like the the the e-bike analogy here, I think it's important that we continue to pedal. I think it's important that we continue to flex that judgment muscle and and find ways to strengthen that. I think that's the important human skill right now in our AI relationships is is that human judgment. AI can help us go further and faster, but but judgment, judgment may be one of the cognitive muscles that humans can least afford to let weaken at this moment. So our our goal is not, or the the question is not human or AI. that that gets at kind of the core, I think, of what we try to do in this podcast. We're not anti-AI. So it's not a question of human or AI. It's AI provides capability, but the human retains judgment. And if we're passing that judgment over to the AI, we're potentially setting ourselves up for failure and even potentially setting ourselves up to be weaker the next time that we have to make a judgment decision. So offloading those judgment decisions don't necessarily doesn't necessarily do us any favors, I think is what this research is showing us. So kind of a a final line I pulled out from the research that I thought was really interesting. Designing AI assistance to preserve selective human disagreement is essential. And I I think that's one of the premises that I'm trying to hit on in this this whole entire podcast is. Is that human judgment is essential. that AI can help us do a lot of things, and we're to have a productive, healthy relationship with AI, I think is it's about recognizing those boundaries, right? What it can do and what it can do well, and where it falters, where it starts to fail on us. And that's what we're all trying to learn right now. So I I hope that you'll stay with us on this podcast journey because this is what we're trying to explore. Right? I think that the the brave new world, if you will, of of AI human interaction is gonna change things in ways that we can't really even comprehend yet. societally, psychologically, that's the kind of thing we try to hit on in this podcast. So if you you want to perhaps give me some suggestions of of topics you want to hear, or, you know, I I'm I'm Do episodes like this with myself, but I also will do episodes with guests from time to time as well. So if there's a guest you want me to have on, please let me know. you can email me at Brian at agilityevolved.com. That's my company, AgilityEvolved. So Brian at agilityevolved.com, Brian with an I. And I'd love to hear from you. I'd love to hear your suggestions or any feedback on the on the podcast in general. I want to make this podcast. Podcast you enjoy. So if there's things I can do to make it better for you, please let me know. my ask from you is just a few simple things. One, make sure you're subscribed to this so that you don't miss an episode. And then tell a friend about it if you l enjoyed this one. point them to a specific episode. If there's one that you found really particularly interesting, point them to that episode. And that's the way that we actually grow and and spread the podcast, is just through your word of mouth. no. ads anywhere else, no ads here in the show itself. so if you want to help support it, that's the best thing you can do is just try to pass it on to somebody else who might enjoy it. And other than that, we're gonna keep plowing away here and trying to put these in-depth discussions before you just about every week as much as we can. So we hope you'll you'll stick around with us. And as I I often say, I hope you're having a great week. I hope you continue to have a great week. And we'll talk to you next time when we get back here on another episode of People Over Prompts.