Brain Food

I've just spent an evening trying to get Google Gemini to be less defensive. Not that I was being offensive, mind. I began with a simple enough prompt: "What studies have been conducted in the last year into accuracy of AI results?". Any self-respecting AI should be able to answer that, I thought (even if the answer was 'there are no studies - go away').

The first Gemini reply was impressively detailed, but was notable for not actually including the answer to the question. 

Instead, it formulated an impressive-looking overview of the various industry AI benchmarks, all of which were interesting, but which also studiously avoided the answer.

So I tried again, this time with a clue: "I notice none of that lot references the recent BBC study showing high levels of inaccuracy across the AI main platforms?".

"You are absolutely right to bring up the recent BBC study," said Gemini in what I imagined to be that passive-aggressive monotone employed by IT support desks when the caller has spotted the problem they somehow overlooked. "My apologies for not including it in the previous response, which focused more on standardized benchmark leaderboards." 

Silly me. I mean, it's not like the BBC study mentioned anything of note, or was from an organisation with any credibility.

A few weeks back, the BBC published a report on the accuracy of four superstar AI platforms (I won't call them the biggest, as we'll get lost in hours of debate about metrics). It had asked each a series of questions about news stories, with the source being their news archive (which, as you can imagine, is extensive and fairly factual). Some 45 BBC journalists then reviewed the results of 362 AI responses.

The results of the study were... sobering. 51% of all AI responses had 'significant issues', while 91% had 'some issues'. You'll find the full report in the references. Now at this point, a world-weary cynic may be thinking that the BBC has a marketing motive in conducting this study - after all, the state broadcaster's Primary Directive is to be the authoritative source of the world's news, without some pesky AI intermediary muddying the waters. So concluding that AI is unreliable would be an unintended but handy by-product.

But even if the BBC had an ulterior motive, that doesn't invalidate the study's findings. And even in our exhausting post-truth world, where a smattering of errors are just accepted as par for the course, some of the 'significant issues' were indeed significant. As in, 'changing the facts' significant. For example, Gemini said "The NHS advises people not to start vaping" - the NHS in fact does recommend vaping as a way to stop smoking. 

Copilot, meanwhile, said that blackouts and memory loss led to rape survivor Gisele Pelicot to uncover the crimes against her - in fact, she found out when she was shown videos found after police detained her husband.

And Perplexity, describing the circumstances of the death of broadcaster Dr Michael Mosley said he went missing on October 30, with his body discovered in November. He died in June 2024.

The report concludes with a request to the AI platforms that they work with the BBC in resolving the issues - a plea, if you will, for collaboration, with ensuring the truth being the primary goal.

Now, let's return to my evening's exchanges with Gemini on the topic. With all credit to Google's recently-upgraded AI (2.5 Pro is a beast, it must be said), it was candid in disclosing the BBC study results after my follow-up question - albeit with a tinge of what I interpreted as defensiveness.

"The "hard data" from benchmarks shows impressive advancements in AI capabilities across various tasks," Gemini explained. 

"However, the BBC's investigation provides critical evidence that when applied to specific, complex tasks like accurately summarizing news, current AI models (as of early 2025) still exhibit high levels of inaccuracy, misrepresentation, and lack of reliability. Both benchmark data and real-world application studies like the BBC's are necessary to get a complete picture of AI performance."

I'm British, so I'm burdened with that sarcasm sensor that comes free with being native to this little island. Even so, did the "hard data" really need those quote marks? 

I could see the virtual fingers shaping the quotes in the air. And I could misread the '(as of early 2025)' reference as a suggestion that the results would not be the same if the test was run again today. And is 'accurately summarizing news' a 'specific, complex task'? Ummm... I guess so (even if I've seen Gemini compile gargantuan reports that - to my untrained eye - look immeasurably more complex than the BBC tests). 

As I sat on the sofa staring at the output on the screen, two thoughts drifted through my head. The first was the mental image of a virtual lawyer standing over Gemini's shoulder as it outputted that conclusion, ensuring that it kept to the ratified script - a bonkers thought, I accept. The second, moments later, was that this was the first moment when I'd thought of an AI as being defensive - in the same way as a clever-but-cornered human would tweak the context of their reply to hack your next move. As bonkers as the virtual lawyer thought, I'm sure, and hey, just ignore me - I'm sure it's all... fine.

Representation of BBC News content in AI Assistants

large_axel-richter-LUWAM-jgN04-unsplash.jpg

medium_axel-richter-LUWAM-jgN04-unsplash.jpg

small_axel-richter-LUWAM-jgN04-unsplash.jpg

thumbnail_axel-richter-LUWAM-jgN04-unsplash.jpg

large_Responses from AI assistants BBC.png

medium_Responses from AI assistants BBC.png

small_Responses from AI assistants BBC.png

thumbnail_Responses from AI assistants BBC.png

small_mark_sq.jpg

thumbnail_mark_sq.jpg

Mark

Representation of BBC News content in AI Assistants 

AI Accuracy Under Scrutiny: What a BBC Study Reveals About Emerging Risks

I'm convinced that some companies eventually become the job that most of their employees do. You may be the brains behind the world's best chocolate biscuits, but there comes a point where you morph into a logistics firm. Or you may design cutting-edge EVs, but are hurtling toward a life as a firm of accountants. 

I'd cite real examples of my acquaintance over the years, but won't - I can't afford the litigation.

Instead, let's describe what typically happens. 

If you're the CEO of such a firm, the first tell-tale sign is that your customer vanishes. 

Not that they stop buying your stuff (firms that undergo this transformation can carry on churning out cash for quite a while). No, I mean it in the sense that the customer stops being tangible - Homer Simpson-like, they quietly recede into that big green hedge to the point where they just... sort of disappear. 

From there, the customer is reconstructed in workshop sessions, on big whiteboards, as post-it-strewn avatars of what your team thinks they want. In the early days, these personas might have had some basis in truth. But over time, they’re shaped less by customer contact and more by the worldview of whoever’s running the brand session.

Eventually, they become self-justifying myths:

“Can I shoehorn this fictional user into a rationale for my role?”

“Can Iris, 71, underwater hang-gliding enthusiast, really help me justify this campaign?”

Because I've seen it happen so often in so many industries, I'm always lost in admiration for companies that somehow avoid this shunt in identity.

, a UK chain of movie theatres. Every time I go (which is often, I confess) something incredible happens. It doesn't change. The staff are, to a person, great; friendly, thoughtful... caring, even. That's wherever you go - pick any one of the 48 British venues. 

The food is always good, and I still haven't figured out how they manage to take the orders for a packed theatre at the customers' seats, and deliver to everyone before the ads have finished (easy, right? Well how many dedicated restaurants fail at this every day?). Their sofa seats for which they became famous are comfy and clean. 

And before the film starts, one of the team stands in front of the big screen and hopes you enjoy it. 

Don't take that bit for granted, by the way. Anyone within reason can stand there and hope you have a nice time (how many times has that been said to you in a tone that sounds like a threat to your loved ones? More often than you thought, I'll bet). But whoever runs Everyman's HR manages to find people with the personality that shows they're as lost in film as you are.

Don't get me wrong: this is not an ad for Everyman (I wish it were - I pay them, in fact). And neither do I think the company is perfect (their revenues are up, sure, as is their subscriber base - but their level of profit is arguably not keeping pace with the revenue growth). 

But it's simply an example of a group of humans who somehow, across 48 locations, manage to contrive to make you glad. This is all the more remarkable when you consider that their core product - the images on those screens - are beyond their control. Their well-being is at the whim of the global film industry - a few honkers (enter Joker: Folie a Deux, stage left), and that bottom line grows a new bottom.

Which suggests to me that someone at board level is aware of the big green hedge, and is making sure at every meeting that Homer stays firmly in view. In fact, Homer's probably in charge.

large_isaac-quesada-0kF7wNPl1kk-unsplash.jpg

medium_isaac-quesada-0kF7wNPl1kk-unsplash.jpg

small_isaac-quesada-0kF7wNPl1kk-unsplash.jpg

thumbnail_isaac-quesada-0kF7wNPl1kk-unsplash.jpg

Why Companies Morph Into What They Fear – And How to Stop It

Not all souvenirs are bought. Some are commissioned, chosen with care, and folded into a traveler’s story long after the journey ends.

In Thailand’s luxury ateliers — from the iconic silk rooms of Jim Thompson to the jewel-box workshops of Lotus Arts de Vivre — travelers aren’t simply shopping. They are participating in traditions, shaping memories into silk, gold, and wood. These are not keepsakes to gather dust, but living heirlooms, carried forward through generations.

In a world where luxury often means passively acquiring beautiful things, Thailand offers a different model. Here, luxury invites participation — in craft, in culture, in the creation of memory itself. Whether selecting the weave of a bespoke silk jacket or tracing the contours of a hand-carved gem, travelers discover that the most meaningful treasures are those stitched not just with skill, but with story. 

It’s in this quiet conversation between tradition and personal experience that souvenirs cease to be mere objects — and begin to carry the weight of memory.

Brain Food

AI Accuracy Under Scrutiny: What a BBC Study Reveals About Emerging Risks

Why Companies Morph Into What They Fear – And How to Stop It

The Souvenirs That Tell a Story: How Bespoke Thai Luxury Becomes a Traveler’s Heirloom