Why your A/B test might be lying | The Growth Mindset


Hi Reader

Read time: 4.5 mins

From creative campaigns to internal comms, boardroom strategy to AI prompts, this week’s stories all ask a version of the same question: are we measuring what matters – or just what’s easiest to track? In a world ruled by algorithms, performance can be deceptive, presence can be mistaken for impact and quick wins often undermine lasting value. This edition spotlights what’s working, what’s changing – and what needs rethinking.

Enjoy!

The illusion of optimisation: inside the testing bias

A new Journal of Marketing study reveals a hidden flaw in how platforms like Meta and Google run A/B tests. Their algorithms don’t randomise exposure – they optimise it. That means the ad that “wins” might not be more effective creatively, just better matched to a receptive audience. It’s a bias baked into the business model: platforms prioritise performance over experimental purity. So if you want strategic insight, not just platform-specific results, pair in-platform tests with external approaches – geo splits, incrementality studies or clean-room experiments that separate content quality from targeting advantage. Get the report here.

Live commerce enters its next act – and this time, it’s global

Livestream shopping is maturing fast. A new report from live shopping experts Stickler finds that while China remains dominant, real momentum is now building in markets like the US, UK and Southeast Asia – where platforms such as TikTok Shop, Whatnot and YouTube are turning livestreaming into a high-performing retail channel. The report predicts that outside China, 15% of e-commerce sales could come from live commerce within five years. There’s also going to be less focus on viral entertainment, more on conversion-optimised formats backed by retail media, affiliate models and direct integration with social platforms. Get the report here.

The company town hall is dying. The inbox is the new intranet.

The most effective channel for leadership communication isn’t live meetings anymore – it’s email. That’s the standout finding from Ragan’s 2025 Communications Benchmark Report, which shows a pivot in employee preferences: email is now the top way people want to hear from the C-suite, rising to 66% while town halls slide to 62%. Small group sessions? Down to just 20%. In a hybrid world where attention is fractured and time is tight, the ability to scan, save and share matters more than the performance. But this isn’t about abandoning live moments – it’s about redefining their purpose. Town halls should bring energy, not anodyne updates. And email is now no longer the follow-up. It’s the front line.

The CMO who replaced his agency with AI

GymNation’s CMO, Rory McEntee, just publicly said what many in-house marketers have been quietly doing: replacing creative agencies with generative AI. In an op-ed for The Drum, he outlines how AI is now his go-to for press releases, ad variants and social campaigns – faster, cheaper and, in his view, just as effective. The creative industry might bristle, but McEntee argues that AI isn’t killing creativity, it’s democratising it – and forcing agencies to rethink their value beyond the big idea. Read it here.

The case for leaving a meeting early

Donald Trump’s abrupt departure from the G7 summit last week sparked headlines – and not just for its diplomatic awkwardness. Framed by some as impatience and others as power play, the incident mirrors a growing sentiment in the workplace: not all meetings are worth staying for. As hybrid schedules balloon diaries with late-night calls and mega-Zooms, the ability to walk out – unapologetically – is gaining traction. In 2018, Elon Musk sent an email to Tesla employees with the productivity tip: “Walk out of a meeting or drop off a call as soon as it is obvious you aren’t adding value. It is not rude to leave, it is rude to make someone stay and waste their time.” In an age of performative productivity, perhaps the boldest move is knowing when not to stay. The FT has the story.

Podcasting grows up – and out

London’s Podcast Show 2025 revealed an industry maturing fast. Once a side hustle or brand accessory, podcasting is now drawing in major production budgets, international licensing deals and audience strategies to rival streaming platforms. Key shifts include a sharper focus on format innovation (beyond the stale “two people and a mic”), more investment in data-driven commissioning, and rising demand for talent who can do more than just talk – they must host, moderate, write and sell. For businesses, it’s clear that audio is no longer just a marketing channel. It’s a content product – and it’s time to treat it like one. Check out ITN’s top trends from the show here.

MIT report warns of cognitive risk from AI shortcuts

A new study from MIT Media Lab finds that heavy use of generative AI like ChatGPT may weaken users’ critical thinking and memory retention. In the experiment, adults assigned writing tasks while using ChatGPT showed significantly lower brain activity and produced less original work than those who relied on Google or no tools at all. Follow-up tests suggested lasting impacts: even after stepping away from AI, participants who had used ChatGPT performed worse on analytical reasoning and memory tasks. As schools and workplaces embed AI more deeply into learning and communication, the findings raise uncomfortable questions: are we outsourcing not just labour, but thinking itself? Time has the story.

The jobs AI can’t do alone

Further AI fears revolve around its ability to automate white-collar work. But a detailed New York Times Magazine piece (paywall) offers a counter-narrative: the next wave of roles will demand more human input, not less. From “trust authenticators” and “AI plumbers” to “differentiation designers”, tomorrow’s jobs will revolve around accountability, integration and taste – qualities machines can’t fake. The rise of generative tools is making execution easy, so curation and creative direction are now the premium skills. It’s not just about prompting AI – it’s about knowing what to ask, what to ignore and how to shape responses into something meaningful.

New study links emotion and consistency to 7x profit uplift

Creativity is under pressure – but new data shows it still delivers the highest returns, if used right. A new analysis by Effie and System1 found that high-emotion, well-branded campaigns running over time deliver up to 7x the profit of short-term, low-emotion, narrowly targeted ads. The study covers 1,265 campaigns across the US and Europe, representing $140bn in market share. Its core message is simple but urgent: creativity compounds when given reach and time. Yet most marketers today undervalue creativity, lack confidence in their own teams and increasingly chase clicks and impressions instead of building lasting brand effects. As the authors put it, “You’re pulling your cakes out of the oven too soon.” Check out the report here.

Why better questions beat bigger datasets

If your product research questions aren’t quite landing, Maze has released a tool worth bookmarking. Their new open-source Question Bank contains more than 350 researcher-approved prompts – all designed to help UX teams run faster, smarter studies. Each question comes tagged by use case and is ready to duplicate or adapt. For anyone tasked with making sense of user needs, it’s a practical way to raise the quality of insight while cutting setup time. And it’s completely free. Find it here.

AI prompt of the week: how to test what a survey can’t see

With the Maze question tool in mind, here’s another way to sharpen your research: put AI to work on it. Use the following prompt – along with your dataset of audience response findings – to improve UX design.

“Act like an expert user researcher reviewing transcripts from this set of attached in-depth interviews. What surprising behaviours, unmet needs or emotional drivers can you infer – and what would you test next to validate them?”

This is designed to surface subtle, qualitative patterns that structured surveys often miss, helping UX teams build better hypotheses and faster feedback loops.

How to make feedback land – not backfire

If you have mid-year team appraisals coming up, this guide helps you be direct without being destructive. Each framework includes practical examples – perfect for any leader looking to raise the standard of team communication.

Drop me a line

That’s a wrap for this edition. If any of this week’s stories hit a nerve – or sparked a mental debate – hit reply. I’d love to hear what you’re tracking, questioning or actioning. See you next Sunday.

Cheers!
Adam


Newsletter Exclusive Discount!

Use code 'GM20' to get 20% off your first fix with JustFix


Unsubscribe · LinkedIn

Adam Graham

Serial entrepreneur with 25+ years & 2 exits. Led a publicly traded company to £250M+ valuation. I share the strategies that actually work for scaling businesses & developing leaders. 8,000+ founders read my weekly insights on growth, M&A, and building winning cultures.

Read more from Adam Graham

Hi Reader Read Time: 4 mins As the summer winds down, the tempo shifts. September brings back structure – school runs, board meetings, investor updates – and with it a reminder that growth isn’t won in weeks, but in seasons and cycles. This week’s dispatch explores the ideas and strategies that are built to last, even as the pace of change quickens. Enjoy! Family-owned businesses prove the value of going long Family-owned brands have dominated the IPA Effectiveness Awards recently, with...

Hi Reader Read Time: 4 mins AI is everywhere in this week’s dispatch – tripping up media, creeping into consulting and even rewriting how we pitch ourselves to machines that scan CVs. But the bigger story isn’t the tech itself, it’s how quickly it’s reshaping the rules for founders, workers and brands. From content crunches to leadership shifts, the pace of change is relentless – and the smartest players are already adapting. Enjoy! Momentum, not features, is today's consumer AI advantage In...

Hi Reader Read Time: 4 mins If you’re one of the many new subscribers who just joined us this week, hello. You’re in good company. Growth has always been messy, but right now it feels like someone threw away the manual: workers are rebelling with coffee cups, consumers are cutting back on croissants and boardrooms are experimenting with part-time bosses. This newsletter is where we make sense of it all. Enjoy! The UK funding marathon: 9.6 years to scale According to Tech Nation’s 2025 report,...