How to Survive CRO Politics: Workshop Notes on Getting People to Give a Damn About Your Experiments

Experimentation isn't just a numbers game. It's a people game, too. And sometimes the people part is considerably harder.

As CRO practitioners, we're pretty good at talking about the technical side of experimentation. We discuss statistical significance, sample sizes, testing velocity, analytics setups, research methods, and all the other things that make our little experimentation brains happy.

But there's another side of the job that doesn't get quite as much attention: getting people to actually support the work.

Who needs to approve your experiments? Who gets to decide whether the results are credible? What happens when your internal champion leaves? And how do you convince an organization to keep investing in experimentation when not every test is a winner?

These were some of the questions behind our workshop, How to Survive CRO Politics and Get People to Give a Damn About Your Experiments (Without Losing Your Mind).

We spent an hour discussing the human and organizational side of experimentation, sharing some of our own experiences from running CRO programs, and looking at practical ways to make the whole process a little less politically exhausting.

Here are the main takeaways.

1. Your biggest threat might not be a bad experiment. It might be a new executive.

Let's start with a story.

Almost three years ago, we started working with a company that was enthusiastic about bringing in external CRO support. The person who bought the project understood what we were doing and was ready to get started.

Two weeks into the engagement, he left the company.

Not ideal.

We suddenly found ourselves working with a bunch of senior stakeholders who hadn't been involved in the original decision and didn't really understand what our program was supposed to do.

Instead of getting straight into testing, we had to step back and explain the value of experimentation all over again. We eventually reached a point where we said something fairly simple:

For this to work, we need an internal champion.

Someone with the authority to make decisions, coordinate internally, and act as our main point of contact.

They appointed a VP of Growth, who became a fantastic champion and remains one to this day.

Since then, the company has gone through significant restructuring. We've seen around ten product leaders at executive or director level leave, alongside another major agency.

And somehow, the experimentation program survived.

The experience taught us a few things:

• Your original buyer might not be around for very long.

• Senior stakeholders might have very different understandings of what your program actually does.

• Organizational changes can completely reset the priorities and expectations around your work.

• Relationships beyond your original champion are essential.

Ryan made another observation during the workshop: in his experience, a new executive joining an organization is one of the biggest threats to an existing experimentation program.

New leaders often arrive with pressure to make an immediate impact. They want to shake things up, demonstrate progress, and sometimes bring work in-house or restructure existing teams.

That's not necessarily unreasonable. But it does mean experimentation practitioners need to be prepared to explain the value of their work to a new audience, potentially more than once.

Practical takeaway: Don't build your entire program around one person's support. Develop relationships across the organization, identify potential champions, and make sure more than one person understands why the program exists.

2. You're not just getting experiments approved. You're getting three different things approved.

When we talk about CRO politics, the conversation often revolves around getting a test approved.

Someone doesn't like your idea. A stakeholder wants to change the design. An executive insists on testing something that doesn't make much sense.

But there's a bigger picture.

We see three layers of approval that experimentation practitioners need to navigate:

Layer 1: The experiment

Can we run this test?

This is the most obvious one. You need buy-in for the idea, the resources, the implementation, and the time required to run it.

Layer 2: The interpretation

When the results come back, will we be allowed to believe them?

This is where things get more complicated.

Different stakeholders may trust different data sources. They may prioritize different metrics depending on how their own performance is measured. They may disagree about what constitutes a meaningful result or whether the evidence is strong enough to act on.

If these expectations haven't been discussed beforehand, interpreting results can become a political exercise.

You might have a perfectly valid experiment, but if nobody agrees on what evidence counts, the conversation can quickly turn into an argument about whose numbers are correct.

Layer 3: The existence of the program

Does experimentation deserve continued investment?

Even if individual experiments are approved and the team agrees on how to interpret the results, there's still the bigger question of whether the program itself is worth funding.

This is where the conversation shifts from individual tests to business impact.

It's not enough to say that you've run 40 experiments or learned a lot. Eventually, someone wants to know what the organization is getting in return.

We'll come back to this when we discuss measuring experimentation ROI.

3. Stop treating losing experiments like personal failures

This one is easier said than done.

When you're responsible for an experimentation program, it's very easy to start identifying yourself with its results.

You come up with an idea, research it, defend it, convince stakeholders to support it, coordinate the implementation, and wait for the results.

And then it loses.

Suddenly, it can feel like everyone is looking at you and asking why you got it wrong.

But a wrong hypothesis doesn't mean you're bad at your job.

One thing that can help is increasing experimentation velocity. Gerda shared a comparison with creating content: when you publish occasionally, every unsuccessful post can feel like a personal failure. When you publish regularly, one post underperforming becomes another data point.

Experimentation works similarly.

The more you test, the easier it becomes to see individual results as part of a larger learning process rather than a verdict on your competence.

Of course, not every losing experiment is automatically valuable. Poor planning, bad implementation, or avoidable mistakes should still be examined. But an experiment producing an undesirable result is not the same thing as the team making a bad decision.

Reframe what experimentation offers the organization

There's another important psychological shift here.

When introducing experimentation to an organization, practitioners sometimes position it as a way to prove that other people's ideas are wrong.

"We're going to test everything and let the data decide."

It might sound reasonable from a methodological perspective, but imagine hearing that as a designer, marketer, or product manager.

You've spent hours working on something, and now someone is telling you they're going to test it and potentially prove that your work was a bad idea.

Nobody particularly enjoys having their homework graded.

Instead, position experimentation as a way to take pressure off the team.

Without experimentation, every change can feel like it has to be perfect before it reaches customers. There are endless revisions, debates, and attempts to predict what users will do.

With experimentation, you can try something, observe what happens, and decide whether it's worth investing more resources in.

The message becomes:

You don't have to be right every time. We can figure it out together.

The psychological safety problem

This connects directly to psychological safety.

If people are punished for proposing ideas that don't work, or if only winning experiments are celebrated, the organization creates incentives to play it safe.

People may start:

• Proposing only ideas they believe will win.

• Avoiding experiments that challenge senior stakeholders.

• Cherry-picking results that make their work look successful.

• Hiding uncertainty instead of discussing it.

• Prioritizing the appearance of success over actual learning.

A team that punishes every negative result creates incentives to hide uncertainty rather than learn from it.

And that's a pretty terrible environment for experimentation.

4. Learn to speak the language of your CRO characters

One of the frameworks we introduced during the workshop was the CRO Character Framework.

The idea is simple: different stakeholders can look at exactly the same experiment and care about completely different things.

Consider a few familiar characters:

These are broad examples, of course. Roles overlap, and every organization has its own dynamics.

But the important point is that your stakeholders don't necessarily measure success in the same way you do.

An executive might want to understand the potential business impact. A designer might be worried about spending days creating variations that never make it live. An engineer might be thinking about the technical debt created by another experiment.

If you communicate with everyone in exactly the same way, you're likely to miss what actually matters to them.

Same evidence. Different translation.

This isn't about manipulating people or changing the facts to suit whoever you're talking to.

It's about explaining why the evidence is relevant to their responsibilities.

For example, the same experiment might be communicated as:

• To an executive: Here's what this could mean for business performance.

• To a product lead: Here's what we've learned about customer needs and product priorities.

• To a designer: Here's what the experiment tells us about the user experience.

• To an engineer: Here's what we learned relative to the implementation effort.

• To an analyst: Here's how reliable the evidence is and what limitations we need to consider.

The underlying evidence stays the same. The context you provide changes.

5. A little homework: Who needs to believe you?

During the workshop, we asked attendees to choose one real person in their organization whose support is important to their experimentation program.

We encouraged everyone to grab a pen and paper.

Not because handwriting is magically superior to typing, but because putting your thoughts somewhere outside your head can help you examine them more clearly.

Try answering these five questions:

1. Who are they?

What's their role in the organization, and what role do they play in your experimentation program? Those two things aren't necessarily the same.

2. What do they care about?

How do they measure success in their own job? What pressures or priorities might they be dealing with?

3. What evidence do you have?

What have you actually learned from interacting with them? What do you know about their concerns, and what are you simply assuming?

4. What would help them understand?

What evidence, context, or explanation would make the decision more relevant to their responsibilities?

5. What do you want them to do?

Do you need them to approve something, fund the program, help with implementation, reconsider a decision, or simply trust the process?

The point isn't to write a persuasive message for them. It's to clarify your own thinking.

Sometimes the problem becomes much easier to approach once you've separated what you know about a person from what you're assuming about them.

6. Make the system the bad guy, not the characters

A lot of experimentation politics comes down to competing ideas and unclear decision-making.

Everyone has an opinion. Everyone wants their idea tested. Senior stakeholders have their own priorities. And suddenly, the experimentation team is stuck negotiating every single decision.

One way to reduce this friction is to create a transparent prioritization system.

Instead of making every decision a personal negotiation, agree on the criteria that determine which ideas move forward.

These might include:

• Business impact.

• Customer evidence.

• Implementation effort.

• Technical risk.

• Strategic alignment.

• Research supporting the idea.

• Expected test duration and traffic requirements.

The criteria should be discussed and agreed upon with the relevant stakeholders.

Then, when an idea doesn't get prioritized, the decision comes from a system everyone understands rather than from one person rejecting another person's idea.

People can disagree with a decision without feeling that the process was unfair.

What does this look like in practice?

Ryan shared an example from the Airtable experimentation management system we've used across our client projects.

When someone submits a test idea, they're asked to provide:

• The test description and background.

• The hypothesis.

• How the test will be measured.

• Where it will run and on which devices.

• Whether it's an iteration of a previous test.

• The resources required.

• The research supporting the idea.

One particularly useful field asks what research the test idea is based on.

Depending on the organization, this can be a required or optional field. But even asking the question helps build the habit of connecting proposed solutions to actual observations.

The prioritization score can then incorporate the amount and diversity of research supporting an idea.

For example, an idea supported by analytics, customer survey responses, and user testing may receive more weight than an idea based on a single observation.

This helps encourage a shift from "I think we should test this" to "Here's the problem we're trying to solve, and here's the evidence that it exists."

Don't forget to account for test duration

Another useful consideration is how long an experiment is expected to run.

A promising idea might require ten weeks to reach the necessary sample size. That matters when you're comparing it with another opportunity that could be tested in a much shorter period.

Prioritization shouldn't just be about which idea sounds most exciting. It should account for the resources and time required to get a reliable answer.

And one important caveat: a prioritization system isn't going to eliminate all politics.

Sometimes a senior executive's preferred idea will still get an extra point because that's simply the reality of working inside an organization.

You don't necessarily have to fix everything overnight. Sometimes the best approach is to meet people where they are and improve the system gradually.

7. Stop solutionizing. Start with the problem.

During the discussion, we also talked about what happens when organizations have too many ideas and not enough meaningful prioritization.

Ryan pointed out that an endless stream of test ideas can be a symptom of solutionizing: coming up with solutions before identifying and validating the problem they're supposed to solve.

Someone suggests changing a headline, moving a button, adding an arrow, or redesigning a section.

But what customer problem are we actually trying to address?

Before discussing solutions, try asking:

• What problem is this intended to solve?

• Where did we observe this problem?

• What evidence do we have that customers experience it?

• How many customers might be affected?

• Is it significant enough to justify solving?

This is where qualitative research becomes particularly useful.

Instead of starting with a proposed design change, start by understanding what customers are struggling with.

Then connect those observations with other sources of evidence, such as analytics, surveys, and user testing.

The more you can validate the problem before investing in a solution, the better positioned you are to prioritize meaningful opportunities.

And yes, this can be uncomfortable. People can be surprisingly resistant to explaining the problem behind their proposed solution.

The trick is to ask questions without making people feel like they're being interrogated.

Being right isn't particularly useful if you've made everyone defensive in the process.

8. Making the value of experimentation visible: Revenue impact without overselling it

Eventually, we have to talk about money.

You can have a fantastic experimentation process, build a culture of learning, and help teams make better decisions.

But if the organization is investing resources in your program, people will want to understand the financial impact.

This was Ryan's part of the workshop, where he walked through the thinking behind our Revenue Impact Calculator.

The central idea is to forecast the revenue impact of experiments without making unrealistic promises.

Don't multiply one winning test by your annual revenue

A common mistake in CRO reporting is taking a winning experiment's observed conversion uplift and applying it directly to annual revenue.

For example:

• An experiment shows a 14% conversion uplift.

• The website generates $1 million in annual revenue.

• The CRO report claims the experiment generated an additional $140,000.

The problem is that this assumes the observed uplift will persist indefinitely and apply consistently across all future traffic.

That's a pretty big assumption.

It can also damage your credibility if the promised impact never materializes.

Ryan's recommendation is to be conservative with these estimates. Your credibility is one of the most important resources you have as the person responsible for an experimentation program.

It's better to understate the impact than to make promises you can't defend.

Account for the winner's curse

The first adjustment is an initial reduction to the observed effect.

When you select experiments that have reached a statistical significance threshold, their observed effects tend to be exaggerated estimates of the true effect. This is often referred to as the winner's curse.

The calculator uses an initial 10% reduction by default to account for this.

The exact adjustment can vary depending on your methodology and assumptions, but the principle is to avoid treating the observed uplift as the guaranteed real-world effect.

Account for impact decay

Another important assumption is that the impact of a winning experiment won't necessarily last forever.

If you publish a winning variant and then do nothing else, its performance advantage may diminish over time.

The calculator uses a conservative default model:

• Hold the impact steady for four months.

• Apply a 20% monthly linear decay afterward.

• The modeled impact eventually reaches zero.

This isn't presented as a universal scientific law. It's a practical forecasting assumption.

If your organization has the resources to measure actual long-term effects, for example through holdout groups, you can use that evidence to calibrate the model.

The goal is to make your assumptions explicit and improve them as better data becomes available.

Account for rollout coverage

Another consideration is how much of your audience actually receives the winning implementation.

Perhaps you tested on desktop and mobile but only rolled out the winner on mobile. Or maybe you tested in one region and are planning to roll it out elsewhere.

The rollout coverage adjustment helps account for the portion of users who will actually benefit from the change.

Be careful when applying results to audiences that weren't included in the original experiment. The impact there is less certain.

Include losses prevented, when appropriate

Experimentation isn't just about the revenue generated by winning ideas.

It can also prevent the organization from implementing changes that would have made performance worse.

The calculator includes an option to account for losses prevented by experiments.

But whether you include this depends on what you're comparing your experimentation program against.

If the alternative is implementing every proposed idea without testing, then preventing harmful changes is a meaningful part of the value.

If the alternative is doing nothing, you can't necessarily claim that you've saved money by preventing an idea that nobody would have implemented anyway.

The comparison matters.

Be clear about what your experimentation program is being compared against.

Keep the assumptions visible

The calculator brings these different factors together into a revenue impact forecast.

You can account for:

• Experiment start and end dates.

• Control and variant users.

• Revenue in control and variant.

• The delay between stopping a test and implementing the winner.

• Rollout coverage.

• Initial effect reduction.

• The duration of consistent impact.

• Monthly decay.

• Losses prevented.

For lead-generation businesses, you'll also need an agreed value for a lead so that lead volume can be translated into an estimated financial impact.

The point isn't to create a perfect prediction of the future. It's to create a transparent, defensible model that helps communicate the potential value of experimentation.

9. A few questions from the workshop

We also had some great questions from attendees. A few recurring challenges are worth highlighting.

"What if I have too many stakeholders pushing ideas, and not enough understanding of statistics?"

This is almost the opposite of having stakeholders who don't care about experimentation.

Having lots of people interested can be great, but it can also create chaos if everyone expects their ideas to move straight into the testing pipeline.

A few things to consider:

• Introduce a transparent prioritization system.

• Include research quality and evidence in the prioritization criteria.

• Make expected test duration and sample size visible.

• Encourage stakeholders to explain the customer problem behind their proposed solution.

• Use pre-test calculations to explain why certain experiments need more time or aren't feasible with current traffic.

The goal isn't to discourage people from contributing ideas. It's to give those ideas a consistent way of being evaluated.

"What should happen when traffic or spend is reduced and conversion becomes the priority?"

Less traffic means fewer opportunities to reach the sample sizes needed for reliable experiments.

That makes prioritization even more important.

Rather than trying to maintain the same testing volume with fewer resources, become more deliberate about what you test.

Focus on:

• Better research.

• More meaningful customer problems.

• Opportunities with stronger business impact.

• Ideas supported by multiple sources of evidence.

• Avoiding experiments that require significant traffic but address relatively minor opportunities.

When resources are limited, solving the right problems matters even more.

"What if my role has become unclear because leadership keeps changing?"

One attendee described joining an organization in a CRO-focused role, only for the position to gradually shift toward experimentation program management. Their director left shortly after they joined, and it took several months for a new VP to arrive.

This created uncertainty around their responsibilities, the experimentation model, and how they should work with product managers.

Our advice was to address the ambiguity directly.

Talk to the new leadership and ask how they see experimentation fitting into the organization. Speak with the other teams involved and understand what would make their work easier.

You can prepare a point of view, but ultimately, role expectations and organizational priorities need to be clarified with the people responsible for setting them.

And, just as in our opening client story, identifying a new internal champion can make a significant difference.

"How do you deal with stakeholders who insist their idea has more potential?"

Ask them why.

Not in an accusatory way, but genuinely: what's the reasoning behind the idea? What problem does it address? What evidence supports it?

Sometimes the conversation itself reveals that the idea is worth exploring. Other times, it becomes clear that the proposal is based mostly on someone's personal preference.

The important thing is to create a process where ideas can be challenged without turning every disagreement into a personal conflict.

10. The bigger lesson: Experimentation is an organizational capability

If there's one thing we hope people took away from this workshop, it's that experimentation doesn't happen in a vacuum.

It happens inside organizations full of people with different responsibilities, incentives, pressures, assumptions, and definitions of success.

You can have the best testing tool, the cleanest analytics implementation, and a very clever hypothesis.

But if nobody understands the purpose of the program, if stakeholders don't trust the evidence, or if people are afraid of negative results, you're going to struggle.

A healthy experimentation program needs more than technical competence.

It needs:

• Clear ownership and internal champions.

• Shared expectations about how results are interpreted.

• Psychological safety around learning from unsuccessful tests.

• Communication that considers different stakeholder perspectives.

• Transparent prioritization.

• Research that starts with actual customer problems.

• A credible way to communicate business impact.

And perhaps most importantly, it needs people to understand that experimentation isn't about proving someone wrong.

It's about making better decisions together.

You don't need everyone to love every experiment. You need people to trust the process enough to keep learning.

Want to explore the practical side?

• Experimentation process: We shared our Airtable-based experimentation management system during the workshop. It's designed to connect test ideas, research, prioritization, metrics, and program operations.

• Revenue impact calculator: helps forecast the potential financial impact of experiments while accounting for conservative assumptions such as effect reduction, decay, and rollout coverage.

• Katsed: The calculator is also available within Katsed, where experiment data can be connected through BigQuery and reports can be saved and shared.

• Pre-test calculator: understand what kind of MDE, sample and timeframe you need to run your tests.

• Ryan's article on How to Forecast A/B Test Revenue Impact Without Overselling It

If you're navigating your own CRO stakeholder circus, we'd love to hear about it. We're always happy to compare notes.

Catch the workshop recording below: