When the players are the referees
Good morning everyone,
Last week, I pushed back a post on “Joe Rogan and podcast advertising” so that I could provide some immediate analysis on the ways that brands were responding to protests across the country. Bad news for those of you waiting to hear my thoughts on Rogan and Spotify — I am delaying that article one more week so I can share my perspectives about a startling change in public health policy.
The podcast-themed post will be released next week (assuming, of course, that the next few days unfold without any more urgent stories). May we all live in less interesting times.
—Edward
When the players are the referees
For the past few months, the majority of citizens have trusted the expert advice of epidemiologists. To minimize the risk of spreading COVID-19, we (mostly) followed instructions to “social distance” ourselves. We closed restaurants, theatres, and churches. We restricted access to parks, beaches, and sports fields. Basically, we arranged our lives to avoid any places where we might encounter people — especially large groups of people.
All of that changed last week.
Hundreds of thousands of people gathered in the streets to demonstrate for racial equality and the Black Lives Matter movement. You might expect epidemiologists to raise COVID-related concerns about the mass congregations of people, but 1288 public health advocates signed an open letter supporting the protests. Here are some key sections from the letter
White supremacy is a lethal public health issue that predates and contributes to COVID-19. … Black people are also more likely to develop COVID-19. Black people with COVID-19 are diagnosed later in the disease course and have a higher rate of hospitalization, mechanical ventilation, and death. COVID-19 among Black patients is yet another lethal manifestation of white supremacy. In addressing demonstrations against white supremacy, our first statement must be one of unwavering support for those who would dismantle, uproot, or reform racist institutions.
Staying at home, social distancing, and public masking are effective at minimizing the spread of COVID-19. To the extent possible, we support the application of these public health best practices during demonstrations that call attention to the pervasive lethal force of white supremacy. However, as public health advocates, we do not condemn these gatherings as risky for COVID-19 transmission. We support them as vital to the national public health and to the threatened health specifically of Black people in the United States. We can show that support by facilitating safest protesting practices without detracting from demonstrators’ ability to gather and demand change. This should not be confused with a permissive stance on all gatherings, particularly protests against stay-home orders.Those actions not only oppose public health interventions, but are also rooted in white nationalism and run contrary to respect for Black lives. Protests against systemic racism, which fosters the disproportionate burden of COVID-19 on Black communities and also perpetuates police violence, must be supported. [Emphasis mine]
Written with the authority as “public health advocates,” the letter clearly states that anti-racist demonstrations “must be supported.” In contrast, protests against stay-at-home orders “not only oppose public health interventions, but are also rooted in white nationalism.”
Let’s think about the implications of this message. The letter asserts that the risk of spreading COVID-19 at protests is a LESS urgent public health crisis than the structural discrimination facing Black people in the American healthcare system. This is a complicated argument, but one that experts — like maybe 1288 of them — could reasonably endorse.
But to reject the validity of other protests suggests a type of MORAL authority that sits outside the expertise of medical researchers.
Public health professionals are meant to serve as apolitical referees who help us understand the rules of the game. But their letter clearly shows their desire to participate as players. The signatories, both individually and collectively, are using their official positions to advance their own values – only some of which are directly tied to immediate health concerns.
When the players are the referees, we lose confidence in the legitimacy of their information. Public health professionals are not the only group to blur the objectivity of their perspectives; you can find examples of this behavior in any industry or discipline.
Scott Alexander — a psychiatrist on the US West Coast (and one of my favourite bloggers) — wrote a post about a particularly egregious case of self-interested research. Here’s what he wrote about parapsychologists (people who believe in telepathy, telekinesis, etc.):
Parapsychologists are able to produce experimental evidence for psychic phenomena about as easily as normal scientists are able to produce such evidence for normal, non-psychic phenomena.
Are you surprised? You might be even more astonished when you learn how well those studies were designed. Alexander measured the methodological “quality” of the parapsychology research projects, by evaluating them in relation to the best practices from academic publications (factors like large sample sizes, independent replication, systematic meta-analysis, etc.). After reviewing many parapsychology studies, Alexander discovered that their papers were generally BETTER at proving their theories than most scholarly publications from the “legitimate” sciences.
So, what should we think and believe? Should we start trying to develop our sense of ESP? Or maybe we should abandon our confidence in the theory of evolution? Perhaps we should respect the validity of some scientific research — but only the projects we believe; we can ignore everything else.
In the case of the parapsychology research, here is the most likely explanation for their ability to consistently prove their claims: the researchers were motivated (by their own hopes) to identify parapsychological evidence, even when faced with “normal” situations.
When people want to believe something, they can rationalize almost anything. Like the idea that mass protests are not “risky for COVID-19 transmission,” as stated in the letter from public health advocates.
Ask the experts
In case you think I am unfairly pointing fingers at public health professionals for supporting the protestors, let’s flip things around and look at the police. What is the “right” amount of money needed for a city to fund a police department? (This question is no longer hypothetical. Minneapolis city councillors announced their intention to dismantle the municipal police force; both New York and Los Angeles are taking unprecedented steps to reduce funding for their police departments).
Because most police budgets are allocated toward salaries, we can modify our question to, “How many police officers does a city need”? When faced with a public policy dilemma, we should consult expert perspectives rather than personal opinions. But who are the experts when it comes to determining the optimal number of police officers for a region? In most cases, consultants for police issues are drawn from the ranks of law enforcement agencies (both current and former members).
What do you think happens when you ask a police officer, “how many police officers are needed”? The answer is always the same: “we need more.” In a June 5 post on Mother Jones, Kevin Drum looked into the relationship between police staffing and crime rates since 1992:
The violent crime rate has fallen by half since its peak in 1992, but the number of police officers per capita has stayed nearly flat. This divergence is even more dramatic in places like Los Angeles and New York City, where the crime rate has fallen by about 75 percent since 1992.
One might argue that crime levels have decreased because we maintained the number of police officers. However, in the regions that did reduce the size of police departments, crime rates still continued to drop; in other words, there is no clear correlation between police staffing levels and crime reductions. (Analysts suggest a range of reasons to explain why crime rates have plummeted since 1992).
In a similar situation, the number of total fires in the US has dropped by more than 50% over the last 30 years. And yet, cities increased the number of fire fighters by more than 50% over the same period of time.
We ask police for advice on policing and we ask firefighters for advice on fighting fires.
We also trust public health officials to provide expert perspectives on public health guidelines.
And, unsurprisingly, experts generally believe not only in the importance of their own field, but also the necessity of greater resources, authority, and autonomy.
Self-aggrandizement is especially pervasive in the marketing world.
In my own experience with conversion rate optimization teams, they can show off dozens of experiments that verify their ability to boost conversion rates by 10% or more. The teams claim that all their strategies have been “proven” with 95% confidence on randomized A/B tests. And yet, what happens when their ideas are implemented? Inevitably, they don’t produce the promised lift.
The results are even worse when you look at the total effect. Adding up ALL the conversion rate rollouts over a year, it’s not uncommon to see that their total “estimated” impact on the business is a 10% or more lift in total conversion of website traffic into leads or revenue. But when you look at actual conversion rate change, you discover that it has barely moved — at least nowhere near a 10% change. When you press the team about this outcome, they usually respond with something along the lines of, “Yes, but there are a lot of other factors that drive conversion rate. Maybe your marketing teams are buying lower-converting traffic. Or maybe our competitors or customers or products have changed. It is clear from our scientific tests that if we had not made these changes, the conversion rate would have been even lower!”
In these cases, marketers rely on the same logic used by police experts: yes, the crime rate has declined significantly over the last 30 years, BUT ONLY because we maintained the same number of police officers.
Car mechanics
What makes a good car mechanic? Presumably someone who excels at fixing cars.
But how do potential customers find a mechanic who excels at fixing cars? They could ask for referrals, but that assumes previous customers could identify the quality of the mechanic’s work. If someone asked me to evaluate a mechanic, I’m not in any position to judge. Sure, I might be able to identify a bad mechanic — someone who could not fix my car (or left my car in even worse shape). But I have no way to determine whether my mechanic’s repair skills are average or exceptional.
I CAN, though, judge a car mechanic based on his customer service and my general impression of his integrity. As such, I would be more likely to recommend a personable-but-mediocre mechanic over an exceptional-but-cantankerous one.
Let’s assume that my perceptions of mechanics are typical — what should auto repair shops do with this information? If the owner of the shop wants to run a successful business, the first step is ensuring that all of the mechanics can fix cars without obvious screwups (the kind that customers like me could spot). But then what? Where does the owner invest next? Does he find the best mechanics in the city and pay them top salaries to join the shop? That plan sounds like a good way to go out of business, because it increases operating costs without bringing in any more revenue. No, the best way to grow the auto repair shop has less to do with leaky transmissions and more to do with sleek communications — expanded marketing campaigns to recruit new clients and enhanced customer service to maintain the current ones.
The car mechanic is not only a player in the game, but also the referee who tells you what you should be doing to your car.
Marketing vendors face the same challenge.
Most companies can identify bad marketing, but they can’t tell the difference between good marketing and great marketing — that’s why they hire external marketing vendors. The vendor provides advice about marketing strategies and then builds reports to analyze the results. If a vendor wants to grow their business, they do not need to focus on developing the effectiveness of their marketing tactics. Instead, they need to improve their customer sales (to grow their funnel) and customer service (to decrease churn).
Marketing vendors are players and the referee.
In most companies, CMOs both lead the marketing team and create the metrics that report on the effectiveness of the marketing team. Plus, the CMOs are the ones to interpret those reports for the CEO. CMOs are the player and the referee.
And as I mentioned earlier, when the players are the referees, we can no longer trust the legitimacy of their recommendations.
There is no simple solution for this problem. At best, we can try to focus on the objective facts of any situation. But that’s easier said than done, because our perceptions are shaped by our beliefs, biases, and interests. The parapsychology researchers aren’t pulling some kind of performance art prank — they genuinely believe in their ability to scientifically prove the presence of paranormal activity.
Just like police officers who believe that the streets need more police. Just like firefighters who believe that cities need more firefighters. Just like progressive public health advocates who believe that protests against racial oppression are more important than containing a deadly pandemic.
Just like marketing vendors who believe your company will see greater results by using their retargeting campaign. The takeaway message: trusting people’s best intentions (even your own) will not solve the problem.
There is one strategy that companies can implement: set up individuals (or teams) with competing incentives. When I joined Expedia, my job duties were a little nebulous. I oversaw the revitalization of two dying loyalty programs and a credit card business on autopilot. Ultimately, though, I believe my most valuable contributions related to an assessment of our marketing channels. Because I was not incentivized to reach any particular outcome (that is, I was not personally rewarded for expanding or contracting the size or scale of marketing campaigns), I was able to maintain a sense of objectivity.
As part of that process, I noticed the incredible open and clickthrough rates for emails with subject lines like “Your itinerary has changed.” You can understand why — if you have an upcoming trip, you want to know about any disruptions to your plans.
At the time, Expedia used a simple “30 days last touch” attribution methodology. Whenever a customer purchased something from Expedia, credit would be given to the last channel clicked on by the customer. Because of the facts that (1) people tended to open itinerary-related emails and (2) frequent travellers received a higher number of those messages, the email team often received credit as the last channel touched within the previous 30 days. In essence, the email team was “tricking” high value, frequent-travelling customers to open and click on their messages — all in a move to gain credit for future sales that would probably have happened anyway.
The email team was entirely forthcoming about the situation; in fact, they were proud of the reputation of their “most effective” emails. No one on the team recognized the lack of real impact. (Or even their negative effects — they looked for any opportunity to send those itinerary notifications AND they hide the details behind an additional click. They designed a terrible customer experience, all for the purposes of goosing their attributed numbers.)
I only discovered this issue because the CMO directed me to assess all of our marketing channels. I had no personal incentive to increase the size or budget of any particular channel. Had the CMO waited to learn about the “30 days last touch” glitch from the email leader — a person WITH an incentive to increase their department’s size and budget — then the CMO would have remained in the dark for much, much longer.
I am lucky to have spent most of my career sitting outside the traditional reporting structures. Plus, I have a strong desire to “find the truth” in any situation. I don’t always succeed, in part because I — like everyone — allow my own biases to shape how I interpret the evidence.
Keep it simple and stay safe,
Edward
If you are enjoying Marketing BS, the best thing you can do is share it with friends and recommend they subscribe. The next best thing you can do is comment on this piece or click the heart (which makes this article more visible for other readers).
Edward Nevraumont is a Senior Advisor with Warburg Pincus. The former CMO of General Assembly and A Place for Mom, Edward previously worked at Expedia and McKinsey & Company. For more information, including details about his latest book, check out Marketing BS.