Marking BS: Book Review for Don't Trust Your Gut
Hi everyone,
In 2017, former Google data scientist Seth Stephens-Davidowitz landed on the New York Times Best Seller list with Everybody Lies: Big Data, New Data, And What The Internet Can Tell Us About Who We Really Are.
He’s continued to dig deeper into the world of data-driven thinking with his new book, Don't Trust Your Gut: Using Data to Get What You Really Want in Life.
In today’s post, I’m going to share my perspectives about the first part of the book — where Stephens-Davidowitz writes about dating, relationships, and raising kids. It is more relevant to data-driven marketers than the subject areas may suggest. In a future post, I’ll offer some comments about the second half of the book, which looks at career advice and happiness.
Even if you’ve never read Stephens-Davidowitz (or don’t expect to pick up the book soon), there’s lots to think about in today’s post.
—Edward
Trusting data
I love data. I built my career on working with numbers to recognize issues and solve problems. I never would have become a marketer back when the profession was primarily focused on qualitative storytelling and stakeholder management. But I also appreciate the need to balance data with intuition. That’s why I often repeat the following two quotes:
The thing I have noticed is when the anecdotes and the data disagree, the anecdotes are usually right. There’s something wrong with the way you are measuring it. —Jeff Bezos
Not everything that counts can be counted, and not everything that can be counted counts. (often paraphrased as “What you can measure is not always what matters”) —Albert Einstein
In my experience, Bezos and Einstein are correct. When data gives you a very counterintuitive answer, the bar for believing that data should be correspondingly high.
With these ideas in mind, I was excited to read Seth Stephens-Davidowitz’s latest book, Don't Trust Your Gut: Using Data to Get What You Really Want in Life.
Stephens-Davidowitz positions the book as a self-help guide that puts the data first. Nothing new here — most self-help books CLAIM to ground their ideas in data. In reality, of course, many authors start writing their book project having already reached a set of firm conclusions about the subject. We’ve all read books where the author tries to back up their “big idea” with a collection of anecdotal stories and cherry-picked research studies.
With Don’t Trust Your Gut, Stephens-Davidowitz attempted to reverse the conventional process for writing self-help books. Instead of building arguments to support his own long-held viewpoints, he started by collecting the most important questions about life — such as how to find a mate, how to raise children, how to be successful in your career, how to be happy, etc. With that list of big questions in hand, he searched for the most compelling data and best available research. Stephens-Davidowitz launched his project without any preconceived perspectives; he was equally ready to be comforted or surprised by what he discovered. Moreover, he waited until “science” gave him the “right answers” before adding any anecdotes or examples to illustrate the point.
Many self-help books that combine research and anecdotes to prove a counter-intuitive thesis have been discredited, especially when they rely on a limited number of studies (or even a single study!). In contrast, Stephens-Davidowitz not only explains his conclusions, but he also traces his research journey — which studies he used, WHY he selected those studies, and WHY he discounted other studies. The end result is a higher level of confidence in the conclusions he presents (or, for places where a reader might dissent, a clearer understanding of the author’s thought process).
But “just believe the data” has its own set of problems (as articulated by Misters Bezos and Einstein). I want to highlight how Stephens-Davidowitz’s reliance on data may have blurred his analysis about what is really happening. Whenever our personal intuition disagrees with data sets, we should dive a little deeper to understand where the data may have gone wrong. And that’s a core lesson every marketer trying to run their business “by the numbers” should be careful to remember.
Dating and relationships
The book’s first major conclusions deal with dating. According to Stephens-Davidowitz, the data confirms that our romantic preferences for a mate are exactly what you would expect: someone who is attractive, who shares your common interests, and who comes from a similar background. That last point is surprising only for the extreme degree to which it matters; Stephens-Davidowitz references data that shows people are not only more likely to date people from the same religion or who attended the same school, but also people who share the same initials!
Other conclusions match our general assumptions. For instance, men are especially attracted to women who are younger (18-22 years old) — regardless of how old the men are. Women are partial to tall men with high incomes.
Plus, Stephens-Davidowitz cites some research about online dating sites and the shockingly significant impact of a man’s occupation on his romantic desirability.
For example, all else being equal, males can expect significantly more romantic attention if they are firefighters than if they are waiters.
It turns out, sometimes a switch to a different, more attractive occupation can make a male more desirable than a large salary increase. For example, the data from online dating sites suggests that a man who earned $ 60K in the hospitality industry would become more desirable, on average, if he earned the same amount as a firefighter than if he stayed in the same industry but upped his salary to $200K. In other words, a male firefighter who earns $60K tends to be more attractive to the average heterosexual woman than a hospitality worker who earns $200K.
I have no doubts about the accuracy of the data that Stephens-Davidowitz referenced. Moreover, I’m not skeptical of that fact that women using online dating platforms are about as likely to message a firefighter earning $60K as a hospitality worker making $200K.
But WHY might this happen — especially when other research concluded that women are partial to men with higher incomes? What’s the explanation? Could it be that women just REALLY like firefighters? That’s the conclusion Stephens-Davidowitz reaches: “the data suggests that having a cool job is frequently more attractive than having a boring, but lucrative, job.”
On this point, I think Stephens-Davidowitz is wrong. There is another reasonable explanation: men regularly lie on their online dating site profiles. And women are smart enough to know that men lie. One of the easiest things to lie about is income. Lying about your occupation, on the other hand, is much riskier. A few minutes of casual conversation — or just a few seconds of scanning social media accounts — can expose the lie.
And so, occupation can function as a proxy for income. For a woman swiping through dating profiles, a firefighter claiming to make $60,000 per year seems authentic (especially because firefighter salary ranges are easily searchable for any city). But the profile for a “hospitality worker” who claims to make $200,000 per year? They’re probably telling the truth about working in the hospitality industry, but maybe they exaggerated their income by a little (or a lot…).
Ultimately, then, the conclusion that “women really like firefighters” might be an example of blindly trusting the data. A more nuanced take might consider that (1) although women generally prefer men with higher incomes, (2) occupation is a more trustworthy way to gauge income than believing what men wrote on their online dating profile.
On a related question: how much do average women care about their prospective partner’s income, rather than social class and status? I’ll bet most upper-class women would be more interested in dating a poor professor than a rich garbage collector. Just as occupation functions as a proxy for income, it could also be possible that income works as a proxy for other attributes that women are looking to find. I wonder what the data would say if occupations were all tagged by educational requirements (or some similar marker) to see how much they explain message response rates?
Stephens-Davidowitz also finds that dating profiles with certain names get higher response rates than others. My hypothesis: names can sometimes provide a clear signal of social class (and income). As with occupation, I’ll wager that some names are better predictors of income than stated income.
(I also suspect that dating sites see a fair amount of counter-signaling. Men with very low salaries are probably better off omitting any income information on their profiles, but I expect that men with very high salaries would also avoid listing their income. My Twitter bio states that I am a former CMO and lists the companies I used to work for. In contrast, Malcolm Gladwell’s bio consists of just two words: “Skinny Canadian.” If Gladwell created an account on Bumble, I assume he wouldn’t list his income — or the fact that he’s a bestselling author. Nevertheless, I expect his message response rate would be just fine).
After the section on dating results, Stephens-Davidowitz identifies some even more shocking conclusions: of all the stuff that we cared about when finding a mate, none of it is predictive of happiness within a relationship.
According to my read of the research of Joel and her coauthors, the best three questions to figure out whether John is happy with Sally would have nothing to do with Sally; in fact, all would be related to John. The best questions to predict John’s happiness with Sally would look something like these:
“John, were you satisfied with your life before you met Sally?”
“John, were you free from depression before you met Sally?”
“John, did you have a positive affect before you met Sally?”
I’m not surprised that people who are happy before they get involved with someone are more likely to be happy after they get involved with someone. That idea seems neither surprising nor insightful. But Stephens-Davidowitz spends a chapter trying to take that piece of research — combined with the lack of any counter research — and build a case for an incredible epiphany about human relationships.
What I think he really misses here is that absence of evidence is not evidence of absence.
To recap, Stephens-Davidowitz found a study that could not identity significant impacts of any of the major partner traits on happiness. But let’s be clear about what the research did NOT confirm. The study did NOT confirm that your choice of partner has no impact on your relationship happiness. That would, of course, be a very provocative claim — does anyone who’s ever been in a relationship really believe that could be true?!
Likewise, I’m also skeptical of other conclusions that Stephens-Davidowitz drew from the research. He notes that the specific traits examined in the study — race, religion, height, occupation, attractiveness, previous marital status, sexual tastes, similarity — do not matter for relationship happiness. Maybe. Or maybe the lack of connection can be attributed to limitations with the data collection and/or how they analyzed the results.
I’m not convinced the researchers found anything profound. Instead, they likely found NOTHING, and then tried to spin their work into something profound. There is a difference, obviously, but one we don’t mention enough in data-driven marketing.
Here is a possible alternative explanation on why they found nothing: selection effect.
Selection effect
In the NBA, height is not a predictor of how many points a player will score in a game. But that fact does NOT mean height is irrelevant for people dreaming about an NBA career — in fact, height matters a LOT. Tall people are far more likely to reach the NBA than short players. If you are a healthy young male in America — and you happen to be 7’8” — your chance of playing professional basketball is about 50%. On the flip side, players shorter than 6’ rarely crack team rosters. In the 2021 NBA season, there were 529 players and only nine of them were shorter than the 6’ mark. (For reference, only about 15% of American men are taller than 6’). Bottom line: if you are 5’9” and land a spot on an NBA team, it’s because you are really good at basketball — your elite skills compensate for a huge disadvantage in height.
What is true in basketball might explain exactly what we are seeing in relationships. What if things like attractiveness, intelligence, income, and other quantified metrics matter in relationship happiness just as much as height matters in the NBA? Now imagine there are other characteristics that also matter. Imagine these characteristics were not as easy to quantify: things like generosity, empathy, flexibility, organization, conscientiousness, extraversion, friendship circles, great brother-in-laws, etc. It’s not a stretch to think that all of those things could affect relationship happiness, too.
Now imagine that everyone who is dating is trying to find a mate that makes them happy (hopefully that’s not hard to imagine!). The result is they find someone with a combination of traits that does well enough that they are willing to commit to be with them for life. Maybe they aren’t the best looking, but they are very kind. Maybe they don’t have the highest income, but they give great back rubs. Or maybe they aren’t that kind, but they are really hot. Or maybe they are kind of selfish, but they are heir to the Walmart fortune.
Suppose you find two heterosexual couples, where a man in one couple is attractive and the man in the other couple is ugly. If the women are both equally happy with their relationships, that does NOT mean that attractiveness is irrelevant. The women were not paired with the men randomly. Each of the women CHOSE to partner with their respective man. Attractiveness was one of an infinite number of things that went into the decision.
What I am saying is that maybe factors like attractiveness (or income or social class) are all just like height in the NBA. And our happiness is like the number of points a player can score. Those quantifiable traits matter, but they are traded off against all the things that we can’t measure. So, the final result is that the traits don’t seem to affect happiness (but do affect the chance of getting married). Just like height does not seem to affect points in NBA basketball (but does affect the chance of getting into the NBA).
We know what’s happening in basketball, so when we see the counterintuitive height conclusion we look deeper. The same thing should have happened here.
What is most damning about the study Stephens-Davidowitz used was that it could find traits that made people happy in a relationship, but it found no traits in a partner that could make you happy in a relationship. For those of you who have been in a relationship, take a second to think about that. One of the most important things that makes me happy in a relationship is knowing if my partner is happy.
So if researchers can predict traits that make someone happy, shouldn’t — at the very least — those same traits in the partner predict happiness?
If positive affect and lack of depression make men happy, shouldn’t women be happier when they are with men who display positive affect and lack of depression. The data suggests they aren’t. Why? This selection effect analysis is left as an exercise for the reader.
Raising children
About questions on raising children, Stephens-Davidowitz references Bryan Caplan’s 2011 book, Selfish Reasons to Have More Kids, which argues that “twin and adoption studies find that the long-run effects of parenting are shockingly small.” Research indicates that genetics and non-shared environments (i.e., siblings having different friends or teachers outside of the home) play a much more important role in determining how your kids will turn out. (After I read Selfish Reasons to Have More Kids, I was pretty convinced of these theories — and maybe it even persuaded me to have an extra child or two than I might have otherwise).
In one compelling anecdote of the book, Stephens-Davidowitz traces the paths of the three “Brothers Emanuel” who all achieved career success in very different fields. Rahm became Chief of Staff for Barack Obama and then Mayor of Chicago; Ari is CEO of Endeavor (the entertainment company that owns UFC); and Ezekiel is Vice Provost for Global Initiatives at the University of Pennsylvania. In addition to raising three boys, parents Benjamin and Marsha Emanuel also adopted a girl. Stephens-Davidowitz points out that while the brothers’ successes were undeniable, there was a problem with the lady (Hamilton reference. You’re welcome). A 1997 profile in The New York Times provides some further detail about the siblings:
Today, the brothers argue just as passionately about the role that environment and genetics played in the life of their sister, who in recent years has been on and off the welfare rolls that Rahm worked so hard to cut. … Intellectually, Shoshana developed normally — like her brothers, she graduated from New Trier, one of the most competitive high schools in the country — but… She had a difficult adolescence, and today Marsha Emanuel, at the age of 63, is raising Shoshana's two illegitimate children. (None of the Emanuels will talk about Shoshana in detail, and she declined to be interviewed for this article.)
Research is clear that genetics matters — even if genetics only has a small correlation with adult income. Perhaps, though, the limited correlation is explained by the reality that many people spend more effort pursuing status (and class) than they do income. Of the three Emanuel brothers, only one would be considered extremely wealthy — Ari, the CEO of an entertainment company. Rahm and Ezekiel, while doing just fine economically, chose career paths in politics and academia. Both, given their genetic characteristics, could likely have made a great deal more money by going into business like their brother, but that was not what they were optimizing for. Genetics likely predicts “ability to succeed” far more than it predicts income — which is just a subset of “success” for most people.
I can think of one place, though, to challenge the conclusion that “parenting and shared environment does not matter” — the lives of of very top performers. In the nature versus nurture debates, many people reference the story of the Polgar sisters, whose father wanted to test his theory that “geniuses are made, not born” (and found a wife who supported the plan to raise their children as an experiment). All three girls were homeschooled, with an obsessive focus on chess. As very young children, the Polgar sisters began winning chess tournaments against older competitors. As adults, all three siblings dominated the game. First-born Susan became a Grandmaster and the top-ranked female player. Middle sister Susan reached the level of International Master, famous for one of the best-ever tournament performances. And youngest sister Judit outshone them both by becoming the youngest player in history (male or female) to become a Grandmaster and the only woman to ever beat a reigning world champion (including Garry Kasparov).
Clearly the Polgar sisters were genetically talented. But I believe it’s also pretty clear they would not have become three of the top six female chess players in history without having been raised in the environment created by their father.
In Don't Trust Your Gut, Stephens-Davidowitz mentions another example of atypical parental influence: billionaire Charles Kushner and his son Jared. From all accounts, Jared is not particularly bright; he was accepted into Harvard after Charles donated millions to the school. After Jared graduated, his father handed him a real estate empire to control, which opened the door to meeting his future wife Ivanka Trump, which placed him in the position to influence the president. Stephens-Davidowitz brushes away the Kushner anecdote as proof that parenting choices are only impactful when they are extreme:
But the data suggests that the average parent—the one deciding between, say, how much to read to their kids, rather than how many millions to give to Harvard—has limited effects on a kid’s education and income.
If the overall effects of parenting are smaller than we expect, this suggests that the effects of individual parenting decisions are likely to be smaller than we expect. Think about it this way: if parents face thousands of decisions and the parents who make far better decisions only have kids who turn out some 26 percent more accomplished, each of the thousands of decisions, by itself, can’t make a large difference.
Here’s my concern with this conclusion: a research study’s inability to detect parental influence does NOT validate the conclusion that parenting choices have limited effects. Once again, the absence of evidence is not evidence of absence.
Even Bryan Caplan — a leading proponent of the “parenting has limited impact” belief —homeschooled two of his kids, helped them take AP classes in middle school, provided access to sit in university lectures and have casual discussions with GMU professors, and facilitated their production of a podcast where they interviewed leading historians (some of whom I expect chose to participate because of who their father was).
Genetics clearly matters a lot, but success also results from opportunities — and parents can exert a great deal of influence about which opportunities their children receive. You might not be able to get your kid to the Olympics for figure skating, but you can definitely take steps buy them skates or take them to a rink.
Neighborhood and non-familial adults
In one of Don’t Trust Your Gut’s most interesting sections, Stephens-Davidowitz describes a fascinating research project that looked at the impact of where children are raised:
[“Genius Grant” recipient Raj] Chetty and a team of researchers—including Nathaniel Hendren, Emmanuel Saez, and Patrick Kline—were given by the Internal Revenue Service de-identified and anonymized data on the complete universe of American taxpayers. Most important, by linking the tax records of children and their parents, Chetty and his team knew where people spent every year of their childhood—and how much they ended up earning as adults. If a kid spent the first five years of her life in Los Angeles and then the rest of her childhood in Denver, Chetty and his team knew that. They knew that not for a small sample of people; they knew it for the entire population of Americans. It was an extraordinary dataset in the hands of an extraordinary mind.
Some conclusions stand out: when families move from a “worse” to a “better” neighborhood, the outcomes for younger children improve more than they do for older kids. Stephens-Davidowitz makes a point of highlighting not only the intelligence of the lead researcher, but also the sheer quantity of data the researchers could access.
I agree that the scope of the project is impressive. Without the tax records of every single American over an extended number of years, how else could you tease apart the effects on siblings who happened to move, at different ages, from one city to another?
In fact, the scale of the research should make us question some earlier findings. Recall that other adoption studies showed no impact from a shared environment. A shared environment is mostly code for “parenting,” but it also means “where the kids are living.” So why does one (giant) study show neighborhood effects, but many other studies show no effect from all non-shared environment effects combined? When you find a discrepancy between research conclusions, you can often dismiss the single study as being an outlier that is unlikely to replicate. But in this case, the single study was far larger and more rigorous than all the others. As such, we can presume that all the other studies were too under-powered to detect the finding.
And if the shared environment matters when it comes to neighborhood, should we be so sure that it does NOT matter when it comes to parenting choices? If nothing else, we should expect that richer parents are more likely to live in nicer neighborhoods than poorer parents. But the earlier studies did not pick up on any impact from this at all. Maybe they didn’t find it because the studies were too small — or, more likely, maybe they just never looked for it. What other parenting choices did researchers not look for?
So WHY did neighborhoods matter? The study looked at that, too, and identified the top three characteristics that predicted a good neighborhood:
Percent of Residents Who Are College Graduates
Percent of Two-parent households
Percent of People Who Return Their Census Forms
Stephens-Davidowitz notes that “these three factors all relate to adults who live in that neighbourhood.” He suggests that the presence of strong role models — the non-familial neighbors — are responsible for the positive impact on the kids living in the area. He backs up that conclusion by citing other studies that show living around inventors makes it more likely that a child becomes an inventor.
Here is an alternative hypothesis: what if the impact from the other adults in the neighborhood is not on the kids directly, but rather on the parents of those kids? We already know that peer effects can be significant. Why do we assume that the effects of having responsible parents in the neighborhood will be on children rather than on the other parents? Maybe being surrounded by responsible, upstanding parents can influence people to become better parents themselves?
In contrast to the IRS data trove of the Chetty research, many earlier studies focused on things like whether a parent graduated college or smokes or spent time in prison — perhaps those elements were NOT the best proxies for “good parenting.” Maybe peer effects from being surrounded by other responsible, conscientious adults cause us as parents to be more responsible, more conscientious, AND BETTER PARENTS.
To recap:
We have one set of studies that shows no effect from parenting (or neighborhood) on outcomes.
We have another (more comprehensive) study that shows there is a SIGNIFICANT neighborhood effect, and that effect is driven by what other adults in the neighborhood are like.
We have numerous anecdotes that parenting seems to impact how kids turn out — at least at the highest performance levels.
As I noted in the introduction with the Bezos and Einstein quotes, we need to pay attention when the data and the anecdotes don’t line up. In these examples from Stephens-Davidowitz’s book, maybe it will turn out that the anecdotes are right and there was something wrong with the data. Or maybe we were just looking at the data the wrong way.
Keep it simple, and stay tuned for more comments about Don’t Trust Your Gut in the coming weeks,
Edward
Postnote
The pricing for Don’t Trust Your Gut is very smart. The hardcover is priced at a reasonable $23, but the publishers also released a paperback simultaneously and charged almost the same price ($19). In addition, there is also an audiobook for $23 (which is the same as the hardcover but high for an audiobook). They also put out the Kindle version for only $3. This pricing model allows profit maximization for those who care about format, while also offering a very low price for the Kindle to drive the book onto bestseller lists. I wonder why more books don’t do this, or if this is just the start of a trend. In any case, I like the idea and plan to do the same thing when I release Marketing BS.]
Edward Nevraumont is a Senior Advisor with Warburg Pincus. The former CMO of General Assembly and A Place for Mom, Edward previously worked at Expedia and McKinsey & Company. For more information, check out Marketing BS.
hi