Maybe Facebook is not so bad

Publication bias and incentives

If you asked a random person in America if using social media is bad for you, I expect most would say it is. This poll from 2020 found 64% think that social media has a negative effect on society. I expect the numbers would be more negative today. Apart from the negative rhetoric, one hears, from time to time, about scientific studies that prove that Facebook and other social sites are bad for teenagers (adults too, but mostly teenagers). The studies mostly focus on Facebook, Instagram and Twitter. I have not seen any study showing that LinkedIn use results in overuse of corporate jargon or an insatiable desire to collect endorsements (but I would love to read that one!)

Stuart Ritchie at inews looked into the most commonly cited studies and was not impressed with the conslusions (ungated link here). There was generally two problems with the studies:

Multiple Comparisons

The first study he writes about is the “Facebook arrival” study. The study looked at self-reported well being and how that varied as Facebook rolled out across college campuses from 2004-2006. These types of studies are interesting because they create “natural experiments”. Mark Zuckerberg was not trying to run an experiment when he was rolling out access. He just couldn’t handle the demand of launching everywhere at once. But the result was that researchers could look at before-and-after Facebook at different places at different times to see what the CAUSATION effect was.

One risk of these studies (and any open-ended study that is not pre-registered) is that false positives are possible. If the mental health survey had 100 questions, some of those questions will vary significantly pre- and post-Facebook just from random chance. Researchers can correct for this with specific statistical tests. But these tests was NOT done on the Facebook Arrival study. When the results are subject to the “multiple comparisons” test, there is no longer any statistically significant difference pre- and post-Facebook.

The second study Richie examines has the same problem. “Deactivating Facebook” was an experimental study that asked the test group to shut down their Facebook account for four weeks, and then measured their self-reported well being vs a control group who was allowed to continue to use social media. This study also looked at dozens of mental health questions. When subjected to the multiple-comparison’s test only one question remained statistically significant: “How lonely are you?”, and Richie is not even convinced that question would remain significant if he knew all of the statistical tests that the team ran. When the researchers tested for impact based on the amount of time people used Facebook even the loneliness difference disappeared. The team also sent daily text messages to the test and control groups asking about things like happiness. No differences were found between the two groups.

Broadband Rollout

The second group of studies Richie looked at used the gradual rollout of broadband to see what effect faster and more frequent internet use had on well-being. The first problem with these studies is that it is only measuring social media in so far as people need internet to access social media. The stronger claim on these studies would be the effect of the internet on mental health rather than the effect of social media specifically. This is especially true given that in many cases (and studies) broadband rolled out before Facebook even existed (I somehow doubt Facebook was responsible for a well being drop in 2003. Say what you will about Mark, his pivot to AR is not a pivot to time travel).

Some more modern studies of this ilk fall into the multiple comparisons problem. Some studies find an decrease in “severe problems for teenage girls” but not “moderate problems for teen girls” or any problems at all for boys, or any problems for older girls. A study in the UK found a drop in performance for “age 16 GCSE exams”, but an increase in performance for “age 10 SAT exams”.

What I think is really going on is that researchers are incentivized to find something. Both because a result that says Facebook is bad will get more media reach, but also because a finding that Facebook is bad is a finding of SOMETHING. Running a study where the conslusion is “there were no statistically significant results in any direction”, is not great for anyone involved (except maybe Facebook).

The studies that show Facebook doesn’t affect our mental health one way or the other are sitting on the shelf. No one wants to read those.

Keep it simple,

Edward