BONUS MarketingBS - Follow-up and Links April 3rd, 2020
What I got wrong about false positives, face masks, and where to learn more
I usually land in your inbox every Tuesday morning with some sort of analysis (with pretty pictures and all my spelling mistakes fixed by my editor). That will continue. This is an experiment. This email is NOT edited. I received lots of comments on Tuesday's newsletter, so I thought I would provide some follow-up. Iam also including some links to other interesting related content (some sent by this community). Please let me know if this is valuable, and I should continue the practice from time to time, or stop it and leave your inbox alone for the other six-days a week.
Follow-up Stuff
One of the arguments I made on Tuesday was that false positives will overpower signal if the base rate is very low and the test is not very close to perfect. I used the example of Target attempting to identify pregnant women based on shopping behavior. I related that business example to what is happening with COVID testing - specifically COVID testing in Iceland, which found 50% of COVID-19 cases were asymptomatic. I showed that if accuracy was even 99% (i.e., false positives were only 1%), then you would expect a high degree of "no symptoms" just because those false positives would overpower the signal. I did not have data on what the actual false positive rate was for the COVID test in Iceland - just that it was done with Nasopharyngeal (nose) swabs that detected the RNA of the virus, and that these swab tests were less accurate than serology (blood) tests. For lack of a better choice, I looked at the accuracy and sensitivity of flu swab tests to see what the false positive rate in Iceland might look like.
A reader forwarded the email to his wife, who has a PhD in Medical Science and a background in biology research. She pointed out that the flu tests I used as a baseline are "immunoassays that detect the presence of viral proteins." I believe another word for that is “antigens”. There are two types of flu tests that use a nose swabs - antigens tests and RNA tests. She claims the antigen tests are, "Very very different to the coronavirus test that you are comparing them to, which detects viral RNA by RT-PCR." A better comparison would be to look at how accurate the flu swab tests are when used to detect RT-PCR (the gold standard) given the coronavirus tests aim at RNA and NOT antigens.
So I may just be wrong that these tests have a 5% or higher false positive rate (and a 70% accuracy rate).
That said, I still am not aware of good data on what the actual false positive and false negative rates are. I was unable to find specific numbers on the accuracy and sensitivity of flu RNA tests, but the CDC has a page that claims they are “low”, and that most false positives/false negatives happen when procedure is not followed (i.e., human error rather than test error). Note that the if the base rate in a sample (i.e., total coronavirus penetration into the population) is low enough, even a VERY low false positive rate could give a significant over-estimate of the asymptomatic cases.
This article in the WSJ yesterday questions the accuracy of the Coronavirus tests. Generally it focuses on the issues of false-negatives - people who had the virus, but were "cleared" by the test. In many ways this is the bigger concern. These people think they are clear of the virus, and go about their lives - and in the process infect other people. The article claims "health experts believe nearly one in three patients who are infected are nevertheless getting negative test results." but, "their estimates are based on their own experience in the absence of hard science... we can't be completely confident in how they will perform... he estimates the sensitivity of the tests is in the neighborhood of 70%". 70% sensitivity is very similar to what we see with rapid-flu antigen tests. But that does not tell us what is happening with false positives, and whether that rate is similar or different from the flu tests.
The end of the article briefly talks about false positives. There was a case in Florida of a baby testing positive to COVID-19, but when tested a second time was found negative. This may have been a case of a false positive, or it is possible that the baby cleared the virus out of its system in-between the tests. The doctor involved claimed, "It's really hard to have a false positive [on the type of tests used to detect coronavirus]"
If any reader knows more about the false positive rate of the various tests, please let me know and I will share it back with the group. Every one of these recent newsletters have made me nervous as I am writing about things that are definitely “not in my lane”. I am humbled when someone points out when I make an error. Thank you.
Links from readers:
In four of the last five weeks I have written something related to the Coronavirus:
Pandemics and Plagiarization (how in copying others you need to look beyond the obvious features)
Coronavirus and Cohorts (using cohort analysis to measure rates)
Corona - the beer and the virus (The impact of a brand name being associated with a "bad thing")
(On Tuesday I will talk about issues with using metrics to set goals - specifically related to what is happening in China).
As I have written these pieces I have received significantly more comments than I did when just writing about marketing effectiveness. Many of you have shared additional links to interesting related things. I thought I would share some of those here, as well as a few of my own.
Michael shares this article on the impact of viral load on the seriousness of infection (it is looking more and more like viral load matters a lot)
David shares a link to an organization focused on getting everyone to wear masks when they are in public.
I think the data supports that this is a good idea. If you cannot get a mask, then even using a bandana can be better than nothing. SlateStarCodex has written up two great pieces on what we know about mask effectiveness:
The question is why is the WHO, CDC and Surgeon General still arguing that we should NOT be wearing masks. I think they are wrong. Ben Thompson at Stratechery [ungated] has partial summary of why that is. I may expand on this in a full post in the future. And as I was writing this yesterday around 6pm ET the White House finally decided to “urge” people to wear masks in public. Here is an app you can use to add a mask to a photo. I used it for both Twitter and Facebook. #MaskUp
Enmi is an old friend from business school. She is raising money to supply Elmhurst Hospital in Queens (a public hospital in the center of the COVID storm) with N95 masks to protect the healthcare workers there. You can read more details an make a donation here.
Related: Here is a summary from Tyler Cowen at Marginal Revolution on why the US is having such trouble furnishing masks for our health care workers. This is a massive bureaucratic failure that is costing us lives of our front line medical workers. This type of stuff happens all the time, but this crisis is making it visible in a very depressing way.
My Wife, Jessica Webster, has written a book on explaining the Coronavirus to kids. She attempted to publish is on Amazon, but they blocked publication. It is unlcear why it is being blocked, but it seems like Amazon is coming down hard on all Coronavirus content in order to protect people from “bad actors”. The result it though that great books like this one are not made available to the public. Here is a medium post on how Amazon has handled the whole thing (spoiler: Not well, inconsistently and opaquely) . Here is a free PDF of the book (not ideal for reading to kids at bedtime, but better than nothing). And here is a link to the GoFundMe Jess has set up to raise money for local child-care businesses impacted by the shutdown.
Duncan has written a research piece on the long term impacts of the 1918 Spanish Flu and the 1930s depression. The idea was to see what that could teach us about the potential long term impacts of Coronavirus 2020 . Recommended.
Other Interesting Links:
While Netflix usage is WAY up, podcast downloads are DOWN. I expect this is because people are commuting less?
Corona beer did not only not take a hit because of the coronavirus, it is actually seeing a big INCREASE in sales. I think people are drinking it ironically? Our neighbor gave my wife a six-pack of Corona for her March "Corona Birthday". I think it might be a thing?
The Economist has a lesson you can do with your kids around the Coronavirus. It's not about the science. Instead it is about general lessons on how you can figure out who and what to trust in the news.
Most of you are likely aware of the Reddit community focused on Coronavirus news. There is a second community that started around the same time (by the same people) that has pivoted to the more speculative news around the virus. Make sure you take the Economist lesson before taking anything here at face value.
It seems every newsletter and podcast is talking about the coronavirus these days. But this podcast, Virology, has done a weekly show on viruses for the last five years. They are now doing episodes every few days on what we know about the latest science and medical treatments for coronavirus. It can get pretty technical (last week's episode spent the first 20 minutes talking to a doctor in New York about technical treatment and diagnostic choices being made right now, targeted to an audience of doctors around the world who might be listening), but if you want to know what is happening on the cutting edge of science from a legitimate source, this is not a bad place to go.
Bill Bishop has been writing about China for a long time. His newsletter, Sinocism, is one of the most readable sources in English on what is going on in that country. It is now a paid newsletter, but he has a free letter once a week. Knowing what is happening in China is pretty important these days, and this is an easy way to get that information.
This virus has shown how effective Twitter is at getting to the truth (as long as you are careful and following the right people). Here is a Twitter list I have been following since February that was very helpful well before "everyone" was talking about COVID. You can add the individuals onto it into your feed, of just follow this list and check it when you want to go a little deeper into the subject.
Keep it simple. Stay Safe. Talk to you on Tuesday,
Edward