Your best customers love your terrible website
Established customers don't like change
On Thursday’s premium post I wrote about how AI can improve customer service, especially new agents, and implications for lead generation companies. You can subscribe here:
On April 13th, Ben Thompson interviewed Scott Belsky, Chief Strategy Officer for Adobe (gated unfortunately). Most of the interview was about Scott’s background and the launch of Adobe’s text-to-image generator, Firefly, but about 20 minutes into the interview Scott mentioned how adding features to Adobe’s products that make things better for new users, often annoys established users. This is Scott:
One of the daily dramas I find managing products of this nature is you have established customers that are like, “Just don’t change anything, just stop. No, stop, stop don’t touch,” and then you have new customers that are like, “What the hell is this? How do I even orient myself and why can’t you make this simpler? It’s like a cockpit.” There have been a lot of scenarios — we just recently changed the onboarding to Premiere Pro, for example, which is what Hollywood uses, a lot of the Oscars were made in Premiere Pro — and we made a change that affected how you source the footage that you bring into a project when you’re making a film or a movie or whatever. It’s fascinating that when we shipped the change, the existing customers, a lot of them really hated it, they were like, “Why’d you change my flow?”. But the new customers were converting at a materially higher rate, and so you have this juxtaposition of you’ve welcomed new customers, but the NPS went down for a period of time. That’s an example of what we face. With Photoshop and Illustrator on the web, it was an opportunity to actually use these products for new customers that would convert in an easier fashion, and we also didn’t have to bring forty years of everything to the browser, because you know what? If you really want that, download Photoshop, but if you’re new to Photoshop, you actually don’t know what you’re missing, so we can bring you in with a much simpler and progressively disclosed experience. So there’s a key part of the strategy that is related to getting new people successful more quickly.
This is a very insightful point when it comes to managing change. Scott is talking about changing products, but the same applies to changing websites and landing pages.
I have seen many companies move to new CMS platforms, and when they do it is inevitably bundled with a re-design of the website. What begins as an engineering project gets combined with design, front end development, copyediting — almost the entire organization less finance. The original plan is to test the new site, but then someone realizes that testing across two platforms will be difficult and delay the launch. And the launch has already been delayed months from the original plan. Everyone looks at the new design and says, “It would be good to test, but we don’t need to. The new site is CLEARLY better than the old site.”
So the new platform and new design launches without testing.
And immediately conversion rate drops significantly.
Everyone freaks out. No one was prepared for it. There is no way to roll back to the old site. Sometimes the testing platform hasn’t been built yet and the team can’t even test their way into changes. But something has to be done!
Whether through an actual testing platform or just before and after measurement, the team will re-design the site backwards into something looking like the old site, searching for what it is that is causing the conversion rate to drop. Eventually they get to a design that sort-of looks like the old site (but different because the new platform doesn’t support some of the design elements of the old site), and conversion rate is only down a little bit. But no one understands how it was possible that conversion was down on a page that was clearly better than the old one.
There are two reasons this happens
Optimization takes time
The old site was likely operating for years. Someone had been making small changes and testing the site to make it better — marginally better every month, week and day. Imagine the site existed on a hill. The team was wandering that hill trying to find a way to go up the hill. Over time they will move up the hill to make the page better and better. Maybe they even find the peak and then get stuck, unable to improve performance more.
The new design is like a new hill — or maybe even a mountain. But when the company jumps to that new mountain it is unlikely they land right at the peak. They are somewhere on that mountain. Now the hard work begins trying to find the peak of the mountain. There is a good chance the peak of the new mountain is higher than the peak of the hill (not a guarantee, but a high likelihood if the management team has good judgment), but it is far less likely that the new (non-optimized) spot on the new mountain is higher than the peak of the old hill. The potential is higher, but the initial height is likely to be lower.
For most of my career that was what I believed. The goal became how to minimize the initial drop, and then accelerate the testing to get back to, and beyond, the old conversion rate in the new (better?) design.
Now I have a second theory.
Old vs new customers
Many users take more than one visit to your site to convert. What percent take more than one visit, and how many visits it takes depends a lot on your industry, your product, the type of customers your marketing is attracting, and more — but no company has 100% of converting customers converting on their first visit.
I would expect that the more someone comes back to your site, the higher their likelihood to convert.
So measured conversion rate is disproportionately driven by returning traffic.
And returning traffic is impacted by changes to your site.
Imagine I come to abcxyz.com and decide not to buy today. I think about it some more and then come back in a week, planning on making a purchase. Now imagine that when I get to abcxyz.com the second time the site looks completely different. I’ll bet my chance of converting just went down. Even if the new site looks “better” than the old site, and the conversion rate for new visitors on the new site is higher than new visitors on the old site.
This is the phenomenon Scott Belsky is describing in product design. Your new design is better for new users, but old users are just more familiar with the old design, and off-put by any changes. This will drop the NPS of the old users. My theory is that it will also drop the conversion rate of returning users.
I don’t know if my theory holds, but it is clearly testable. The next time I help a company through a website re-design, I am going to get them to split conversion rate measurement between new and returning visitors to see what the effects are. If returning visitor CR is down, but new visitor CR is up, then you have a lot less to worry about. You may still see a short term drop in conversion, but conversion will slowly improve and eventually surpass the old website conversion rate WITHOUT YOU HAVING TO DO ANYTHING. You just need to wait to flush through the visitors whose cheese you have moved.
If you are doing a re-design now, please check this, and let me know what you find. I will report back anything I find.
Keep it simple,
Edward