My worst experiment ever 🤯

I was so sure my test would work, I told my boss I'd jump off a cliff if it didn't.

At the time, I was working as the product growth lead at The Black Tux, an e-commerce company that sells and rents tuxedos. My task was to increase conversion to purchasing and renting. We had a lot of eyeballs coming in, but not so many opening their wallets. It was my job to figure out why and to fix it.

Over the years, I have run thousands of tests with hundreds of different companies, helping them to derisk marketing, product and service decisions before going all in.

If my experiences with these companies have taught me anything, it's that you're always better off making small, strategic bets than betting the whole house. You are more likely to OVERINVEST in the wrong direction than to UNDERINVEST in the right one.

But this particular month, I was feeling lucky.

Well, not lucky per se but informed! I had run a couple tests before this one and I had a bundle of evidence (or so I thought) that this test would succeed and convert more people into buyers.

Boy, was I wrong.

I put two dedicated engineers on my test for one month. They spent A MONTH building an alternative funnel to purchase that I planned to A/B test against the standard checkout.

This was a lot of labor. It was a lot of money. And I made the case that it was worth it because (*I believed*) it would convert significantly better for revenue.

I released the test on a Friday (again, a sign of my confidence, because I never recommend launching right before a weekend).

Within two hours, it started glowing red. It wasn't just a neutral result. I had TANKED purchases.

Imagine the team's disappointment! All those wasted hours. Where did we go wrong?

This is a very common mistake. I made it 8 years into my career! And I see entrepreneurs do it every week:

Overinvesting in the wrong direction without testing the waters that it's going to pay off.

My mistake here was twofold:
1) making too many assumptions about our customers and
2) waiting too long (a month) to learn if I was right about it.

(Luckily, I had run several successful tests before and after and overall, increased conversion for the company by 10%.)

But this test will stick with me forever as a cautionary tale of what happens when you dive in head first without validating that you're on the right track. As small businesses, we have even less wiggle room to be wrong. We need to be wrong in smaller ways so we can be right faster.

I don't have special powers. You don't either. Neither did Steve Jobs or Travis Kalanick. All we have is how fast we can learn with the runway we have.

I had long been sold on the importance of testing your business assumptions but this test really cemented it for me. I needed to move faster towards learning rather than take big swings at risky ideas.

Make small bets. Track their results. Decide whether or not it's worth continuing to invest before betting your whole house.

Taking back our time together—