Just before midnight on the East Coast, and just before 9 p.m. on the West, Amanda Krantz and her developer at Dohje were preparing to go live on their biggest product test to date. They were being featured on AOL’s home page as part of Nurses Week, and had the opportunity to get tons of new users on their site. But, they screwed up the timing — or rather, the time zone — and the opportunity came three hours earlier than they’d planned, three hours earlier than they were ready for. Their new user flow wasn’t live, but they started seeing visitors on their site at exactly at 12 a.m. Eastern — not Pacific.
Krantz panicked. Then she took a deep breath and said, “No, we’re A/B testing.” Rather than give up the opportunity to experiment and to learn, Krantz waited until the late night crowd slowed down, and switched over to the new site at 2:30 a.m. — 5:30 a.m. Eastern. “I figured that, as long as we could roll out at some point, we were testing our old flow versus the new one.”
Founder and CEO of Dohje, an app that makes it easy for people to thank their caregivers — hospital staff who help them through some of life’s most difficult moments — Krantz’s response to the time zone fiasco sums up her attitude about being in the nascent stages of a startup. “When something goes wrong, there’s always something you can learn,” she says today, a month later.
With possibilities endlessly swirling around you, prioritizing experiments and what to build is a bit like herding cats. Krantz says you have to give yourself a constraint (usually, she uses time), set up a test, and see what you can learn. And actually, Dohje started as a pretty basic experiment.
“Originally, I didn’t really think this was a business,” Krantz said. “I come from startups and so, rather than jumping to business model, I really just wanted to see: Would patients want to use an online tool to say thank you? And, would staff be interested in receiving gratitude that way?”
So she built a basic site with a few core functions of her idea, one where a person could find their caregiver — by photo or by name — write a note, attach a photo if they wanted, and send it. The sender could be a former patient or a hospital staffer looking to thank a colleague. Through testing, Krantz soon realized that people were more likely to send notes if more staff were listed.
Because nurses and other hospital staff weren’t keen to put up listings, she set up a test to see if she could get more people to create listings if they had nice photos of themselves to put on the app. (About.me, LinkedIn, and Airbnb have tried similar approaches with success). To do that, Krantz and a Dohje team used an empty hospital room to create a cheap photo studio.
Where before only 10 percent of hospital staff had created profiles and started thanking each other, 90 percent of those who had their photos taken would create profiles and begin sending Dohje notes. And, bonus, Krantz and her team were able to gain some valuable insights while talking to the staff for the three minutes they were each in the room getting their photos taken. Another bonus: Hospital staff morale went up as people who had otherwise felt insignificant now had not only a professional photograph of themselves, but also tons of thank you notes from their colleagues.
Rather than be deterred by criticism that this professional photo experiment wouldn’t scale, Krantz went ahead and learned some deeply valuable lessons. Because of limited resources, Krantz often prioritizes product features and tests based on customer — hospitals are their clients — demand. It’s a practical approach for a very early stage business, where Krantz constantly maneuvers between securing clients and funds, and building out her original vision. When it works right, she can run low effort product tests that are specifically paid for by client hospitals looking to make a match with Dohje.
“If we had unlimited resources, I think we’d make decisions really differently, but we have to be really, really pragmatic on what we’re going to prioritize. Like often, you can’t run any more tests until you just fix the things that appear to be broken.”
And anyway, it all keeps coming back to the start, Krantz said. “If you look back to our initial ideas, nothing has changed. You just get a little bit more data to validate digging back into that original idea.”
Getting data to validate or invalidate an idea is at the core of testing, but obviously there’s a lot more to it than that. In 2013, Wyatt Jenkins gave a comprehensive talk on the subject, in which he promises, “At the end, you’re going to be better at testing.” Here it is [45:51]:
One hospital asked Dohje to try adding a “Donate” option after somebody sent a note. At that point, Dohje was only targeting patients from the hospital’s labor and delivery department. “We could have A/B tested different language, or tried using a button, or see if it worked better in email,” Krantz said. “But first, we just had to do something. Because, most likely, all those little tweaks aren’t going to make or break it.” They put the link up, and nobody donated. Then the questions about other options reemerged: Do we need to change the language, or should the link be in another place?
“It turns out,” Krantz realizes in retrospect, “That you can ask people who have a lot more experience than you about the user behavior of whatever new thing you’re doing.” So they approached experienced fundraisers at ten different hospitals, who all concurred: “We never get donations from labor and delivery.”
With that unequivocal response, Krantz realized another test was no longer necessary. Sure, she could’ve asked the fundraisers beforehand and saved the test, but ultimately, adding the “Donate” link was the faster first option, even though digging in provided the most validating answer.
After the AOL test this May, Krantz had put together some significant lessons around experimentation. Some things were obvious, like getting the time zone right and making sure any new code is ready before it goes live. But to do all of that, to run a successful test, Krantz recommends that an entrepreneur “just keep scoping down.” Pare down your test, she says, without giving yourself excuses to put it off.
“You can’t be afraid to try. I got really close to not even pushing out the new code because there were bugs. There were problems. We learned so much more, and got good results, but it was really scary.” Opportunities that big are rare for such an early company, and you only stand to gain from them, Krantz advises. “It’s worth testing, sometimes more for the chance of learning, even if you can’t isolate as granularly as you’d like.”
And yet, testing doesn’t really fit into a textbook, she adds. “You don’t necessarily have follow Lean Startup exactly.” But setting a constraint, she says, is crucial. “I start with just putting a stake in the ground, and date is the best. We kept cutting things at the end in order to hit that.”
She cuts features according to the question: Could we test what we’re going to test without that extra thing in there?
At the 2014 Lean Startup Conference, Dan McKinley offered his experience as an example for how to vet your many, wonderful sounding ideas with data: [16:51]
All those extra things that are cut have a place to go: The Dohje team keeps their Pivotal Tracker icebox full of ideas, old and new, for features and future tests. It also serves as a repository for itchy ideas that need to be written down, and for institutional memory of what’s been considered but ultimately passed over. Some ideas are based on optimizing current features, but as a small startup, those stay at the bottom of the list. “I don’t think you need to slow down to decide how to optimize until you grow to a certain level,” she says. “And we’re not there yet.”
Krantz knows that she’s chasing an evermoving target, and that her tests won’t be more clear-cut until Dohje gets a bit larger. For now, she says, the learning is the triumph.
“How you define what is success is where I debate with my husband a lot. We’re both engineers, but his approach is more, ‘You need to be very specific on success.’ But I don’t think you do. I think that’ll waste time sometimes, when — who cares? You can usually tell if you don’t get enough information and clear results, and then maybe you get more specific. But I have never had an experiment where I felt like it’s inconclusive. There’s always something you can learn.”
This week, the Lean Startup is taking over the blog on Intuit Labs with original stories and a fresh perspective. Centered around experimentation and investigating all parts of a business or product idea, this week’s posts include case studies, tips, Q&As, startup stories and more. If you want to learn more about Lean Startup and how it’s applied at Intuit, visit the Intuit Innovation Institute. This piece was written by Mercedes Kraus.