Test in the Wild
Test your app via real world usage
There's no substitute for real people using your app in real ways. Get real data. Get real feedback. Then improve based on that info.
Formal usability testing is too stiff. Lab settings don't reflect reality. If you stand over someone's shoulder, you'll get some idea of what's working or not but people generally don't perform well in front of a camera. When someone else is watching, people are especially careful not to make mistakes — yet mistakes are exactly what you're looking for.
Instead, release beta features to a select few inside the real application itself. Have them use the beta features alongside the released features. This will expose these features to people's real data and real workflow. And that's where you'll get real results.
Further, don't have a release version and a beta version. They should always be the same thing. A separate beta version will only get a superficial walk through. The real version, with some beta features sprinkled in, will get the full workout.
The Beta Book
If developers are nervous releasing code, then publishers and authors are terrified of releasing books. Once a book gets committed to paper, it's seen as a big hairy deal to change it. (It really isn't, but perception and memories of problems with old technologies still linger in the industry.) So, publishers go to a lot of trouble (and expense) to try to make books "right" before they're released.
When I wrote the book Agile Web Development With Rails, there was a lot of pent up demand among developers: give us the book now — we want to learn about Rails. But I'd fallen into the mindset of a publisher. "It isn't ready yet," I'd say. But pressure from the community and some egging on from David Heinemeier Hansson changed my mind. We released the book in pdf form about 2 months before it was complete. The results were spectacular. Not only did we sell a lot of books, but we got feedback — a lot of feedback. I set up an automated system to capture readers' comments, and in the end got almost 850 reports or typos, technical errors, and suggestions for new content. Almost all made their way into the final book.
It was a win-win: I got to deliver a much improved paper book, and the community got early access to something they wanted. And if you're in a competitive race, getting something out earlier helps folks commit to you and not your competition.
—Dave Thomas, The Pragmatic Programmers
Do it quick
- Decide if it's worth doing, and if so:
- Do it quick — not perfect. just do it.
- Save it. upload it. publish it
- See what people think
Though I'm always reluctant to add new features to things, once I have that "yeah!" moment of deciding something is worth doing, it's usually up on the website a few hours later, flawed but launched, letting feedback guide future refinement of it.
—Derek Sivers, president and programmer, CD Baby and HostBaby