User experience designer Arthur Bodolec believes designers and entrepreneurs waste time when building products because they don’t seek user feedback early enough.
In his presentation, "Designing with your ears", the co-founder of news reader app feedly and creator of the popular resource UX Archive presents specific ways designers can collect user insights as a product gets built. With an approach somewhat similar to the Lean methodology, Bodolec argues that you waste less time when you spend more time listening to users.
Arthur Bodolec’s Guide to User Testing Your Product
Why You Need Feedback Now
As an entrepreneur, you start in darkness. You have no idea what you’re going to build, but then when you find your idea and you start seeing this light.
"As you’re thinking about your product, this light is becoming super clear," Bodolec says. "You’re starting to code with your friends, it’s becoming so clear what the product should be so you’re coding more and more and more."
This step is taking months and months, and all along you’re hoping the day you launch your product, it’ll go from this dim, fuzzy light to fireworks.
"You’re going to go viral because you have viral mechanisms, it’s just going to go crazy and maybe you’re going to go IPO the next year or something," he says.
The reality is that almost never happens.
The normal course is that you launch, people come to your site and they don’t understand your offering. At that point, you starting learning from your users, begin iterating and it becomes exciting.
"The only sad this is you’ve just spent six months implementing it," Bodolec says. "Only now, you’re starting to learn stuff from your users, which really sucks, because you could have learned stuff before."
Why can’t designers talk with potential users before launching? The truth is, they can, both for features and whole new products, the feedly co-founder believes.
Bodolec proposes a different model: work with users earlier so that, "the day you launch your product, you actually launch the thing that’s going to work and people are going to understand."
Find People to Talk to
Friends
You can start with your friends. That can be good, but your social circle might not match your target audience. That said, it’s still better than nothing.
Social Media
Creating social media accounts ahead of launch may be a good way to go, because those channels can attract people who are interested in the problem you are trying to solve. That might give you people who are a little closer to the problem you are trying to solve. And if your product already exists, then your existing users are an excellent source.
Customer Support
"If you have a support line, people are sending your emails to complain or talk about something," Bodolec says. "It’s great to just catch these people and email them back and say, ‘Can we take 15 minutes and talk about why you’re using my product or why are you interested?’"
Craigslist
Bodolec also suggests sources like Craigslist as a place to find people who can test your product or talk about it with you. When doing so, include the following:
Title: Compensation, task and location
Small Description: Time commitment, conditions, such as "sign an NDA", and whether or not you’ll be filming it
Link to Screener: Form used to help you understand your applicants
"You want to ask questions that don’t lead to answers," Bodolec says. You don’t want them to know what sort of person you are looking for so that they can’t strategically answer "right." If you are building something for avid readers you don’t want to pay for a user test from someone who never reads books.
To write your own screener, Bodolec recommends using a worksheet shared by Michael Margolis of Google Ventures.
Who do you want to talk to?
"I want to talk to non-feedly users because I want to see how people react when they use feedly for the first time," he says. "I want to talk to people who like to feed their interests on the web because these are the people that come to feedly to use feedly. And I want to talk to a mix of men and women."
What exact criteria will identify the people you want to talk to?
"Maybe they are already reading TechCrunch, they are reading CNN, they have already maybe a news reader, maybe they have a news app," he says.
What screening questions will you ask?
"The key thing is here: when you ask the question, they shouldn’t know what you’re looking for," he says.
Who do you want to exclude?
"You never want to talk to someone who is unusually technical, so people who work in tech—people who are (project managers), designers, maybe people who work for competitors," Bodolec says. "You don’t want to talk to them because experts, they already know a lot about it. So they’re going to give you opinions and it’s not going to be useful."
After posting the Craigslist ad, promoting it across your social channels and collecting submissions, the next step is to jump into the entries and highlight users that you’ll invite to participate.
Collect 10-20 User Stories
Now that you have your list of product testers, you want to understand them.
"One of the things that I do even before starting a new feature or designing something new for feedly, I gather user stories," Bodolec says. "The idea is to collect real stories, from real people to help you guide your design."
This counters the problem of the assumptions you start making about your users. Entrepreneurs are likely to make a profile of their users in their mind that may not be at all accurate. It’s better to really find out about users in order to genuinely know what they are like.
If you don’t gather stories first, Bodolec says, "you’re going to start designing features for people who don’t really exist."
Collect enough stories to do about ten actual profiles of users. Then when you are designing features, you can relate those features to people who really do exist in your set of real user profiles. You can get the story you need through a short email or 15-minute call. Then, as you build, your team can tell each other that one feature is for Josh in Cincinnati and another for Carol in Delaware, etc.
Key Questions for Interesting Stories
Bodolec shares the questions he asks in order to write user stories:
Who are they?
What is your occupation?
What is your passion?
Do you use any other product before feedly?
The first two questions relate to the average feedly user, which, he says, reads content related to their occupation or passion. For the third question, if your product hasn’t yet launched, ask what they’re doing now to solve the problem you’re attacking.
Why are they using your service?
"This is most important question of all and the most difficult to ask," Bodolec says. More often than not, you get an uninteresting answer, something too surface to be useful. For example, he says that when he asks people why they use feedly, they just describe its basic functionality back to him: "to have all my feeds in one place."
That’s just telling him what the site does, though. He already knows that. It doesn’t help him design it better or create new features. So he says he often has to rephrase the question two or three times.
Bodolec: Why do you need all your feeds in one place?
Subject: To get all my different information.
Bodolec: Why do you need to take in so much information?
Subject: To get inspired to write my book.
By asking roughly the same question over and over, you can finally start to get into the specifics of what users use your product or your competitors’ products for. Keep asking "why" questions until you get to something informative.
Once you get a sense about how a person is really using it, that can inform your other questions.
Another good question is, "In what context are you using the product?"
Where are you when use the product?
What was the last time you used the product?
What other related products do you use?
A lot of people who use feedly tell Bodolec that they also use Twitter, which also helps people find the sort of information that they need or are interested in. This allows you to start to get really interesting perspectives from users, especially if you provoke them a little.
So Bodolec likes to say something to people like, "Why do you use Twitter? That does the exact same thing as feedly." Whether he believes that or not doesn’t matter. What matters is what people show you about how they view the two products when they respond.
More often than not, when he says that to users about feedly versus Twitter, they respond by saying, "Feedly is where I get information, but Twitter is where I talk about it."
Additional Questions for Usability Feedback
If your product is up and running and you are talking to someone who uses it, you can get some feedback quickly.
Questions Bodolec likes to ask feedly users:
What frustrates you?
What’s one thing that makes you happy when you use feedly?
If you could add one thing, what would it be?
Sometimes you ask that last question, you get one tiny feature that would be easy to build but would really help certain users. It can, occasionally, be remarkable feedback.
"Almost all the time, it’s not what you really expect."
Alternatively: Gather 10-20 Feature-Based User Stories
Another way to gather user stories is to focus on feature-based user stories.
"The idea is even before starting to draw anything or to talk about it with your colleagues or anything, let’s go out, find some people and gather some stories related to the specific feature you want to launch," Bodolec says.
To do this, he asks three questions:
Who are they?
Why are they using your product?
Why do they need this feature?
Bodolec offers an example of feature-based user stories by applying this process to a new feature in development at a feedly, feedly profile. This feature will enable feedly users to share the categories in the feeds they have created.
As he was building it, anyone who mentioned something related to that feature, either as customer feedback or on Twitter or other forums, he reached out with a quick call or email to ask his three questions. What he would get back is a small story why the new feature was needed.
For each set of answers he got back, he would try to translate their response into the shortest version of it that he could write, noting the subject, subject’s audience and goal.
Subject's Answer
"I am a copywriter specializing in digital content. I use feedly to curate content from social networks (FB, Twitter, G+, LinkedIn) that I'll review/ share via my social channels. My goal is to share my feedly feeds so that my team can pull content from my "preferred" sources when they post social updates to my business' Twitter or FB page. I need to delegate that task to members of my team (I don't have time to do it myself, all the time). So I would like to have everyone access the same list of feeds so as we fine tune the content we want to share everyone is on the same page."
Bodolec's Notes
From: Copywriter
To: Her team
Goal: Let them access a curated list of feeds so they share articles from her favorite sources
Subject's Answer
"I am a food photographer. I use Bundles as a major part of my website, Eating Appalachia. Through Google Reader, I created individual Bundles for the categories of blogs/websites I read on a regular basis. As you can see on the right sidebar of my site, these Bundles are divided into "vegan," "omni," "gluten- free," and "other." If someone visiting my site clicks on "Vegan," for example, they're taken to my vegan bundle. There they can explore the blogs I read and link over to other people. I've also used this function to help promote a food blog group that I'm a part of—SSFC, Southern SOLE Food Challenge (as you can see when you click on that logo on the right sidebar of my site)."
Bodolec's Notes
From: Food photographer and blogger
To: His readers (foodies)
Goal: So his reader can discover and follow food blogs based on subtopics (vegan, omni, gluten-free)
"The idea here is that you’re going to gather lots of small stories, super specific—that are real. Then you can organize them, maybe you want to categorize them by category," he says.
Some people might want to share their collection of feeds with their team, with their friends or post it on their website. These are are all different problems. Once he has built the feature, he goes back to all the people he gathered the quick stories from and shows it to them. He asks whether or not his version solves their problem.
He doesn’t wait till he feels like it’s beautiful. He says he gets to something that basically does what he wants, even if it’s a bit ugly and hard to understand. The point is to show a basic build so you can see if you are iterating in the right direction.
"It really sucks, it’s really not nice, but I’m just calling back these people and (asking), is this kind of solving your problem—yes or no? They talk me through that and then I evolve from that," he says.
Bodolec uses Skype and email for this process. He says that Skype is really the best way to do it, but sometimes you may not have time.
Test Your Product with These Approaches
"When I say testing your product, it’s not really having an awesome product and millions of users. It’s having anything, even a drawing on a napkin, and you want to test that right away," Bodolec says.
Discovery
There are two approaches to testing. First, testing discovery: the first interaction the person has with your product. Do they find what they are supposed to find? Do they seem to understand it? Can they execute its key features?
Watch them play with the site or the prototypes and don’t answer their questions. They will probably ask you questions about how it works, but for a discovery test you should answer by saying, "I don’t know."
Features
The second approach is to test specific features. Come up with a specific use case, like posting a photo or buying a product and see if your user can do it. "Can they find it and can they use it? That’s really what you want to know," Bodolec says.
Discover + Features
You can do both at once. For example, you can watch them during their initial discovery. Once they seem to have gone through everything that is there, then you can ask them to try to do specific tasks. So you can move from discovery testing to features testing in the same session.
So, for feedly, that would mean letting them play until they can’t find anything new, and then ask them to do specific things, such as adding a feed or deleting a feed or clicking from a post in the feed onto the actual website.
"That’s really a good way to quickly see if people understand everything," Bodolec says.
Use These Test Tips
Create a Welcoming Environment
Your initial instructions to participants should set them at ease and help them understand what’s happening.
There is nothing wrong you can do
Don’t worry about hurting our feelings, we want to learn
Think aloud and describe what you are looking for and trying to do
The screen and your voice are going to be recorded
Asking them to think aloud is really important. The more you can hear what’s going on in their mind the better you will understand their behavior and what mistakes they are making while using your product.
Feedly had been coding for months when they did some usability testing. After going back through the recorded sessions, they found that people were getting lost right away because they hadn’t used quite the right words on the initial screens.
Don’t Pitch Your Product
Bodolec offers these tips for the tester’s attitude:
Don’t pitch your product, don’t lead the user
Let people use your product and get lost
If they fail to do something, ask them why they did what they did
Answer questions with questions
If it’s too general, ask for specifics
When in doubt, clarify
Look for These Two Things
You’ll be taking in so much information that it could be overwhelming, so you want to look for two things during the live test.
Aha moments
Confusing and misleading things
"What you want to do is find these aha moments and then redesign your product so they hit these aha moments as quickly as they can," Bodolec says. He shares how he discovered such a moment during testing for feedly mobile.
People came in, they would try feedly and then they’re like, ‘Oh yeah, I really like tech. So I’m going to look for tech websites.’ So they start using your product. Then they start searching for tech, they find TechCrunch, they look at TechCrunch and really nothing happens […] they open an article, they start reading—great. And then they’re like, ‘Oh yeah, but I also really love Gizmodo.’ So they start searching for Gizmodo, they look at the articles and they’re like, ‘Ah, feedly’s awesome!’
At first, I didn’t really understand. Clearly, here there is an aha moment, there’s something where it clicks in this guy’s mind, but why didn’t that work with TechCrunch and now it’s working with Gizmodo.
So I did a few tests like this and I noticed that if people find a website they already know and they already read almost everyday, and now they see it in feedly, then they get it. It’s not just finding a website from the topic you like. It’s really finding a website that you already know, that you already ready and now you see it in feedly and now you get it. … the first thing we say is, ‘Find a website you know," because we know that if this happens, something is going to happen in their brain.
Confusion is simpler. If you spot users stumbling, try to get them to talk about what they are trying to do and how they are trying to do it. Then go fix that part later.
Three Ways to Test Your Product
Each test takes about 30-60 minutes and costs about $50 to do in the U.S. Bodolec aims to do five tests. You only need to do five. If you’ve never done any tests before, you may just need to do one.
He shows the test up for testing mobile apps in his office, with a dedicated desk, a GoPro or tripod for recording their behavior on the screen and a mobile device with the prototype loaded. He also likes to set something up, like a pad, to show people where to keep their hands during the test so you can record them effectively.
If you do a desktop test, all you need is Quicktime with the mic turned on to record the screen and what the user says during the test.
15-30 minutes, on Skype or Google Hangout. Aim for five of them or one per week.
For the discovery part, he uses screensharing to show them the feature in a screenshot and asks them what they understand from what he’s showing. Can they tell him, in words, what the screen seems to be suggesting that a user can do? He wants to hear whether or not they get it or don’t get it, and then ask why.
When he showed users the feedly profile concept, he learned that one of its features had the appearance of being too permanent. The feature lets a user make a collection of feeds permanent, but it didn’t make it clear whether or not making it public was permanent or not. So he changed the design a little to make that more clear, with a switch rather than a button.
UserTesting.com is a service that does the work of the interview for you. It costs $49 per participant. The user spends 20 minutes working with your product. You give the service the criteria of the user you are looking for and they find testers for you.
"This is so useful that we do two or three every week," Bodolec says. They get awesome feedback every week. They revise their questions regularly.
You get a video back of people working on your site. Your team can also come view the tests and everyone can make notes in the same document about what they see, so different people can come up with different insights from the tests but everyone can see it in the same place. The comments are also tagged to the moment in the video, so if a comment interests a team member, they can easily jump right to that part of the video.
The first time he used the site, his first question was, "Use it for five minutes and tell me what you understand." His second was, "What do you think you should do next? Tell me and then do it." The point was for them to get deeply into the product and to hear how they understood it.
Iterate, Iterate, Iterate
Bodolec concludes with his most important piece of advice: iterate on your product.
"Once you have your feedback and everyone knows it," he says, "you iterate again and again and again as quickly as possible.
Further Reading
Bodolec recommends the following resources for further reading.
"How to Find Great Participants for Your User Study" by Michael Margolis
"Startup Lab workshop: User Research, Quick 'n' Dirty" (Video) by Michael Margolis
"A Five-Step Process For Conducting User Research" by David Sherwin
"HackDesign Lesson: Designing With Your Ears" by Arthur Bodolec
You may also enjoy this article: "How Focus on Core Features Grew Bootstrapped Feedly to 15 Million Users"
Elephant photo courtesy Brittany Hock. Arthur Bodolec photo from Twitter. Arthur Bodolec talk courtesy Le Camping via YouTube. Slide images courtesy Arthur Bodolec.
Use Feedly with Zapier
Combine feedly with Zapier, an app automation service, to make more use of the news reader app.
