Success Story: Comcast IVR user study
Comcast gauges customer experience with its IVR system
The Comcast Cable Customer Care team knew customers’ opinions about certain aspects of their service from market research in the form of focus groups and customer feedback surveys. But they didn’t know how users felt about the experience, first hand, when calling the Comcast customer service interactive voice response system (IVR).
This was the missing piece in their understanding of their customers’ experience.
With a goal of continuous improvement of all aspects of the customer’s experience, Comcast contacted the Usability Center for services in usability testing.
Gauging the market
IVR systems tend to get a bad rap. Most people have strong, negative opinions about using them, even while they understand the necessity of using these systems to transact business.
Comcast knew what their customers didn’t like, but they didn’t know the specific reactions of customers while using the IVR system for typical tasks like paying a bill, adding or removing a service, or getting support for a move. And Comcast wanted to learn if their users liked their system better or worse than other IVR systems they used.
User testing is truth serum
It’s one thing to ask someone in a focus group to recall their experience of using a product. It’s quite another to observe them using your product and then immediately sharing how they felt about the task. That’s the beauty of usability testing. It’s truth serum.
Users perform a task: for example, press 1 for a certain type of transaction; then choose from 4 other options to get closer to your goal. They complete the task (or fail to complete it) and immediately tell you how that went for them.
We observe them completing the task and can ask in-the-moment questions like, "Tell us what you were thinking when you frowned (or made a face) at a particular prompt from the system."
Users open up in the immediacy of the moment. They know exactly what they meant when they frowned (or made a face), and they are pleased to tell you.
The power of small numbers
If you hear a particular comment or concern once, you make a note of it. If you hear it twice, you begin to wonder whether you might be seeing a pattern. You confirm that, often, when you hear the same comment or concern from a third, or perhaps fourth participant.
Soon, you know the issues because they quickly emerge. Even with small numbers, you come away with big results and can take action immediately on the findings.
Invite lots of observers to learn from your users
When you have lots of observers in the executive viewing room, you spread the gospel according to the user. Comcast did just that, inviting lots of people in different divisions of the company, quite a few of whom had never met their colleagues in person before they arrived at the usability center.
The energy in the executive viewing room was electric. Everyone was talking about what they were seeing. And they were motivated to make changes to the user experience based on what they learned from their users. Immediate observation + immediate response leads to direct action.
Now Comcast wants to add other pieces of user experience to create a complete picture.