In the summer of 2004, I had my first entrepreneurship experience in an unlikely place. I was still working on my PhD, when I received an invitation to spend the summer at Microsoft Research. Some of the finest researchers there have been working my topic of interest, and I was eager to see what they’d been working on, and to contribute to it. So I took the blue pill.
After the first day orientation, I went to my mentor’s office to find out which project I’d be working on. When I sat across the desk, he peeked at me through the stacks of research papers and notes, and said with a big smile: “Well, here you are. You’ve got 12 weeks to spend with us, so come up with something useful and exciting!” I looked at him waiting for a specific task, and he proceeded ” You’ve got access to hundreds of researchers and thousands of employees. Make good use of it. Good luck!”. He then introduced me to the rest of the team members, and showed me the way to my office where I would spend the next 12 weeks coming up with the next big thing. Or at least, that’s how I felt back then.
On the following morning, other interns were already printing out research papers, looking at source code, and discussing tasks among their teams. I didn’t even know where to start. I was scared and excited.
I started by browsing through existing tools, and I read previously published papers. That led me to plan B: If I failed to come up with something original in the first couple of weeks, I could improve on an existing project.
I also brainstormed several ideas with my team, but I couldn’t decide on any of them. I spent the first weekend in the office, trying to come up with my “cool and exciting” idea in the stereotypical lone scientist way: a whiteboard full of ideas and papers scattered all over my new desk. But that didn’t seem to go anywhere, since I had no criteria to judge on which idea is better than the other. That was when my first realization hit me: I’ve been too focused on ideas and I forgot about the most valuable resource: people. My research was about programmers productivity, and I had access to tens of thousands of programmers at Microsoft.
I erased the whiteboard. And with all the ideas gone, I was free to focus on finding out, instead of guessing, the real pain points that developers go through every day.
Lessons learned: Great ideas are seldom found behind office walls.
Step 1: Listen
The following Monday, I prepared a list of interview questions, bought a voice recorder, and prepared a list of developers to interview. That week, I spent close to six hours each day asking them about about their projects, their environment, and their teams. I didn’t know what I was looking for, so I listened to everything they had to say. And they talked as long as I was willing to listen. Every day, I would refine my interview questions based on interesting patterns or triggers I was getting from the day before. Having voice recording helped me go back and confirm a point that someone made earlier.
At the end of the week, I met with my mentor to analyze the results. It became clear to us that there was a common pattern among all interviewees: The most painful and time consuming part of the development process wasn’t writing new code, but understanding existing one. In a large organization like Microsoft, source code moves across several developers, and sometimes they need to go back to what someone wrote few years ago to comprehend it and add a new feature. Without access to the person who originally wrote that code, it was a pain task.
Lesson learned: Be willing to listen without judgment.
Step 2: Validate
I’d found a good pain point, and I needed to know what people did to solve it. I thought about writing another set of questions and go interview more developers. But I wanted to get deeper insights; I wanted to know what they did when faced with the pain point. So I instrumented Visual Studio, the development tool of choice at Microsoft, to record all navigation and editing activities. Then I designed an experiment where users were given an existing project (A Tetris game written in C#) , and they were asked to fix a bug and add a feature. To accomplish these tasks, a developer needs to navigate a couple of the 25 classes in the project, and learn the role of few existing functions and variables.
I had two assumptions I wanted to validate: First, experts were faster than novices in program comprehension (easy one). Second, there is something in common about how expert went about understanding code, that’s less experienced developers don’t do. So, I invited 5 senior and 5 junior developers to help with the experiment. I recorded time, success rate, and frequently asked them to think aloud about what they were trying to accomplish and what they would imagine the solution to be. While they were working their way through the code, their activities were being saved to log files.
Later that week, I wrote some Excel macros that aggregated all the “activity streams” inside visual studio and created charts that summarized the various code areas that developers navigated, and the various paths they took.
To my delight, both assumptions were true: experts were faster than novices in understanding source code, and they took common paths to understand it.
Lesson learned: Don’t just listen to what users say, watch what they do.
Step 3: Collaborate
At that point, I knew I had a problem that was worth solving, and I had some ideas about how to solve it. But I wanted to do more than just get in a meeting room with my team and brainstorm more ideas; I wanted users to be involved. So I scheduled more interviews with developers, told them about what I am trying to accomplish, and brainstormed solutions with them. And because these users were suffering the pain point I was trying to solve, they came up with some of the most creative ideas! That was new to me, as I’ve always thought that good ideas will come from the team creating them, not people who’d be using them. Once again, I was realizing how abundant this resource was, and how little it was being used.
At the end of the week, I had gathered enough ideas to keep the whole team busy for the following couple of years. We met to discuss those ideas, we agreed that some of them had good potential and were worth prototyping and testing.
Lesson learned: Customers may have problems, but they also have solutions.
Step 4: Prototype and test
During one of the brainstorming sessions, a user asked an intriguing question: “What would it be like for me to browse source code like I browse books on Amazon? you know? with all the social recommendation goodies!” That question inspired us to create the first prototype: the FAN (Frequently Accessed Next) – a social recommendation panel that provided users with a list of places to which previous users navigated after leaving the current location. It carried the same spirit of Amazon’s recommendation language: “People who debugged this variable also looked at the following functions”.
Another prototype was inspired by the recorded log files: users typically worked with a small subset of classes for each task, which required them to repeatedly find it through a long list of all classes. To address this, we created a customizable working set of classes that is automatically filtered to provide users with the classes they need for a specific task. Finally, we wanted to provide users with a gestalt view of their source code, so we created a heat-map UML diagram that provided users with the “hottest” areas in the in the project.
We were curious to know if these prototypes delivered on their premise, so we ran a series of experiments with seven programmers, and measured their performance with and without the prototypes. The results were significantly better.
Lesson learned: Small changes can lead to big improvements.
The prototype has since evolved into a much larger project, with several full time researchers working on it. It has also been internally deployed to help multiple product teams at Microsoft.
I feel that I can take very little credit for what’s been created: users told me about their problems, they helped me with the solution, and they showed me how well the solution performed. They deserved most of the credit!
If you are interested in reading more about the experiment and prototype, you can read the paper published at InfoVis’05 .
If you find this post useful, I highly recommend Steve Blank’s book on customer development, and Eric Ries’ blog on Lean startups.
Updates:
- I am thrilled that this post made it to the homepage of Hacker News, and stirred many conversations with great people. It’s even been translated to Japanese!
- I no longer work at Microsoft, so I can’t confirm the current state of development on this project. Feel free to contact any of the three co-authors of the paper if you’d like to schedule a demo, or learn more.
- I still consult with startups to help with user experience and product design. If you’d like to work with me, drop me a line.