My four steps to the epiphany: Lessons learned from creating a minimally viable research product



In the summer of 2004, I had my first entrepreneurship experience in an unlikely place. I was still working on my PhD, when I received an invitation to spend the summer at Microsoft Research. Some of the finest researchers there have been working my topic of interest, and I was eager to see what they’d been working on, and to contribute to it. So I took the blue pill.

After the first day orientation, I went to my mentor’s office to find out which project I’d be working on. When I sat across the desk, he peeked at me through the stacks of research papers and notes, and said with a big smile: “Well, here you are. You’ve got 12 weeks to spend with us, so come up with something useful and exciting!” I looked at him waiting for a specific task, and he proceeded ” You’ve got access to hundreds of researchers and thousands of employees. Make good use of it. Good luck!”. He then introduced me to the rest of the team members, and showed me the way to my office where I would spend the next 12 weeks coming up with the next big thing. Or at least, that’s how I felt back then.

On the following morning, other interns were already printing out research papers, looking at source code, and discussing tasks among their teams.  I didn’t even know where to start. I was scared and excited.

I started by browsing through existing tools, and I read previously published papers. That led me to plan B: If I failed to come up with something original in the first couple of weeks, I could improve on an existing project.

I also brainstormed several ideas with my team, but I couldn’t decide on any of them. I spent the first weekend in the office, trying to come up with my “cool and exciting” idea in the stereotypical lone scientist way: a whiteboard full of ideas and papers scattered all over my new desk. But that didn’t seem to go anywhere, since I had no criteria to judge on which idea is better than the other. That was when my first realization hit me: I’ve been too focused on ideas and I forgot about the most valuable resource: people. My research was about programmers productivity, and I had access to tens of thousands of programmers at Microsoft.

I erased the whiteboard. And with all the ideas gone, I was free to focus on finding out, instead of guessing, the real pain points that developers go through every day.

Lessons learned: Great ideas are seldom found behind office walls.

Step 1: Listen

The following Monday, I prepared a list of interview questions, bought a voice recorder, and prepared a list of developers to interview. That week, I spent close to six hours each day asking them about about their projects, their environment, and their teams. I didn’t know what I was looking for, so I listened to everything they had to say. And they talked as long as I was willing to listen. Every day, I would refine my interview questions based on interesting patterns or triggers I was getting from the day before. Having voice recording helped me go back and confirm a point that someone made earlier.

At the end of the week, I met with my mentor to analyze the results. It became clear to us that there was a common pattern among all interviewees: The most painful and time consuming part of the development process wasn’t writing new code, but understanding existing one. In a large organization like Microsoft, source code moves across several developers, and sometimes they need to go back to what someone wrote few years ago to comprehend it and add a new feature. Without access to the person who originally wrote that code, it was a pain task.

Lesson learned: Be willing to listen without judgment.


Step 2: Validate

I’d found a good pain point, and I needed to know what people did to solve it. I thought about writing another set of questions and go interview more developers. But I wanted to get deeper insights; I wanted to know what they did when faced with the pain point. So I instrumented Visual Studio, the development tool of choice at Microsoft, to record all navigation and editing activities. Then I designed an experiment where users were given an existing project (A Tetris game written in C#) , and they were asked to fix a bug and add a feature. To accomplish these tasks, a developer needs to navigate a couple of the 25 classes in the project, and learn the role of few existing functions and variables.

I had two assumptions I wanted to validate: First, experts were faster than novices in program comprehension (easy one). Second, there is something in common about how expert went about understanding code, that’s less experienced developers don’t do. So, I invited 5 senior and 5 junior developers to help with the experiment. I recorded time, success rate, and frequently asked them to think aloud about what they were trying to accomplish and what they would imagine the solution to be. While they were working their way through the code, their activities were being saved to log files.

Later that week, I wrote some Excel macros that aggregated all the “activity streams” inside visual studio and created charts that summarized the various code areas that developers navigated, and the various paths they took.

To my delight, both assumptions were true: experts were faster than novices in understanding source code, and they took common paths to understand it.


Lesson learned: Don’t just listen to what users say, watch what they do.


Step 3: Collaborate

At that point, I knew I had a problem that was worth solving, and I had some ideas about how to solve it. But I wanted to do more than just get in a meeting room with my team and brainstorm more ideas; I wanted users to be involved. So I scheduled more interviews with developers, told them about what I am trying to accomplish, and brainstormed solutions with them. And because these users were suffering the pain point I was trying to solve, they came up with some of the most creative ideas! That was new to me, as I’ve always thought that good ideas will come from the team creating them, not people who’d be using them. Once again, I was realizing how abundant this resource was, and how little it was being used.

At the end of the week, I had gathered enough ideas to keep the whole team busy for the following couple of years. We met to discuss those ideas, we agreed that some of them had good potential and were worth prototyping and testing.

Lesson learned: Customers may have problems, but they also have solutions.

Step 4: Prototype and test

During one of the brainstorming sessions, a user asked an intriguing question: “What would it be like for me to browse source code like I browse books on Amazon? you know? with all the social recommendation goodies!” That question inspired us to create the first prototype: the FAN (Frequently Accessed Next) – a social recommendation panel that provided users with a list of places to which previous users navigated after leaving the current location. It carried the same spirit of Amazon’s recommendation language: “People who debugged this variable also looked at the following functions”.

Another prototype was inspired by the recorded log files: users typically worked with a small subset of classes for each task, which required them to repeatedly find it through a long list of all classes. To address this, we created a customizable working set of classes that is automatically filtered to provide users with the classes they need for a specific task. Finally, we wanted to provide users with a gestalt view of their source code, so we created a heat-map UML diagram that provided users with the “hottest” areas in the in the project.

We were curious to know if these prototypes delivered on their premise, so we ran a series of experiments with seven programmers, and measured their performance with and without the prototypes. The results were significantly better.

Lesson learned: Small changes can lead to big improvements.


The prototype has since evolved into a much larger project, with several full time researchers working on it. It has also been internally deployed to help multiple product teams at Microsoft.

I feel that I can take very little credit for what’s been created: users told me about their problems, they helped me with the solution, and they showed me how well the solution performed. They deserved most of the credit!

If you are interested in reading more about the experiment and prototype, you can read the paper published at InfoVis’05 .

If you find this post useful, I highly recommend Steve Blank’s book on customer development, and Eric Ries’ blog on Lean startups.

Updates:

  • I am thrilled that this post made it to the homepage of Hacker News, and stirred many conversations with great people. It’s even been translated to Japanese!
  • I no longer work at Microsoft, so I can’t confirm the current state of development on this project. Feel free to contact any of the three co-authors of the paper if you’d like to schedule a demo, or learn more.
  • I still consult with startups to help with user experience and product design. If you’d like to work with me, drop me a line.

Related Posts



I highly recommend Keynotopia for designing interface mockups for web and mobile apps

Keynotopia is a user interface design toolkit that enables you to use Apple Keynote or Microsoft PowerPoint to prototype, test and demo your application ideas quickly and cheaply, without doing any design work or writing a single line of code.

It includes thousands of wireframe and high fidelity vector user interface components and royalty free icons for mobile, web and desktop apps, all designed from scratch in Keynote and PowerPoint, and can be edited and customized without needing any design tools.

Keynotopia is used by 40,000 entrepreneurs, designers, and developers in more than 80 countries.

8 Comments

  1. This is very interesting. At Brown we had the same intuition. But we realized that people work with a working set. A working set is usually a small collection of methods spread across various classes. Then we transferred this working set UI metaphor to a new IDE called Code Bubbles. We did about 50% better than Eclipse in the user study. The paper on the same is accepted at ICSE and CHI this year.

    Reply
    • Congrats on the paper, Suman! I will definitely check it out.
      We opted for the simplest approach to solve the working set problem: putting checkboxes in the class tree, and enabled users to quickly add/remove classes and members to their working sets.

      Reply
  2. I’m trying to recollect whether we ever crossed paths when you were doing this work. Did you ever talk with MontyH, DonnaWa, JamesLau or JWeber?

    Reply
    • Aaron – It’s highly likely that we crossed paths. I met with DonnaWa and her team several times during my time at Microsoft.

      Reply
  3. Very good thoughts and very well organized Amir. I spent 3 years supporting and modifying a vendor Employees Time Recording application after the vendor company was no longer available to support it, so I could relate to this.
    BTW, step 1, you bought a time “recorder” and there was a pattern among all “interviewees”

    Reply
  4. oops.. my correction needs a correction, “voice recorder”

    Reply
    • Thanks for the corrections, Bassem 🙂

      Reply
  5. That’s fantastic work! I created a VERY basic plugin for Eclipse that is another tangential way to tackle the very same problem. I call it “Doclinks” – I just open sourced it recently, but haven’t gotten around to announcing it or improving it. The idea is to hook up documents (actually, URLs) to the living code using rules. With this, you can define “areas” or “types” of code based on rules, link them to architectural concepts, and so on. You could even filter to see only those files that relate to the document (e.g. a specific feature).

    If you’re interested, check it out: http://sourceforge.net/apps/mediawiki/doclinks/index.php?title=Main_Page

    I’d love to hear more about people working on this specific problem (how to quickly ramp up new developers on an old project).

    Reply

Trackbacks/Pingbacks

  1. links for 2010-02-18 at Scott Porad - [...] My four steps to the epiphany: Lessons learned from creating a minimally viable research product Some excellent examples about…
  2. Flow » Blog Archive » Daily Digest for February 19th - The zeitgeist daily - [...] Shared My four steps to the epiphany – Lessons learned from a research product. [...]
  3. MSでのCustomer Developmentの事例紹介 « Ghost Driver - [...] My four steps to the epiphany:Lessons learned from creating a minimally viable research product [...]
  4. links for 2010-02-25 « Blarney Fellow - [...] My four steps to the epiphany: Lessons learned from creating a minimally viable research product | A... (tags: customer-development…
  5. How to prototype iPad applications in 30 minutes or less using Apple keynote | Amir Khella - [...] My four steps to the epiphany: Lessons learned from creating a minimally viable research product February 17,…

Submit a Comment

Your email address will not be published. Required fields are marked *