Over the next few weeks, we’ll be performing experiments on SmartAssembly to confirm or refute various hypotheses we have about how people use the product, what is stopping them from using it to its full extent, and what we can change to make it more useful and easier to use. Some of these experiments can be done within the team, some within Red Gate, and some need to be done on external users.

External testing

Some external testing can be done by standard usability tests and surveys, however, there are some hypotheses that can only be tested by building a version of SmartAssembly with some things in the UI or implementation changed. We’ll then be able to look at how the experimental build is used compared to the ‘mainline’ build, which forms our baseline or control group, and use this data to confirm or refute the relevant hypotheses.

However, there are several issues we need to consider before running experiments using separate builds:

1. Ideally, the user wouldn’t know they’re running an experimental SmartAssembly. We don’t want users to use the experimental build like it’s an experimental build, we want them to use it like it’s the real mainline build. Only then will we get valid, useful, and informative data concerning our hypotheses.

2. There’s no point running the experiments if we can’t find out what happens after the download. To confirm or refute some of our hypotheses, we need to find out how the tool is used once it is installed. Fortunately, we’ve applied feature usage reporting to the SmartAssembly codebase itself to provide us with that information.

3. Of course, this then makes the experimental data conditional on the user agreeing to send that data back to us in the first place. Unfortunately, even though this does limit the amount of useful data we’ll be getting back, and possibly skew the data, there’s not much we can do about this; we don’t collect feature usage data without the user’s consent. Looks like we’ll simply have to live with this.

4. What if the user tries to buy the experiment? This is something that isn’t really covered by the Lean Startup book; how do you support users who give you money for an experiment? If the experiment is a new feature, and the user buys a license for SmartAssembly based on that feature, then what do we do if we later decide to pivot & scrap that feature? We’ve either got to spend time and money bringing that feature up to production quality and into the mainline anyway, or we’ve got disgruntled customers. Either way is bad. Again, there’s not really any good solution to this.

5. Similarly, what if we’ve removed some features for an experiment and a potential new user downloads the experimental build? (As I said above, there’s no indication the build is an experimental build, as we want to see what users really do with it). The crucial feature they need is missing, causing a bad trial experience, a lost potential customer, and a lost chance to help the customer with their problem. Again, this is something not really covered by the Lean Startup book, and something that doesn’t have a good solution.

So, some tricky issues there, not all of them with nice easy answers. Turns out the practicalities of running Lean Startup experiments are more complicated than they first seem!