At RedGate Software, I work on a .NET obfuscator called SmartAssembly<\/a>. Various features of it use a database to store various things (exception reports, name-mappings, etc.) The user is given the option of using either a SQL-Server database (which requires them to have Microsoft SQL Server), or a Microsoft Access MDB file (which requires nothing). MDB is the default option, but power-users soon switch to using a SQL Server database because it offers better performance and data-sharing.<\/p>\n In the fashionable spirit of optimization and metrics, an obvious product-management question is ‘Which is the most popular? SQL Server or MDB?’<\/p>\n We’ve collected data about this fact, using our ‘Feature-Usage-Reporting’ technology (available as part of SmartAssembly<\/a>) and more recently our ‘Application Metrics’ technology:<\/p>\n
| \n Parameter<\/p>\n<\/td>\n | \n Number of users<\/p>\n<\/td>\n | \n % of total users<\/p>\n<\/td>\n | \n Number of sessions<\/p>\n<\/td>\n | \n Number of usages<\/p>\n<\/td>\n<\/tr>\n | ||||||||||||||||||||||||||||||
| \n SQL Server<\/p>\n<\/td>\n | \n 28<\/p>\n<\/td>\n | \n 19.0<\/p>\n<\/td>\n | \n 8115<\/p>\n<\/td>\n | \n 8115<\/p>\n<\/td>\n<\/tr>\n | ||||||||||||||||||||||||||||||
| \n MDB<\/p>\n<\/td>\n | \n 114<\/p>\n<\/td>\n | \n 77.6<\/p>\n<\/td>\n | \n 1449<\/p>\n<\/td>\n | \n 1449<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n (As a disclaimer, please note than SmartAssembly has far more than 132 users . This data is just a selection of one build)<\/em><\/p>\n So, it would appear that SQL-Server is used by fewer users, but more often. Great.<\/p>\n But here’s why these numbers are useless to me:<\/p>\n What does a single ‘usage’ of ‘MDB’ mean? Does this happen once per run? Once per option change? On clicking the ‘Obfuscate Now’ button? When running the command-line version or just from the UI version? Each question could skew the data 10-fold either way, and the answers only known by the developer that instrumented the application in the first place. In other words, only the original developer can interpret the data – product-managers cannot interpret the data unaided.<\/p>\n About half of people who download and run a free-trial from the internet quit it almost immediately. Only a small fraction use it sufficiently to make informed choices. Since the MDB option is the default one, we don’t know how many of those 114 were people CHOOSING to use the MDB, or how many were JUST HAPPENING to use this MDB default for their 20-second trial.<\/p>\n This is a problem we see across all our metrics: Are people are using X because it’s the default or are they using X because they want<\/em> to use X? We need to segment the data further – asking what percentage of each percentage meet our criteria for an ‘established user’ or ‘informed user’. You end up spending hours writing sophisticated and dubious SQL queries to segment the data further. Not fun.<\/p>\n Metrics can answer the when and what, but not the why. Why did people use feature X? If you’re anything like me, you often click on random buttons in unfamiliar applications just to explore the feature-set. If we listened uncritically to metrics at RedGate, we would eliminate the most-important and more-complex features which people actually buy the software for, leaving just big buttons on the main page and the About-Box.<\/p>\n People do love data. Did you know you eat 1201 chickens in a lifetime? But just 4 cows? Interesting, but useless. Often metrics give you a nice number: ‘5.8% of users have 3 or more monitors’ . But unless the statistic is both SUPRISING and ACTIONABLE, it’s useless.<\/p>\n Most metrics are collected, reviewed with lots of cooing. and then forgotten. Unless a piece-of-data could change things, it’s useless collecting it.<\/p>\n The first things that lots of people do with this data is do a t-test to get a significance level (“Hey! We know with 99.64% confidence that people prefer SQL Server to MDBs!”<\/em>) Believe me: other causes of error\/misinterpretation in your data are FAR more significant than your t-test could ever comprehend.<\/p>\n If the data appears to match our instinct, we feel satisfied and move on. If it doesn’t, we suspect the data and dig deeper, plummeting down a rabbit-hole of segmentation and filtering until we give-up and move-on. Data is only useful if it can change our preconceptions. Do you trust this dodgy data more than your own understanding, knowledge and intelligence? I don’t.<\/p>\n Let’s say we segment the above data, and get this data:<\/p>\n Parameter<\/p>\n<\/td>\n Number of users<\/p>\n<\/td>\n % of total users<\/p>\n<\/td>\n Number of sessions<\/p>\n<\/td>\n Number of usages<\/p>\n<\/td>\n<\/tr>\n SQL Server<\/p>\n<\/td>\n 13<\/p>\n<\/td>\n 9.0<\/p>\n<\/td>\n 1115<\/p>\n<\/td>\n 1115<\/p>\n<\/td>\n<\/tr>\n MDB<\/p>\n<\/td>\n 5<\/p>\n<\/td>\n 4.2<\/p>\n<\/td>\n 449<\/p>\n<\/td>\n 449<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Parameter<\/p>\n<\/td>\n Number of users<\/p>\n<\/td>\n % of total users<\/p>\n<\/td>\n Number of sessions<\/p>\n<\/td>\n Number of usages<\/p>\n<\/td>\n<\/tr>\n SQL Server<\/p>\n<\/td>\n 15<\/p>\n<\/td>\n 10.0<\/p>\n<\/td>\n 7000<\/p>\n<\/td>\n 7000<\/p>\n<\/td>\n<\/tr>\n MDB<\/p>\n<\/td>\n 114<\/p>\n<\/td>\n 77.6<\/p>\n<\/td>\n 1000<\/p>\n<\/td>\n 1000<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n How do you interpret this data? It’s one of:<\/p>\n To choose an interpretation you need to segment again. And again. And again, and again.<\/p>\n Metrics tends to be opt-in. This skews the data even further. Between 5% and 30% of people choose to opt-in to metrics (often called ‘customer improvement program’ or something like that). Casual trial-users who are uninterested in your product or company are less likely to opt-in. This group is probably also likely to be MDB users. How much does this skew your data by? Who knows?<\/p>\n There are some things metrics can answer well.<\/p>\n As you can see, questions about who-the-user-is rather than what-the-user-does are easier to answer and action.<\/p>\n |