Back in October of last year, I blogged the results of some surveys and announced that some telemetry features would be added to the Fiddler beta builds. Now that 10 months have passed, I’ll provide an update on how things are going.
SmartAssembly Telemetry
Microsoft products use the Customer Experience Improvement Program (CEIP, sometimes called SQM) to collect telemetry from customer computers so we can better understand the configuration of those PCs and how the software is being used. Unfortunately, unlike Windows Error Reporting (sometimes called Watson) 3rd-party applications like Fiddler can’t participate in CEIP. Fortunately, as I announced in October, I discovered an alternative. Since then, the beta builds of Fiddler have been instrumented using RedGate’s SmartAssembly tool. SmartAssembly contains a bunch of useful features, including obfuscation, automated error reporting, and the one1 Fiddler uses– feature usage reporting.
I elected to instrument only the beta version to keep the amount of data collected to a reasonable size—while only a small fraction of the overall user-base, the beta audience is large enough to be statistically significant for the entire Fiddler population. There’s probably a bit of a bias because Beta users tend to be more “Advanced” than the average users, but I expect the effect is slight. I probably should change my strategy here to instead use a sampling percentage (e.g. 5% of users) but I haven’t found the need to do so thus far.
A total of 31,000 beta users have opted-in to data collection; data collected from the Version Check webservice (described shortly) suggests that this is about 20% of the users of the beta.
With just one call in Fiddler’s startup routine:
SmartAssembly.ReportUsage.PlatformData.ReportAll();
…a set of basic information about the user’s environment is collected. The results were definitely interesting; more users than I had expected have .NET4 installed, while only a small percentage of users have multiple monitors. XP usage is still common, although Windows Vista and later account for 56 (and growing) percent of the machines running Fiddler.
Despite the fact that Fiddler itself is only available in English, a majority of Fiddler users run Fiddler on machines with a non-English locale (blue and purple are the US and Great Britain; all other colors are non-English):
This data should prove useful if I ever decide to translate Fiddler into other languages.
While the environment information is useful, the most interesting information comes from feature usage reporting. Basic information, including the count of sessions-per-user-per-day and the Fiddler version, is captured automatically. Adding application-specific metrics is simple. SmartAssembly offers two easy methods to instrument your code. First, you can add an attribute to any method and when that method is called, its counter will be incremented:
#if TELEMETRY
[SmartAssembly.ReportUsage.ReportUsage]
#endif
/// <summary>
/// Show Fiddler’s Options dialog
/// </summary>
[CodeDescription(“Show the Fiddler Options dialog”)]
public void actShowOptions()
In some cases, you’ll want to capture events at a more granular level; in that case, just call the ReportUsage method with a string specifying the feature name:
#if TELEMETRY
SmartAssembly.ReportUsage.UsageCounter.ReportUsage(“FiddlerBoot”);
if (CONFIG.bIsBeta) {
SmartAssembly.ReportUsage.UsageCounter.ReportUsage(“IsBeta”);
}
#endif
Feature usage reporting allows me to quickly see which features in Fiddler are most popular, as well as which features need to have their discoverability improved. For instance, I know which of Fiddler’s tabs are used in a session (AutoResponder tab is activated in 7 times as many sessions as the Composer), and which features are getting overlooked (Fiddler’s insanely powerful QuickExec feature gets used in only 1% of Sessions).
By pairing the SmartAssembly telemetry data with surveys and community feedback, a tiny team (i.e. me) can prioritize improvements to maximize the benefits for the tool’s huge user-base.
Version Check Analysis
Of course, while rich telemetry is the most valuable, I’m also interested in getting some basic statistics from the entire userbase. By default, when Fiddler starts, it pings a web service to see whether a new version is available. As a part of this request, the current Windows version and Fiddler version are provided so that the web service can check for a compatible upgrade. I picked a random day, August 1st, and analyzed the data. It appears that only about 66% of users are using the latest version of Fiddler, released on June 23rd. Nearly a quarter of Fiddler users are using builds from the last two years, and about 4% of users are using truly ancient builds. 6% of the calls didn’t provide meaningful version information, either because the calls were fiddled (something that Fiddler is quite good at 
or the builds were so old that the version information wasn’t present.
It’s a little sad to see that so many users aren’t getting the maximum benefit out of the many hundreds of hours of investment made in Fiddler over the past two years. (Not to mention the hundreds of bugfixes!) If you’re running an older version of Fiddler, please upgrade today.
Book Survey
In last fall’s survey, just over a thousand Fiddler users indicated that they would buy a printed Fiddler book, and ~600 users indicated that they would purchase an E-Book on the topic. About 900 users said they’d read the EBook if it were free.
The Fiddler book was completed in June and is now available in paperback and DRM-less PDF format ($19 in paperback and $10 in PDF). The book turned out about 33% longer than I expected (it’s 330 pages) and has received good reviews. Sales have been brisk, and while I haven’t yet surpassed the goals suggested by the survey, I’m pretty happy with the results from the first two months. Additionally, two international publishers have tendered proposals for translated versions (Korean and Chinese) and a third (Japanese) is under consideration but not yet committed. Unfortunately, my plan to make the book available for free borrowing via the Kindle Lending Library hit two snags: first, the book doesn’t work well on the non-Fire Kindles (because images and source code don’t reformat well in .epub/.mobi), and second, Amazon’s terms for books in the lending library prohibit making the book available from any other source in electronic form. That would preclude me offering the book to everyone who prefers the PDF format (which looks great on the iPad and PCs).
-Eric
1 Fiddler doesn’t use the obfuscation feature because most of the classes are chock full of public methods, properties, and fields which must be callable from FiddlerScript and Extensions. Similarly, Fiddler already included a (manual) exception-reporting feature, so I elected not to switch over to the SmartAssembly version.