Note: Please see this post for an update.

Some users of Fiddler who have HTTPS Decryption enabled have found that some of their internal HTTPS sites that used to work properly with Fiddler now endlessly prompt for credentials while Fiddler is running. Even typing the correct credentials into the authentication prompt won't fix the problem. What happened?

The problem is that the servers in question have enabled the new Extended Protection / Channel Binding Tokens security feature. This feature helps to prevent reuse of credentials by binding an authentication response to the channel (e.g. the HTTPS connection) on which the authentication challenge was received. This obviously poses a problem for Fiddler, because Fiddler uses a man-in-the-middle strategy to decrypt HTTPS traffic. That means that the server's Authentication Challenge comes to Fiddler on a different connection than Fiddler uses to return that challenge to the client. By design, the client responds to the server's challenge using the information from its connection to Fiddler, and the server, noticing the mismatch, rejects those credentials.

To know if this problem applies to you, the following must be true:

1. The site works properly when Fiddler isn't running
2. The site is running on HTTPS
3. The site works properly when Fiddler's HTTPS-decryption feature isn't enabled
4. The server sends back a HTTP/401 authentication challenge even when correct credentials are supplied

Now, CBT is not currently widely deployed, but in some major organizations, it has been deployed on one or more critical servers. For instance, some organizations might have enabled CBT on their ADFS proxy server in order to protect ADFS logins to other sites. If that's the case, a user trying to debug that other site will find that they cannot log in with Fiddler running. Annoying.

Fortunately, there are workarounds for the cases where you don't actually care about the traffic to particular sites. While disabling HTTPS-Decryption globally works, it's somewhat annoying. Instead, you can disable HTTPS-decryption for specific sessions. You can do so by setting the x-no-decrypt flag on a given session, or, in Fiddler 2.3.0.6 or later, you can do so by listing the hostname inside the text box Skip Decryption for the following hosts found by clicking Tools > Fiddler Options > HTTPS.

(It is likely that a future version of Fiddler will be able to debug HTTPS traffic even with CBT enabled, because Fiddler runs on the client and has access to the user's credentials. Fiddler itself can provide a proper response to the server's credential challenge; I only need to update the code for the existing x-AutoAuth flag to use the channel information from the HTTPS connection.) Note: As noted in this post, this fix was made for Fiddler v2.3.6.1.

Beyond avoiding problems with servers that have CBT enabled, the ability to skip decryption for certain connections can be useful for other cases as well. For instance, Microsoft Outlook can be configured to communicate with Microsoft Exchange using RPC over HTTPS. You will find that when Fiddler decrypts such traffic, the client typically will no longer work. The problem is that Fiddler buffers all requests before sending them to the server (only streaming of responses is supported) and the RPC client never "finishes" sending a complete HTTP message. Hence, Fiddler will block communication to the server. If you disable HTTPS-decryption for servers used for RPC over HTTPS, you will still see the CONNECT tunnel through which the secure traffic flows, but Fiddler will not interrupt the traffic. Alternatively, you could disable HTTPS-decryption for traffic from an entire application (e.g. boring.exe) using a rule like this inside OnBeforeRequest:

if (
oSession.HTTPMethodIs("CONNECT") 
&& oSession["X-PROCESSINFO"] 
&& oSession["X-PROCESSINFO"].StartsWith("boring")
) 
 { 
 oSession["x-no-decrypt"] = "boring process"; 
 }

thanks,

-Eric

Note: Please see this post for an update.

I’ve recently heard from a number of Silverlight developers who report that certain cross-domain web service requests from their applications work properly with Fiddler running, but fail when Fiddler is not active. Using lower-level tools like NetMon or by watching server logs, the developers note that their applications aren’t even issuing requests for the cross-domain policy file unless Fiddler is running.

So, what’s going on?

The answer is that Silverlight is suppressing web service requests to the target URLs (when Fiddler isn’t running) because the target URL is in the Local Intranet security zone, while the Silverlight application is running from the Internet security zone. Silverlight forbids cross-domain requests from the Internet to the Local Intranet^[1], and doesn’t bother looking for a cross-domain policy file.

So why don’t these cross-zone requests fail while Fiddler is running?

The answer goes back to a post I wrote over half a decade ago: The Local Intranet Zone and Proxies: the Surprising Connection. In these developers’ environments, the target web services are mapped to the Local Intranet Zone by a proxy configuration script specified in the WinINET settings.

So, for example, if the proxy configuration script specifies a proxy for www.example.com, but returns “DIRECT” for services.dev.extranet.example.com, Silverlight will block a request from an application running at www.example.com (Internet) because the target service URL is mapped to the Local Intranet Zone.

While Fiddler is running, it overrides the local proxy settings, removing the proxy configuration script from the IE settings and using the configuration script itself internally when determining the upstream gateway. However, because Fiddler is a directly-configured proxy, Internet Explorer has no proxy configuration script while Fiddler is attached. Because no hostnames are configured to bypass Fiddler, all dotted hostnames are treated as Internet Zone unless they are manually mapped to another zone using the Internet Control Panel.

To resolve this problem such that the application works without Fiddler running, you can untick the “Include all sites that bypass the proxy server” option inside Internet Explorer’s Tools > Internet Options > Security > Local Intranet > Sites dialog. Alternatively, you could use the Advanced button in that dialog to map the page hosting the Silverlight application itself into the Intranet Zone, ensuring that the application and its web service URL are running from the same zone.

-Eric

^[1] Blocking of Zone Elevation is a security mitigation adopted by a number of technologies including IE’s XDomainRequest object.

Back in October, I showed how to debug Windows Phone emulator traffic with Fiddler . Since then, I’ve acquired the LG Quantum phone, and naturally, one of my first goals was to start looking at the traffic from mobile Internet Explorer and some of my WP7 applications. The process for capturing traffic from a phone is similar to the process of capturing traffic from another computer . First, enable Fiddler to capture remote traffic, then configure the other client to point at the computer running Fiddler.

You’ll need three things:

1. A desktop PC running Fiddler
2. A Windows Phone
3. A WiFi network that bridges between the phone and the PC

First, I started Fiddler on my desktop PC, named DELL7 -- obviously, you’ll need to use your machine’s name for all of the subsequent steps. In Fiddler, click the Tools > Fiddler Options menu, and click the Connections tab.

Place a checkmark in the Allow remote computers to connect box and close and restart Fiddler. When Fiddler restarts, you’ll see a prompt from the Windows Firewall indicating that it has blocked some features of Fiddler. Tick the three checkboxes and click the Allow Access button.

Now, Fiddler can capture HTTP and HTTPS requests from other devices.

To verify that this is working correctly, enable your phone’s WiFi connection and open mobile IE. In the browser, type http://DELL7:8888 , and you should see a request in Fiddler and a “Fiddler Echo Service” page in the browser.

•  If you can’t see the page, you’ll need to recheck your work so far. In particular, if your Fiddler computer is running on a corporate network with IPSEC enabled (like Microsoft's corpnet) you will need to get an your IT department to grant you an IPSEC Boundary Computer exception. Otherwise, a device that isn't IPSEC enabled (like your phone) will not be able to exchange traffic with the Fiddler computer.

Now that your Phone and PC are talking, let’s set the phone’s proxy to point at Fiddler. On the phone, visit the Settings screen and open the WiFi applet.

In the WiFi applet, select the active WiFi connection to open the Edit Network screen. Slide the Proxy slider to on. In the Server/URL box, type dell7 and in the Port box, type 8888 . When you are done debugging your traffic later, come back to this screen and simply move the proxy slider to Off.

Save the settings by clicking the checkmark, and go back to mobile IE. Browse around, and you should see your traffic in Fiddler. If you’ve configured Fiddler to decrypt HTTPS traffic and try to browse to a HTTPS page on your Phone, you’ll see a warning message because Fiddler has resigned the traffic with its own certificate.

If you click the Continue to website link, you’ll see the decrypted traffic in Fiddler. However, some applications will not offer the opportunity to bypass certificate warnings. If you’d like, you can add Fiddler’s root certificate to your phone’s root certificate store.

Warning : At present, I have no idea how to remove the certificate, so undertake this step at your own risk. In particular, you will likely have problems later decrypting traffic with a different Fiddler desktop PC with your phone, since each Fiddler instance generates its own certificate.

To add the Fiddler root certificate to your phone, visit http://dell7:8888/FiddlerRoot.cer in mobile IE.

Tap to open the file. Choose Install to install the certificate.

After you do this, you will find that all HTTPS traffic from mobile IE and applications is visible in Fiddler.

Have fun with Fiddler and Windows Phone 7!

-Eric Lawrence

I’m delighted to announce that the now-available IE9 RC includes three significant enhancements for users of proxy-based debuggers like Fiddler.

These improvements are:

1. The default Connections-Per-Proxy limit has been raised from 6 to 12, improving performance and in some cases reducing Observer Effect.
2. Debugging of traffic sent to Localhost / 127.0.0.1 now “just works”—configuration changes are not required.
3. Internet Explorer now can be configured to emit information about why a given HTTP request was issued, which helps you understand your web traffic.

I’ll explain each of these three improvements in this post.

Connections-Per-Proxy Limit

Browsers are typically designed to limit the number of connections made to a single server in order to prevent overloading it or incurring other problems. Some browsers have a different limit depending on whether the server being contacted is a proxy server or a normal host (web server). Internet Explorer 6 and 7 apply the “web server” connection limit to proxies as well; the two connection limit those versions use can severely impact your debugging performance when using Fiddler. Users still using those outdated browsers can re-configure the connection limit to mitigate this problem. Internet Explorer 8 limits the connections per proxy to six, which was a welcome improvement, but still could cause performance problems when debugging sites that “shard” their requests to many different hosts. Internet Explorer 9 maintains the existing connections-per-host limit of six, but also includes an specific connections-per-proxy limit which is set to 12 by default. This increased limit should help reduce the impact of connection limits upon your debugging scenarios.

For comparison, Firefox’s default value for the network.http.max-persistent-connections-per-proxy setting is eight, but the FiddlerHook extension kicks this value up to twenty-four.

Proxying Localhost Traffic

The WinINET networking component that is used by Internet Explorer and many other applications will automatically bypass a fixed proxy (like Fiddler) for traffic bound for //localhost and //127.0.0.1 because these “loopback” addresses point to the local machine and traditional proxy servers will not be able to interpret such addresses properly. However, for a debugging proxy running on the local computer, these are perfectly understandable addresses, and when developers are debugging against a local server (like IIS Express or the Visual Studio Test Server Cassini) they often test against these loopback addresses. To proxy loopback traffic from IE8 and below, somewhat awkward workarounds are needed.

IE9 RC introduces the ability to proxy loopback traffic. To do so, simply include the token <-loopback> (pronounced “minus-loopback”) in the proxy bypass list. When WinINET encounters this token in the bypass list, it will remove the loopback addresses (localhost, 127.0.0.1) from the list of hosts that bypass the proxy. Fiddler 2.3 and above automatically set this option when capturing traffic.

The FiddlerHook add-on automatically sets the equivalent version of this option for Firefox; for Opera, you can manually remove loopback addresses from the proxy-bypass list.

The System.Net networking component used by DotNet Framework applications does not yet support the minus-loopback token; I’ve filed a request that they do so, but if this is an important feature for your work, you might consider filing feedback on Connect.

Understanding Download-Initiator

Proxy-based debuggers have a few key strengths—chief among them is the ability to debug traffic from any application that supports a proxy (which is pretty much all of them). One downside of debugging at the proxy layer, however, is the loss of context—it can be very difficult to trace back to determine why a given HTTP request was issued.

Having said that, Fiddler includes a number of features to help you understand context. First, Fiddler attempts to map inbound requests back to the process that issued them. For browsers like Internet Explorer 8, with its “loosely-coupled” process architecture, this often means that each browser tab sends traffic from an individual process. The process information is shown in the Process column in the session list, and FiddlerScript and extensions may access the Process Name and instance ID (aka “PID”) using the Session object flag named X-PROCESSINFO.

Fiddler also uses the HTTP Referer header to help you associate traffic—type the “P” key on any session in the session list to jump back to the “parent” session. Fiddler assumes that the parent session is the session is the most recent request to the URL specified in the selected session’s Referer header. Or, push the “C” key to select all of the “child requests”, those requests after the current request that have a Referer of the currently selected session’s URL.

Internet Explorer 9 includes two new features that help add more context.

First, the Web Browser now sends a meaningful Accept header for most types of downloads. Previously, IE sent a long, registry-generated string for document downloads and Accept: */* for everything else. This limitation made it impossible to reliably distinguish between a request initiated by a <LINK REL=”STYLESHEET”> element and one initiated by <SCRIPT> element. IE9 RC will send the following Accept headers, depending on context:

For compatibility with legacy sites, the legacy Accept headers are sent when a site is configured to run in Compatibility View.

Within Fiddler, you can display the Accept header in the Session list as a column. To do so just for the current session, enter the following command in the QuickExec box:

cols add @request.Accept

This will add a column labelled “@request.Accept” and as each session is logged, the request’s Accept header, if any, will be listed. To add this column every time Fiddler starts, click Rules > Customize Rules. Scroll to the static function Main() block , and add the following line within:

FiddlerObject.UI.lvSessions.AddBoundColumn("Accept", 50, "@request.Accept");

Second, while understanding what class of Element initiated a request is useful, IE9 includes an even more valuable feature that conveys contextual information about why a request was made. You can see this feature at work in the Internet Explorer F12 Developer Tools, which you can open by pressing (you guessed it) the F12 key. On the Network tab, you’ll see the “Initiator” column that provides more information about the context in which a request was made:

This information, by default, is not sent to the network, but you may set a Feature Control Key to emit the information as a custom HTTP request header that Fiddler will see. Most Feature Control Keys, including this one, are simple flags stored in the registry that change the behavior of the Web Browser when set.

• Enable FEATURE_DOWNLOAD_INITIATOR_HTTP_HEADER
• Disable FEATURE_DOWNLOAD_INITIATOR_HTTP_HEADER

You can display this information in Fiddler using the same technique described previously:

cols add @request.X-Download-Initiator

This will add a column labelled “@request.Accept” and as each session is logged, the request’s Accept header, if any, will be listed. To add this column every time Fiddler starts, click Rules > Customize Rules. Scroll to the static function Main() block , and add the following line within:

FiddlerObject.UI.lvSessions.AddBoundColumn("Reason", 50, “@request.X-Download-Initiator");

Then, when you load a page, you will see the initiator information:

Interpretation of these tokens is as follows. First, there’s a token (e.g. image, script, html) indicating what type of content is being downloaded. Next, an indicator of which document object “owns” the download. Information about what element made the request is present, and in some cases, information about what the exact download trigger was. For instance “html tokenizer” means that the download request occurred during the parsing of HTML. “html lookahead” indicates that the Lookahead parser is making the request. On the other hand, “src property change” means that the download occurred because script changed the SRC property of an <img> tag.

I hope that you find these new features useful, and enjoy an improved debugging experience with the IE9 Release Candidate!

-Eric Lawrence

I think origin stories are fun. I spoke about the origins of Fiddler in my talk at the PDC last year, and it was interesting to share how a little idea bloomed into a surprisingly widely-used piece of software. Cleaning up some old hard drives a few weeks ago, I found an old UI prototype that I’d been playing around with in 2002/2003. Like many of my early prototypes of that era, it was simply a piece of UI without any actual functionality. It made me smile however, so I’m going to publish it here for posterity:

Somewhat familiar perhaps, although I think some would have found the colors a bit jarring. :-) In case you’re wondering, “Bayden Systems” is the name I release most of my freeware under.

In other, more recent news, I’ll be doing another talk at the MiX Conference next month. This year’s talk is titled The Devil Went Down to HTTP: Debugging with Fiddler… it’s a play on a song title, in case you haven’t heard it. I have a few ideas for what I’ll be covering, but if you’ll be at MiX (or watching online) and have some suggestions for topics to cover or demos to do, please let me know!

thanks,

-Eric

One feature added to Fiddler a few years ago is the ability to map a given HTTP request back to the local process that initiated it. It turns out that this requires a bit of interesting code, because the .NET Framework itself doesn’t expose any built-in access to the relevant IPHelper APIs that provide this information.

I found a number of samples on the web, but for Fiddler, performance is a critical consideration because Fiddler needs to determine the originating process for every new connection. Hence, I’ve written the following code, which maximizes performance by minimizing copies between Windows and managed code.

// This sample is provided "AS IS" and confers no warranties.
// You are granted a non-exclusive, worldwide, royalty-free license to reproduce this code,
// prepare derivative works, and distribute it or any derivative works that you create.
//
// This class invokes the Windows IPHelper APIs that allow us to map sockets to processes.
// See http://www.pinvoke.net/default.aspx/iphlpapi/GetExtendedTcpTable.html as a reference
//
// We could consider a cache of recent hits to improve performance, but the performance is already pretty good, and 
// creating a reasonable cache expiration policy could prove tricky. Client connection reuse already provides a significant
// optimization as it behaves in the same way as an explicit cache would.
// 
using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
using System.Net.NetworkInformation;
using System.Net;
using System.Diagnostics;
using System.Collections;

namespace Fiddler
{
    internal class Winsock
    {
        #region IPHelper_PInvokes

        private const int AF_INET = 2;              // IPv4
        private const int AF_INET6 = 23;            // IPv6
        private const int ERROR_INSUFFICIENT_BUFFER = 0x7a;
        private const int NO_ERROR = 0x0;

        // Learn about IPHelper here: http://msdn2.microsoft.com/en-us/library/aa366073.aspx and http://msdn2.microsoft.com/en-us/library/aa365928.aspx
        // Note: C++'s ulong is ALWAYS 32bits, unlike C#'s ulong. See http://medo64.blogspot.com/2009/05/why-ulong-is-32-bit-even-on-64-bit.html
        [DllImport("iphlpapi.dll", ExactSpelling = true, SetLastError = true)]
        private static extern uint GetExtendedTcpTable(IntPtr pTcpTable, ref UInt32 dwTcpTableLength, [MarshalAs(UnmanagedType.Bool)] bool sort, UInt32 ipVersion, TcpTableType tcpTableType, UInt32 reserved);

        /// <summary>
        /// Enumeration of possible queries that can be issued using GetExtendedTcpTable
        /// http://msdn2.microsoft.com/en-us/library/aa366386.aspx
        /// </summary>
        private enum TcpTableType
        {
            BasicListener,
            BasicConnections,
            BasicAll,
            OwnerPidListener,
            OwnerPidConnections,
            OwnerPidAll,
            OwnerModuleListener,
            OwnerModuleConnections,
            OwnerModuleAll
        }

/* This code is now obsolete as I'm now using pointer-arithmetic to directly access the table rows instead of mapping structs on top of the 
 * returned block of data. I'm keeping the code here for now for debugging purposes.
        // http://msdn2.microsoft.com/en-us/library/aa366913.aspx
        [StructLayout(LayoutKind.Sequential)]
        private struct TcpRow
        {
            [MarshalAs(UnmanagedType.U4)]
            internal TcpState state;
            [MarshalAs(UnmanagedType.U4)]
            internal UInt32 localAddr;
            [MarshalAs(UnmanagedType.U4)]
            internal UInt32 localPortInNetworkOrder;
            [MarshalAs(UnmanagedType.U4)]
            internal UInt32 remoteAddr;
            [MarshalAs(UnmanagedType.U4)]
            internal UInt32 remotePortInNetworkOrder;
            [MarshalAs(UnmanagedType.U4)]
            internal Int32 owningPid;
        }
        private static string TcpRowToString(TcpRow rowInput)
        {
            return String.Format(">{0}:{1} to {2}:{3} is {4} by 0x{5:x}",
                (rowInput.localAddr & 0xFF) + "." + ((rowInput.localAddr & 0xFF00) >> 8) + "." + ((rowInput.localAddr & 0xFF0000) >> 16) + "." + ((rowInput.localAddr & 0xFF000000) >> 24),
                ((rowInput.localPortInNetworkOrder & 0xFF00) >> 8) + ((rowInput.localPortInNetworkOrder & 0xFF) << 8),
                (rowInput.remoteAddr & 0xFF) + "." + ((rowInput.remoteAddr & 0xFF00) >> 8) + "." + ((rowInput.remoteAddr & 0xFF0000) >> 16) + "." + ((rowInput.remoteAddr & 0xFF000000) >> 24),
                ((rowInput.remotePortInNetworkOrder & 0xFF00) >> 8) + ((rowInput.remotePortInNetworkOrder & 0xFF) << 8),
                rowInput.state,
                rowInput.owningPid);
        }

 */
        #endregion IPHelper_PInvokes

        /// <summary>
        /// Map a local port number to the originating process ID
        /// </summary>
        /// <param name="iPort">The local port number</param>
        /// <returns>The originating process ID</returns>
        internal static int MapLocalPortToProcessId(int iPort)
        {
            Debug.Assert(((iPort > 0) && (iPort < 65536)), "Unexpected client port value");
            // Stopwatch oSW = Stopwatch.StartNew();
            int result = FindPIDForPort(iPort);
            // FiddlerApplication.Log.LogString("Port hunt took: " + oSW.ElapsedMilliseconds); // Current version seems to take about 1ms on average, with a range up to ~35ms.
            return result;
        }
      
        /// <summary>
        /// Calls the GetExtendedTcpTable function to map a port to a process ID.
        /// This function is (over) optimized for performance.
        /// </summary>
        /// <param name="iTargetPort">Client port</param>
        /// <param name="iAddressType">AF_INET or AF_INET6</param>
        /// <returns>PID, if found, or 0</returns>
        private static int FindPIDForConnection(int iTargetPort, uint iAddressType)
        {
            Debug.Assert(iAddressType == AF_INET6 || iAddressType == AF_INET);
            IntPtr ptrTcpTable = IntPtr.Zero;
            UInt32 tcpTableLength = 0;

            int iOffsetToFirstPort = 12;
            int iOffsetToPIDInRow = 12;
            int iTableRowSize = 24; // 24 == Marshal.SizeOf(typeof(TcpRow));

            // IPv6 tables are a different size, so adjust the offsets accordingly
            if (iAddressType == AF_INET6)
            {
                iOffsetToFirstPort = 24;
                iOffsetToPIDInRow = 32;
                iTableRowSize = 56;
            }

            // Determine the size of the memory block to allocate
            if (ERROR_INSUFFICIENT_BUFFER == GetExtendedTcpTable(ptrTcpTable, ref tcpTableLength, false, iAddressType, TcpTableType.OwnerPidConnections, 0))
            {
                try
                {
                    ptrTcpTable = Marshal.AllocHGlobal((Int32)tcpTableLength);

                    // Would it be faster to set the SORTED argument to true, and then iterate the table in reverse order?
                    if (NO_ERROR == GetExtendedTcpTable(ptrTcpTable, ref tcpTableLength, false, iAddressType, TcpTableType.OwnerPidConnections, 0))
                    {
                        // Convert port we're looking for into Network byte order
                        int iTargetPortInNetOrder = ((iTargetPort & 0xFF) << 8) + ((iTargetPort & 0xFF00) >> 8);

                        // ISSUE: This function APPEARS to work fine, but might blow up on Itanium or exotic architectures like that. As noted in the docs:
                        // The MIB_TCPTABLE_OWNER_PID structure may contain padding for alignment between the dwNumEntries member and the first MIB_TCPROW_OWNER_PID
                        // array entry in the table  member. Padding for alignment may also be present between the MIB_TCPROW_OWNER_PID array entries in the table member. 
                        // Any access to a MIB_TCPROW_OWNER_PID array entry should assume padding may exist. 
                        //
                        // I have absolutely no idea how to detect such padding, or if .NET handles it automatically if I use PtrToStructure rather than the direct pointer 
                        // manipulation calls this function is now using.
                        //
                        int tableLen = Marshal.ReadInt32(ptrTcpTable);          // Get table row count
                        if (tableLen == 0)
                        {
                            Debug.Assert(false, "How is it possible that the API succeeded and there are really no network connections? Maybe pure IPv6 environment?");
                            return 0;
                        }
                        IntPtr ptrRow = (IntPtr)((long)ptrTcpTable + iOffsetToFirstPort);       // Advance pointer to first Port in the table

                        // Iterate each row of the table, looking to see if localPortInNetworkOrder matches. If it does, return the owningPid
                        for (int i = 0; i < tableLen; ++i)
                        {
                            // Check for matching local port
                            if (iTargetPortInNetOrder == Marshal.ReadInt32(ptrRow))
                            {
                                return Marshal.ReadInt32(ptrRow, iOffsetToPIDInRow);    
                                // Note: the finally clause below will clean up memory
                            }

                            // Move to the next row
                            ptrRow = (IntPtr)((long)ptrRow + iTableRowSize);
                        }
                    }
                    else
                    {
                        FiddlerApplication.Log.LogFormat("GetExtendedTcpTable() returned error #{0}", Marshal.GetLastWin32Error().ToString());
                        return 0;
                    }
                }
                finally
                {
                    // Clean up unmanaged memory block. Call succeeds even if tcpTable == 0.
                    Marshal.FreeHGlobal(ptrTcpTable);
                }
            }
            else
            {
                FiddlerApplication.Log.LogFormat("Initial call to GetExtendedTcpTable() returned error #{0}", Marshal.GetLastWin32Error().ToString());
            }
            return 0;
        }

        /// <summary>
        /// Given a local port number, uses GetExtendedTcpTable to find the originating process ID. 
        /// First checks the IPv4 connections, then looks at IPv6 connections
        /// </summary>
        /// <param name="iTargetPort">Client applications' port</param>
        /// <returns>ProcessID, or 0 if not found</returns>
        private static int FindPIDForPort(int iTargetPort)
        {          
            int iPID = 0;
            try
            {
                iPID = FindPIDForConnection(iTargetPort, AF_INET);
                if ((iPID > 0) || !CONFIG.bEnableIPv6) return iPID;
                return FindPIDForConnection(iTargetPort, AF_INET6);
            }
            catch (Exception eX)
            {
                FiddlerApplication.Log.LogFormat("Fiddler.Network.TCPTable> Unable to call IPHelperAPI function: {0}", eX.Message);
                Debug.Assert(false, "Unable to call IPHelperAPI function" + eX.Message);
            }

            // If we got here, we didn't find the connection; this will occur if the connection is from a remote client.
            // FiddlerApplication.Log.LogFormat("Fiddler.Network.TCPTable.Error> Unable to find process information for port #{0} in table of length {1}", iTargetPort, tcpTableLength);
            return 0;
        }
    }
}

// Win2k Doesn't have iphlpapi.dll that we need, so disable Socket Mapping on that platform
if ((Environment.OSVersion.Version.Major < 6) && (Environment.OSVersion.Version.Minor < 1))
{
    bMapSocketToProcess = false;
}

try
{
    System.Diagnostics.Process.GetProcessById(iPID).ProcessName.ToLower();
}
catch (Exception eX)
{
    Debug.Assert(false, eX.Message);
}

It turns out that looking up a process name with the GetProcessById call can take quite a few milliseconds, and Process ID to Name mappings are fairly stable, so Fiddler maintains a cache of these mappings for 30 seconds.

My talk at this year’s MIX conference is now complete and the video, slides, and references are now available for download. See https://www.fiddler2.com/mix/ for links.

One of demos showed off a simple little extension that can be used to manipulate GeoLocation web service responses, allowing you to easily spoof modern browsers into thinking your computer is somewhere it is not.

Thanks to everyone who attended, and for all of the great input and questions I received. I’m doodling up the early plans for a new FiddlerCore-based utility that some folks might find useful. Expect more news on that topic here in the next few months.

-Eric

I recently got a bug report from a user who was attempting to use Fiddler in a pure-IPv6 environment, where IPv4 is entirely disabled. On startup, he saw the following message:

The problem here is an obscure one, which makes it somewhat interesting. What’s happening is that Fiddler is trying to create an IPv4 listener socket listening on port 8888 locally. However, because this is a pure IPv6 environment, permission to create such a socket is denied.

By default, Fiddler will create an IPv4 listen socket unless the following three criteria are met:

1. The computer supports IPv6 (Vista+ support it by default; various installs are required for XP)
2. The Enable IPv6 option is ticked inside Tools > Fiddler Options > General
3. The Allow remote computers to connect option is set inside Tools > Fiddler Options > Connections

Now, the first two requirements are somewhat self-explanatory, but the third bears some explanation. If only the first two requirements are met, Fiddler will be able to connect to IPv6 sites, but the browser will only be able to connect to Fiddler using IPv4.

The reason that the third requirement exists is that some clients (particularly IE6’s WinINET, and likely others) don’t support IPv6-literal proxy addresses, and hence attempting to poke [::1]:8888 into the system proxy settings would fail for such clients. So, Fiddler cannot safely bind to IPAddress.IPv6Loopback, and will bind to IPAddress.Loopback instead. However, if you enable the Allow remote computers to connect option, Fiddler will instead bind to IPAddress.IPv6Any which allows connecting on both IPv4 and IPv6 addresses on any adapter.

In the future, I may change Fiddler such that the preference fiddler.network.proxy.registrationhostname defaults to localhost which would resolve the IE6 concern as well without the “Allow external clients to connect” option being set. But that option is a fine workaround for now.

Please do note, after setting the Allow remote computers to connect option, you must restart Fiddler. You should get a prompt from the Windows Firewall asking for permission to listen for external connections at that time. You should choose to Allow Fiddler to listen for connections to avoid failures due to the firewall.

-Eric

This morning, someone asked me to look into a site-compatibility problem on a HTML5 demo site. When loading the site into IE9 and IE10, the F12 Developer Tools’ Script Debugger showed the following error:

Now, obviously, IE does support getElementsByTagName, and I confirmed that the page is running in IE9 and IE10 Standards Modes in the respective browsers. The next thing that stood out to me is the filename: jquery-1.5.min.js, which indicates that this is version 1.5 of the popular JQuery JavaScript library. Back in March, Tony Ross celebrated the release of JQuery 1.5.1 on the IEBlog, noting that this was the first version of the library to fully support IE9.

So, it looks like this site might be using an outdated version of the library. But is that what’s causing this script error, and will upgrading to the latest JQuery make the site work?

In an earlier case, I had thought that a site might have been broken because it was using JQuery 1.4.4, but the developer upgraded to the latest JQuery and it didn’t help. So, before I got in touch with the owners of today’s demo site, I wanted to make sure that upgrading JQuery was all they needed to do.

Fortunately, Fiddler makes this task super easy. First, visit Jquery.com and download the JQuery library. If you plan on debugging into JQuery, get the Development version; since I’m hoping a simple upgrade will alone suffice, I downloaded the smaller minified version and saved it to my desktop:

Now, I booted Fiddler, and activated the AutoResponder tab. Using the Add… button, I created a new rule to map requests for JQuery-1.5.min.js to the newly downloaded jquery-1.6.2.min.js file I had just put on my desktop:

Be sure to set the Unmatched requests passthrough option to ensure that Fiddler doesn’t automatically generate 404s for requests that don’t match any of the rules.

Now, I went back to the site in IE and hit CTRL+F5 in the browser to force it to reload all of the content from the network. Fiddler intercepts the request for the older JQuery and returns the newer one instead. Fiddler shows the AutoResponse line in blue in the WebSessions list:

I was in luck-- with the new version of JQuery in place, the site ran correctly without any script errors, and the demo content worked beautifully! I could then confidently contact the site’s owner and request that they update their libraries, knowing that doing so will resolve the compatibility problem.

-Eric

Just under a year ago, I wrote a blog post about how the new “Extended Protection” feature (also known as Channel Binding Tokens or CBT) prevented seamless decryption of certain authenticated HTTPS traffic when Fiddler is running.

The quick recap is that CBT binds a set of NTLM or Kerberos authentication credentials to the “channel” upon which they have been sent. If the server detects that the credentials were sent on a different channel than they were received, then it assumes that a credential reuse attack is in progress and it treats the credentials as invalid.

This is a problem for Fiddler, which uses a man-in-the-middle approach for decrypting HTTPS traffic. This approach results in two HTTPS channels (connections), one from the client to Fiddler (secured by the Fiddler-generated certificate) and one from Fiddler to the server (secured by the server-supplied certificate). When the browser authenticates using the Negotiate package (NTLM or Kerberos), the credentials are bound to the connection to Fiddler. When the credentials are passed through on Fiddler’s connection to a CBT-enabled server, the mismatch of channel identity is detected and the credentials are treated as invalid. As a result, the browser prompts the user endlessly for their credentials:

…and even when correct credentials are supplied, those credentials will not work because the server will continue to reject them due to the channel mismatch.

The only workaround I could offer last year was the option to use Fiddler’s Rules engine or UI to disable decryption of requests bound for CBT-protected servers. This works fine if such servers are only incidentally part of your traffic flow—but what if they’re exactly what you need to see to debug something?

I mentioned in the middle of the article that it was theoretically possible that I could resolve this limitation in a future version of Fiddler--

(It is likely that a future version of Fiddler will be able to debug HTTPS traffic even with CBT enabled, because Fiddler runs on the client and has access to the user's credentials. Fiddler itself can provide a proper response to the server's credential challenge; I only need to update the code for the existing x-AutoAuth flag to use the channel information from the HTTPS connection.)

…but I was soon distracted by other feature requests, and CBT wasn’t broadly deployed enough for this limitation to cause many problems. However, last month, a Fiddler user hit this problem and mentioned it on Twitter, and during some downtime over the weekend I decided to take another crack at this, using the technique I outlined last year. In the “Agile” spirit, I tried “the simplest thing that could possibly work” and was utterly shocked that in just a few lines of code I indeed had addressed this problem.

The new functionality can be found in Fiddler v2.3.6.1 and later, now available in the Fiddler Beta channel.

To activate it, you must configure Fiddler to perform authentication on the client’s behalf. That ensures that the proper channel-binding information is presented to the CBT-protected server. The easiest way to configure Fiddler is to update the FiddlerScript file with a rule that will add your authentication information. Of course, precautions should be taken to ensure that only traffic from your own processes and machine is automatically authenticated-- otherwise anyone who can send traffic through your Fiddler instance will authenticate using your credentials! You also want to ensure that Fiddler only responds to authentication challenges from servers you trust (e.g. on your Intranet) lest an evil server be provided with credential hashes that it may be able to crack.

To configure Fiddler to authenticate on your behalf, click Rules > Customize Rules. Scroll to the OnPeekAtResponseHeaders function and add the following code:

static function OnPeekAtResponseHeaders(oSession: Session)
{        
    // To avoid problems with Channel-Binding-Tokens, this block allows Fiddler
    // itself to respond to Authentication challenges from HTTPS Intranet sites.
    if (oSession.isHTTPS &&
        (oSession.responseCode == 401) &&
        // Only permit auto-auth for local apps (e.g. not devices or remote PCs)
        (oSession.LocalProcessID > 0) &&
        // Only permit auth to sites we trust
        (Utilities.isPlainHostName(oSession.hostname)
        || oSession.host.EndsWith("microsoft.com")) 
        )
    {
        // To use creds other than your Windows login credentials,
        // set X-AutoAuth to "domain\\username:password"
        oSession["X-AutoAuth"] = "(default)";   
        oSession["ui-backcolor"] = "pink";
    }

//... function continues

You will probably want to adjust the two values marked in yellow. The first controls what servers Fiddler is willing to release credentials to—by default, I allow Fiddler to send credentials to any “plain” hostname that does not contain a dot (Intranet sites) and I also allow any site ending in Microsoft.com, because some of our CBT-protected servers have hostnames that contain dots. The second value controls what credentials Fiddler will attempt to use. If you specify explicit credentials in domain\\username:password format (note the double-backslash required by JavaScript), Fiddler will attempt to use those credentials. If you instead specify (default), Fiddler will attempt to use the login credentials of whatever user-account that it is running under. Since I run Fiddler in my own account, I use “(default)”.

After you make these changes and save the file, Fiddler will begin to authenticate on your behalf to the servers' you’ve selected. The final line in the script sets the background color of automatically authenticated responses to pink so that they’re easily visible in your session list.

I hope you find this new feature helpful!

-Eric

PS: While I’m talking about Fiddler and Authentication, I’ll mention two other issues that pop up from time to time:

1. Office clients which use WinHTTP may not properly authenticate with Fiddler running. The reason is that WinHTTP does not authenticate in the same way that WinINET does. The X-AutoAuth trick above should work, or see http://support.microsoft.com/kb/956943 for an alternative workaround involving a registry change to the AuthForwardServerList.
2. When you attempt to authenticate to a website running on your local computer with Fiddler running, the credentials may be rejected. This is because Windows Vista and later contains special code to detect “loopback” credential use and block it when such use is not expected. The X-AutoAuth trick above may work. If not, please see http://support.microsoft.com/kb/926642 for two alternative workarounds involving registry changes. I’ve been told that the DisableLoopbackCheck option works, although either technique should be effective.

Over on StackOverflow, a new Windows 8 user asked how to get Fiddler working with new Windows 8 Metro-style applications. These applications work somewhat differently than classic desktop applications, and require a bit of special configuration work to get Fiddler to work properly.

Fear not, however, Windows 8 and Fiddler get along just fine after a few tweaks. Many of my colleagues have been using Fiddler to debug Metro-style applications over the last few months.

There are three important Windows 8 changes that impact Fiddler:

1. The .NET CLR 2.0 isn’t installed by default
2. Metro-style applications require a specific Capability to communicate with the localhost (where Fiddler runs)
3. Metro-style applications do not respect the per-User Trusted Root Certificates store

Each of these three issues can be worked around to restore the full functionality of Fiddler when running on Windows 8.

DotNet Runtime Version

By default, Windows 8 ships with the Common Language Runtime v4 and the .NET Framework v4.5. Fiddler, however, is currently compiled to run on the v2.0 CLR which is installed on most PCs with the .NET2.0, .NET3.0, and .NET3.5 Frameworks. When you first launch Fiddler on Windows 8, you should see a prompt to install and enable the .NET Framework version 3.5 and the v2.0 CLR:

If for some reason you cannot install the older .NET CLR & Framework (or simply don’t want to), then you can try an beta version of Fiddler targeting the version 4.0 .NET Framework & CLR. This build currently trails the v2-targeted release build by a few bugfixes and features but will be updated periodically. One day, it will likely become the default package, in the same way that the v2-targeted version of Fiddler replaced the v1.1-targeted version back in 2007.

Connecting to Fiddler requires an Application Capability or Loopback Exemption

Windows 8’s Metro-style applications run in isolated processes (AppContainers) where permissions are granted based on a capabilities model. An application only is granted the capabilities which are specified in its package manifest.

[Update 12/10/2011: Please read this post for details on a new utility which simplifies the following steps. ]

Fiddler is a proxy running on your local computer, and for a Metro-style application to send its traffic to Fiddler, the application must either declare the privateNetworkClientServer permission or a Loopback Exemption must be created. Without either of these settings, the App will not be able to connect to the Fiddler proxy running on the local computer, and thus Fiddler will not be able to see its traffic.

CheckNetIsolation LoopbackExempt allows the developer to set up Fiddler or his debugging/testing environment without modifying the capabilities assigned to the AppContainer. Generally speaking, using the CheckNetIsolation tool should be preferred to adding the privateNetworkClientServer capability, as doing so is less impactful to the behavior of the App you’re testing. There was a recent talk that describes how to use CheckNetIsolation with Fiddler-- The talk is “Debugging connected Windows 8 apps” and it’s available for viewing here. Minutes 27 to 34 talk about Fiddler, and minutes 19 to 34 discuss use of the CheckNetIsolation tool. To exempt your application’s AppContainer, open an Administrative Command Prompt, and type CheckNetIsolation LoopbackExempt –a -n=AppContainer’sFullName

You can learn more about assigning capabilities to your application by looking for the text To add capabilities to an app inside this whitepaper.

The Windows 8 Firewall blocks Modern Applications from connecting to Loopback

[Update 12/10/2011: This step should not be needed in the latest Fiddler v4 builds]

You will also need to allow loopback connections through the firewall. The simplest way to do that is to click Tools > Fiddler Options and tick the “Allow remote computers to connect” option, restarting Fiddler when complete.

To decrypt HTTPS, Fiddler’s Root Certificate must be placed in the Machine's Trusted Root store

Fiddler’s HTTPS-decryption feature relies on a man-in-the-middle approach to decrypting HTTPS traffic. Upon enabling the feature, Fiddler provides the option to copy its self-signed root certificate into your per-User Trusted store:

However, Metro-style applications will not respect any root certificates that are only installed in the per-User Trusted Root Certification Authorities store. If a Metro-style application encounters a certificate chain that doesn’t lead to the Local Machine’s Trusted Root Certification Authorities store, then the trust chain will be deemed invalid and your HTTPS request will fail.

To address this, recent versions of Fiddler include code that will offer to place the Fiddler Root Certificate into the machine-wide Trusted Root Certification Authorities store when Fiddler detects that it is running on Windows 8. After the prompts above, you’ll see the following prompts:

If you choose Yes and Yes in these prompts, then Fiddler should be able to successfully decrypt the HTTPS traffic from your Metro-style application. Note: If you want Fiddler to have the machine trust the root certificate on earlier Windows versions, set the preference named fiddler.certmaker.offermachinetrust to True.

Please let me know if you have any questions, and thank you for building on Windows 8!

-Eric Lawrence

HTML5 introduces the concept of an Application Cache, which allows a web developer to provide a manifest of pages that should be cached to permit offline use. You can see an instance of AppCache over on the IETestDrive site; if you examine the markup, you can find that the HTML tag contains a manifest attribute which specifies the URI of the AppCache manifest.

The Manifest specifies which resources the browser should download into the AppCache and which should always be retrieved from the network. You can learn more about AppCache from numerous online resources; a few of those include:

• http://diveintohtml5.org/offline.html
• http://appcachefacts.info/
• http://www.html5rocks.com/en/tutorials/appcache/beginner/

Manifests are simply text files and you can create them with your text editor of choice. However, this process can be somewhat tedious and begs for an automated solution.

Fiddler includes a pluggable import/export architecture that makes it easy to export sessions from Fiddler into any format you can imagine. Yesterday, I decided to add an AppCache Manifest exporter, which you can find in Fiddler v2.3.6.2. Fiddler’s new HTML5 AppCache Manifest Exporter makes the process of generating a manifest a straightforward exercise:

1. Clear your browser cache.

2. Start Fiddler.

3. Load your website in the browser.

4. In Fiddler, click File > Export > All Sessions.

5. In the format selection dialog box, pick HTML5 AppCache Manifest

6. Click Next. In the Adjust AppCache Manifest box, check any resources that you wish to exclude from the CACHE section of the Manifest; those will be added to the NETWORK section of the manifest:

7. If you’d like, use the text box at the bottom to specify a Base URL if you would like the URLs to be relative to the manifest. For instance, in the example above, I will place the manifest in the folder http://ie.microsoft.com/testdrive, so I will use that as a base URL.

8. Click the Save button to generate and display the manifest.

9. If you’re happy with the manifest, save it to your web server in the appropriate location. Ensure that your web server is configured to return the Manifest file type with Content-Type: text/cache-manifest

10. In your web page, ensure that your page will run in Standards mode (e.g. using the HTML5 doctype) and add the manifest attribute pointing to your application’s manifest.

Do keep in mind that when the browser is using App-Cached content, those resources will be reused from the cache and not pulled from the network (until the cache is expired). Because the content is cached locally, you will not see requests to re-download that content inside Fiddler. If you wish to use Fiddler to modify such content, you will need to clear your browser’s cache so that the browser is forced to re-download the content from the network on next use.

Please let me know if you encounter any problems with the new AppCache Manifest Exporter. As this is very new code, I’m sure there are a few bugs. :-)

Thanks,

Eric Lawrence

For this morning’s post, I wanted to give everyone a quick picture of some of the data I've collected about Fiddler usage, how I plan to respond, and how I will continue to learn more.

I’ve recently had the chance to run some surveys and spend some time with developers who are using Fiddler, and it has been an extremely eye-opening process. It became quickly plain that most folks are barely scratching the surface of what they can do with Fiddler, and they often are not using the tool very efficiently.

I plan on improving this in two ways: first, by making tweaks to Fiddler itself to improve its ease-of-use, and secondly, by authoring the first-ever book on Fiddler. It looks like many of you would be interested:

… and I’m confident that the book format would allow me to convey information in a more coherent, deeper way than my current series of blog posts, help pages, Wiki’s etc.

I’ve gotten about 35 pages written so far, and I’m presently targeting about 250 pages for the finished product. Currently, the book is organized thusly:

• Using Fiddler (covers core concepts, UI, major features, major extensions)
• Extending Fiddler (covers object model, script, building Inspectors, building Extensions)
• Building on FiddlerCore (covers building applications on the FiddlerCore library)
• Solving problems with Fiddler (explains, step-by-step, how to accomplish different scenarios with Fiddler).

One interesting challenge in me writing a book on a Fiddler is that it’s allowing me to take a good hard look at every piece of Fiddler as I write about it; I’ve already made dozens of small fixes as I’ve been writing, which is great for Fiddler but bad for my throughput as an author. :-)

In addition to the poll above, I’ve run a few other broad surveys about Fiddler usage. I was surprised to learn that Application and WebService testing is more than twice as popular as Web Site testing, which suggests to me that Web Developers haven’t yet discovered the power that Fiddler provides beyond browser-integrated tools. This is definitely an opportunity to help folks get more out of Fiddler.

Lastly, I’ve run a longer survey about Fiddler which will be up for another week or so to collect deeper information about how Fiddler is getting used.

It looks like most at least 80% of Fiddler users use it every month:

The Fiddler Tutorial videos have tons of information on Fiddler that is hard to find elsewhere; unfortunately, it seems that few folks have seen them:

Much of Fiddler’s power derives from the fact that it’s a web debugging platform, with the ability to enhance it with extensions. Unfortunately, many folks seem to be missing out, although I have noticed that downloads of the Syntax-Highlighting extensions skyrocketed after I put an advertisement for it directly within Fiddler:

Lastly, the survey asked users for information on what I should add to Fiddler next. Rather predictably, most of the proposed features are already available in Fiddler, although some are difficult to find or understand. There were also a number of great suggestions that I’ve been picking off one-by-one over the last month or so; I have a few more to do.

As I’ve gone through this process, it’s become plain that despite Fiddler’s huge user-base (over ten-thousand downloads per day!) I know very very little about how the tool is really getting used in the wild. Fiddler’s telemetry is currently limited to the download count, and the number of Web Service calls for version checking that happen per day (which can be used as a rough approximation of the number of Fiddler sessions).

Microsoft solves this problem with the Customer Experience Improvement Program, a mechanism by which the user can opt-in to providing anonymous data to Microsoft on how products are being used in the wild.

Non-Microsoft tools like Fiddler cannot take advantage of Microsoft’s CEIP service, but I recently encountered RedGate’s SmartAssembly, a very easy-to-integrate toolkit for instrumenting applications with telemetry for feature usage reporting. I’m happy to announce that an upcoming version of Fiddler will introduce the “Fiddler Improvement Program,” enabling you to anonymously send me data about how you’re using Fiddler. This data will help ensure that I’m prioritizing my limited development time effectively. It will also ensure that I understand the mix of system configurations in the wild (e.g. could I move to .NETv4 by default? How many Fiddler users have multiple monitors? How many have displays smaller than 1280x1024? What percentage of Fiddler users have 2gb or less of RAM?)

If you see this prompt, please consider opting in!

Thanks for all the feedback!

-Eric

Update 8/4/2012: Review some of Fiddler's telemetry data in my new blogpost.

Recently, a colleague asked me to look at a network capture in which a .NET client application’s communication with a web service was not meeting their performance goals. In particular, he noted that this was primarily a problem on high-latency networks; each of the dozens of requests took hundreds of milliseconds, even when existing connections were being reused.

Even when he took a capture locally on a fast network, most of the time was spent in an unexpected place:

You can see that the client and server were both reusing connections (the session above took place on a client connection that was established for an earlier session 4 seconds earlier, and the server connection was established 63 seconds earlier).  In particular, note that the entire session took 468 milliseconds, but the time between ClientBeginRequest and ClientDoneRequest is 359 milliseconds, nearly 77% of the total time. As Fiddler is a locally-running proxy, and Fiddler only took 93 milliseconds to send the request to the remote server and get back the first byte of the response, it’s surprising that it took over three times as long to actually read the request from the client program on the same computer.

So, why is the client taking so long to send the request to Fiddler?

I noticed that each of these requests was a HTTP POST, uploading a small SOAP XML body, each just over 1kb. So, it’s unlikely that the client was taking a long time to generate the request body or load it off of the disk. However, each request did contain an Expect header.

Section 8.2.3 of RFC 2616 explains what the Expect header is used for—the basic idea is that a client sending a POST body may send an Expect: 100-continue request header to the server. The server is expected to immediately evaluate the (incomplete) request’s headers, and if they seem reasonable (e.g. of a MIME-type the server supports, and with an appropriate Content-Length) the server sends back a non-final response header set with a status code of 100, indicating that the client’s plan to send this body is acceptable. Alternatively, if the headers suggest that the client’s pending post is unacceptable (e.g. the client needs to authenticate itself), the server may immediately return an error code (e.g. a HTTP/401 with an Authentication challenge), saving bandwidth by allowing the client to abort the request before sending the body.

In the case where the request is acceptable, the client receives the HTTP 100 response and then transmits the body as it had promised. The traffic for this scenario looks^[1] somewhat like this:

Client->Server
POST /Page HTTP/1.1
Content-Type: text/plain
Content-Length: 26 
Expect: Continue

Server->Client
HTTP/1.1 100 Continue
Server: Microsoft-IIS/7.5
Date: Wed, 02 Nov 2011 18:51:56 GMT

Client->Server
Here is my request body...

Server->Client
HTTP/1.1 200 OK
Server: Microsoft-IIS/7.5
Date: Wed, 02 Nov 2011 18:51:58 GMT
Content-Type: text/html

...

Importantly, HTTP’s Expect feature isn’t supported by all clients and servers (e.g. Internet Explorer doesn’t use Expect, instead using an authentication-only heuristic) and thus a client generally will only wait a short period for a reply from the server before proceeding to transmit the request body anyway. In the .NET Framework implementation, that wait is capped at 350 milliseconds.

The Fiddler Web Debugger buffers all HTTP requests completely before transmitting the request to the server (in order to enable request tampering), so the .NET application was always waiting the full 350 milliseconds between sending the request headers and the request body when it was running behind Fiddler. In my colleague’s production scenario, he wasn’t running Fiddler, but the connection from the client to the server was high-latency, so the client didn’t rapidly get back the HTTP/100 response from the server, and thus it was forced to wait hundreds of additional milliseconds on each request.

There are some cases in which the Expect feature is useful—these are the scenarios where the client is expecting to transfer a large request body (e.g. a multi-megabyte upload). In my colleague’s case, using Expect isn’t really appropriate, because the requests are small enough that performance is significantly impacted and the Expect feature provides no real savings. The .NET Framework allows callers to disable the Expect behavior using the ServicePointManager.Expect100Continue property (or by setting a property in the application’s manifest^[2] ). You can read a bit more about this property here.

If you'd simply like to have Fiddler itself return the 100 Continue, you can do that by clicking Rules > Customize Rules. Add the following function inside the Handlers class:

static function OnPeekAtRequestHeaders(oSession: Session) {
   if (oSession.HTTPMethodIs("POST") && oSession.oRequest.headers.ExistsAndContains("Expect", "continue"))
   {
     if (null != oSession.oRequest.pipeClient)
     { 
       oSession["ui-backcolor"] = "lightyellow";
       oSession.oRequest.headers.Remove("Expect");
       oSession.oRequest.pipeClient.Send(System.Text.Encoding.ASCII.GetBytes("HTTP/1.1 100 Continue\r\nServer: Fiddler\r\n\r\n")); 
     } 
   }
}

-Eric

^[1] Note that HTTP 100 Responses are handled specially by Fiddler because they are an odd architectural quirk of HTTP—they’re non-final responses, which means that a server that sends the 100 Continue will then send a final response header set (usually a HTTP/200). Representing a single session with multiple sets of response headers would be both complicated and confusing, so Fiddler does not attempt to do so. By default (preference fiddler.network.streaming.leakhttp1xx default: True), Fiddler will return any 100 responses it receives to the client and makes note of this using a Session Flag named x-fiddler-Stream1xx.

^[2] Add the following to clientappname.exe.config:

  <settings>
    <servicePointManager expect100Continue="false" /> 
  </settings>

Much work remains, but the basic architecture is in place for Fiddling with both secure and insecure HTML5 WebSockets:

Back in September, I blogged about the configuration steps required to debug Windows 8 “Metro-style” apps using Fiddler. Since that post was originally written, I’ve made available a new version of Fiddler which runs natively on the .NETv4 Framework, enabling Windows 8 users to run Fiddler without installing older versions of the Framework.

As I mentioned in that post, Metro-style applications run inside isolated processes known as “AppContainers,” and by default, AppContainers are forbidden from sending network traffic to the local computer (loopback). This is, of course, problematic when debugging with Fiddler, as Fiddler is a proxy server which runs on the local computer. The post went on to explain how the CheckNetIsolation tool can be used to permit an AppContainer to send traffic to the local computer. However, using CheckNetIsolation is pretty cumbersome—it requires that you know the AppContainer’s name or security ID, and you must configure each AppContainer individually. To resolve those difficulties, I have built a GUI tool that allows you to very easily reconfigure an AppContainer to enable loopback traffic. This tool requires Windows 8 and runs on the .NET Framework v4. When launched, the utility scans your computer’s AppContainers and displays them in a list view. Each entry has a checkbox to the left of it, indicating whether the AppContainer may send loopback traffic. You can toggle these checkboxes individually, or use the buttons at the top to set all of the checkboxes at once. Click Save Changes to commit the configuration changes you’ve made, or click Refresh to reload the current configuration settings.

After you install the EnableLoopback Utility, a new “Win8 Loopback Exemptions” item is added to Fiddler’s Tools menu; clicking this item launches the utility. To make changes to the exemption list, you must elevate to Administrator.

-Eric

PS: For the technically-inclined, this tool relies on calling the new Network Isolation APIs introduced with Windows 8. Their .NET declarations (as of the BUILD conference) are as follows:

// Call this API to enumerate all of the AppContainers on the system 

[DllImport("FirewallAPI.dll")] 

internal static extern uint NetworkIsolationEnumAppContainers(out uint pdwCntPublicACs, out IntPtr ppACs); 

// Call this API to free the memory returned by the Enumeration API 

[DllImport("FirewallAPI.dll")] 

internal static extern void NetworkIsolationFreeAppContainers(IntPtr pACs); 

// Call this API to load the current list of Loopback-enabled AppContainers

[DllImport("FirewallAPI.dll")] 

internal static extern uint NetworkIsolationGetAppContainerConfig(out uint pdwCntACs, out IntPtr appContainerSids); 

// Call this API to set the Loopback-exemption list 

[DllImport("FirewallAPI.dll")]

internal static extern uint NetworkIsolationSetAppContainerConfig(uint pdwCntACs, SID_AND_ATTRIBUTES[] appContainerSids); 

// Use this API to convert a string SID into an actual SID 

[DllImport("advapi32.dll", SetLastError=true)]

internal static extern bool ConvertStringSidToSid(string strSid, out IntPtr pSid); 

// Use this API to convert a string reference (e.g. "@{blah.pri?ms-resource://whatever}") into a plain string 

[DllImport("shlwapi.dll", CharSet=CharSet.Unicode, ExactSpelling=true)] 

internal static extern int SHLoadIndirectString(string pszSource, StringBuilder pszOutBuf, int cchOutBuf, IntPtr ppvReserved);

I’m a long-time ThinkPad fan and owner—the keyboards are sublime, and I love the TrackPoint interface. What I don’t love is the ThinkPad driver software, which doesn’t send the standard WM_MOUSEWHEEL messages to the system. That bug means that many .NET controls don’t scroll properly. Over the years, I’ve fixed many of the controls in Fiddler to workaround the problem, but not all of them. For instance, ThinkPad users will notice that when they try to middle-click-scroll the HexView controls, they get the scroll-cursor, but no scrolling as they move the cursor:

Over on his blog, Mark Rideout explains the root cause of Lenovo’s scrolling problem and explains how .NET developers can workaround the bug by adding hidden scrollbar controls to trick the driver software. I’ll be looking at using this trick in a future version of Fiddler. For now, a cumbersome but effective configuration tweak to the Lenovo driver settings allows end-users to get scrolling working properly in Fiddler and most other applications.

Start Notepad using the Run As Administrator option. Open the file C:\program files\lenovo\trackpoint\tp4table.dat (note that the tp4table.dat file may be in another path on your computer). Add the lines in blue under the existing lines in grey.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Pass 1 rules (These rules run last)
;
; If no scrolling method was found with the builtin logic, then
; these rules are checked.
;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Fiddler
*,*,Fiddler.exe,*,*,*,WheelStd,1,9

Restart your system to force the driver software to reload its settings, and you should find that middle-click-scrolling works properly in Fiddler.

Until next time…

-Eric

The new Fiddler Beta has a few useful tricks. First, you can copy a DataURI to your clipboard (from Fiddler itself, or anywhere else):

and use the new Edit > Paste Files as Sessions command:

… to instruct Fiddler to parse the DataURI and create a new Session for it. You can then use Fiddler’s inspectors to examine the resulting object:

Next, the WebView Inspector has been enhanced to render previews of WOFF font, MP3 audio, and H264 video responses, if you have IE9+ installed:

Lastly, Fiddler’s AutoResponder now permits you to use Regular Expression Group Replacements to map one URL to another. For instance, this rule:

…will convert http://somesite.com/assets/Test1.gif to http://www.example.com/mockup/Test1.gif. This capability makes it even easier to replace content using Fiddler’s AutoResponder.

This new build of Fiddler has a ton of other fixes and performance improvements. I hope you enjoy it!

-Eric

With the recent kerfuffle about sites providing misleading P3P statements, I decided to throw together a quick Fiddler add-on that displays privacy information simply in the Fiddler UI.

Install the new Privacy Scanner add-on and Fiddler will gain a new top-level menu named Privacy.

The menu has two options. The Enabled option controls whether the add-on does anything at all; it’s a good practice for Add-ons to offer a simple way to turn them off when they’re not wanted.

When the add-on is enabled, it will add a Privacy Info column to the session list and will flag HTTP/HTTPS responses which set cookies. Evaluation of any P3P statements that come along with those cookies will change the session's background color:

Sessions that send a satisfactory P3P policy are shown in green. Sessions that set a cookie without a P3P policy are yellow; in the default IE settings, these cookies will not be sent to a 3rd party context. Sessions that send a P3P policy that does not permit use of the cookie in a 3rd party context are rendered in orange. Sessions that send invalid P3P header tokens will have a red background.

When the menu option Rename P3P header if invalid is checked, if a session presents a P3P statement that is malformed, that P3P header will be renamed to Malformed-P3P to prevent the browser from interpreting it as the P3P 1.0 specification suggested (e.g. ignoring the unknown tokens). You can see the difference when you load this test page; the cookies with bogus compact policies will be dropped in a 3rd party context.

For folks looking to build their own Fiddler extension, the code is provided below.

-Eric

using System;

using System.Collections;

using System.Globalization;

using System.Collections.Generic;

using System.Windows.Forms;

using System.Text;

using Fiddler;

using System.IO;

using System.Diagnostics;

using Microsoft.Win32;

using System.Reflection;

using System.Text.RegularExpressions;

[assembly: Fiddler.RequiredVersion("2.3.9.0")]

[assembly: AssemblyVersion("1.0.1.0")]

[assembly: AssemblyTitle("PrivacyScanner")]

[assembly: AssemblyDescription("Scans for Cookies and P3P")]

[assembly: AssemblyCompany("Eric Lawrence")]

[assembly: AssemblyProduct("PrivacyScanner")]

public class TagCookies : IAutoTamper2

{

    private bool bEnabled = false;

    private bool bEnforceP3PValidity = false;

    private bool bCreatedColumn = false;

    private System.Windows.Forms.MenuItem miEnabled;

    private System.Windows.Forms.MenuItem miEnforceP3PValidity;

    private System.Windows.Forms.MenuItem mnuCookieTag;

    public void OnLoad()

    {

        /*

 * NB: You might not get called here until ~after~ one of the AutoTamper methods was called.

 * This is okay for us, because we created our mnuContentBlock in the constructor and its simply not

 * visible anywhere until this method is called and we merge it onto the Fiddler Main menu.

 */

        FiddlerApplication.UI.mnuMain.MenuItems.Add(mnuCookieTag);

    }

    public void OnBeforeUnload() {  /*noop*/   }

    private void InitializeMenu()

    {

        this.miEnabled = new System.Windows.Forms.MenuItem("&Enabled");

        this.miEnforceP3PValidity = new System.Windows.Forms.MenuItem("&Rename P3P header if invalid");

        this.miEnabled.Index = 0;

        this.miEnforceP3PValidity.Index = 1;

        this.mnuCookieTag = new System.Windows.Forms.MenuItem();

        this.mnuCookieTag.MenuItems.AddRange(new System.Windows.Forms.MenuItem[] { this.miEnabled, this.miEnforceP3PValidity });

        this.mnuCookieTag.Text = "Privacy";

        this.miEnabled.Click += new System.EventHandler(this.miEnabled_Click);

        this.miEnabled.Checked = bEnabled;

        this.miEnforceP3PValidity.Click += new System.EventHandler(this.miEnforceP3PValidity_Click);

        this.miEnforceP3PValidity.Checked = bEnforceP3PValidity;

    }

    public void miEnabled_Click(object sender, EventArgs e)

    {

        miEnabled.Checked = !miEnabled.Checked;

        bEnabled = miEnabled.Checked;

        this.miEnforceP3PValidity.Enabled = bEnabled;

        if (bEnabled) { EnsureColumn(); }

        FiddlerApplication.Prefs.SetBoolPref("extensions.tagcookies.enabled", bEnabled);

    }

    public void miEnforceP3PValidity_Click(object sender, EventArgs e)

    {

        miEnforceP3PValidity.Checked = !miEnforceP3PValidity.Checked;

        bEnforceP3PValidity = miEnforceP3PValidity.Checked;

        FiddlerApplication.Prefs.SetBoolPref("extensions.tagcookies.EnforceP3PValidity", bEnforceP3PValidity);

    }

    private void EnsureColumn()

    {

        if (bCreatedColumn) return;

        FiddlerApplication.UI.lvSessions.AddBoundColumn("Privacy Info", 1, 120, "X-Privacy");

        bCreatedColumn = true;

    }

    public TagCookies()

    {

        this.bEnabled = FiddlerApplication.Prefs.GetBoolPref("extensions.tagcookies.enabled", false);

        this.bEnforceP3PValidity = FiddlerApplication.Prefs.GetBoolPref("extensions.tagcookies.EnforceP3PValidity", true);

        InitializeMenu();

        if (bEnabled) { EnsureColumn(); } else { this.miEnforceP3PValidity.Enabled = false; }

    }

    private void SetP3PStateFromHeader(string sValue, ref P3PState oP3PState)

    {

        if (string.IsNullOrEmpty(sValue))

        {

            return;

        }

        string sUnsatCat = String.Empty;

        string sUnsatPurpose = String.Empty;

        sValue = sValue.Replace('\'', '"');

        string sCP = null;

        Regex r = new Regex("CP\\s?=\\s?[\"]?(?<TokenValue>[^\";]*)");

        Match m = r.Match(sValue);

        if (m.Success && (null != m.Groups["TokenValue"]))

        {

            sCP = m.Groups["TokenValue"].Value;

        }

        if (String.IsNullOrEmpty(sCP))

        {

            return;

        }

        // Okay, we've got a compact policy token.

        oP3PState = P3PState.P3POk;

        string[] sTokens = sCP.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);

        foreach (string sToken in sTokens)

        {

            // Reject clearly invalid tokens...

            if ((sToken.Length < 3) || (sToken.Length > 4))

            {

                oP3PState = P3PState.P3PMalformed;

                return;

            }

            if (",PHY,ONL,GOV,FIN,".IndexOf("," + sToken + ",", StringComparison.OrdinalIgnoreCase) > -1)

            {

                sUnsatCat += (sToken + " ");

                continue;

            }

            if (",SAM,OTR,UNR,PUB,IVA,IVD,CON,TEL,OTP,".IndexOf("," + sToken + ",", StringComparison.OrdinalIgnoreCase) > -1)

            {

                sUnsatPurpose += (sToken + " ");

                continue;

            }

            // TODO: Look up the token in the complete collection and check validity

        }

        // If a cookie contains an unsatisfactory purpose and an unsatisfactory category, mark it

        // http://msdn.microsoft.com/en-us/library/ie/ms537343(v=vs.85).aspx#unsatisfactory_cookies

        if ((sUnsatCat.Length > 0) && (sUnsatPurpose.Length > 0))

        {

            if (oP3PState == P3PState.P3POk)

            {

                oP3PState = P3PState.P3PUnsatisfactory;

            }

        }

    }

    private enum P3PState

    {

        NoCookies,

        NoP3PAndSetsCookies,

        P3POk,

        P3PUnsatisfactory,

        P3PMalformed

    }

    public void OnPeekAtResponseHeaders(Session oSession) 

    {

        if (!bEnabled) return;

        P3PState oP3PState = P3PState.NoCookies;

        if (!oSession.oResponse.headers.Exists("Set-Cookie"))

        {

            return;

        }

        oP3PState = P3PState.NoP3PAndSetsCookies;

        if (oSession.oResponse.headers.Exists("P3P"))

        {

            SetP3PStateFromHeader(oSession.oResponse.headers["P3P"], ref oP3PState);

        }

        switch (oP3PState)

        {

            case P3PState.P3POk:

                oSession["ui-backcolor"] = "#ACDC85";

                oSession["X-Privacy"] = "Sets cookies & P3P";

                break;

            case P3PState.NoP3PAndSetsCookies:

                oSession["ui-backcolor"] = "#FAFDA4";

                oSession["X-Privacy"] = "Sets cookies without P3P";

                break;

            case P3PState.P3PUnsatisfactory:

                oSession["ui-backcolor"] = "#EC921A";

                oSession["X-Privacy"] = "Sets cookies; P3P unsatisfactory for 3rd-party use";

                break;

            case P3PState.P3PMalformed:

                oSession["ui-backcolor"] = "#E90A05";

                if (bEnforceP3PValidity)

                {

                    oSession.oResponse.headers["MALFORMED-P3P"] = oSession.oResponse.headers["P3P"];

                    oSession["X-Privacy"] = "MALFORMED P3P: " + oSession.oResponse.headers["P3P"];

                    oSession.oResponse.headers.Remove("P3P");

                }

                break;

        }

    }

    public void AutoTamperRequestBefore(Session oSession) { }

    public void AutoTamperRequestAfter(Session oSession){ /*noop*/ }

    public void AutoTamperResponseAfter(Session oSession) {/*noop*/}

    public void AutoTamperResponseBefore(Session oSession) { /*noop*/ }

    public void OnBeforeReturningError(Session oSession) {/*noop*/}

}

Recently, a handful of folks have emailed me complaining that some HTTPS sites cannot be reached by their .NET programs or from any program when Fiddler is configured to decrypt traffic. Notably, these users only have problems when running on Windows Vista or later.

So, what’s going on?

When I need to troubleshoot issues that occur below Fiddler, NetMon is a good choice. Taking a look at the HTTPS handshake, the problem is immediately apparent:

This server is returning a TLS Encrypted Alert at the beginning of the handshake. The alert severity is 1 meaning WARNING, and the code is 112 (0x70) which is the unrecognized_name alert. With this alert, the server is indicating that it does not recognize the hostname provided by the server’s Server Name Indication TLS extension. This warning is only encountered when the client is running on Windows Vista and later because the SChannel HTTPS stack on Windows XP doesn’t support SNI and other TLS extensions.

The server’s behavior is explicitly not recommended according to the standards:

If the server understood the ClientHello extension but does not recognize the server name, the server SHOULD take one of two actions: either abort the handshake by sending a fatal-level unrecognized_name(112) alert or continue the handshake. It is NOT RECOMMENDED to send a warning-level unrecognized_name(112) alert, because the client's behavior in response to warning-level alerts is unpredictable. If there is a mismatch between the server name used by the client application and the server name of the credential chosen by the server, this mismatch will become apparent when the client application performs the server endpoint identification, at which point the client application will have to decide whether to proceed with the communication.

The server’s behavior is problematic because the .NET Framework’s HTTPS implementation is unable to ignore this warning and it is not able to automatically “fail over” to an earlier protocol version, leading to a  lengthy delay (~80 seconds) before a connection failure in Fiddler or other .NET applications.

To enable Fiddler to workaround sites with this problematic configuration, click Rules > Customize Rules. Scroll to the OnBeforeRequest method and add the following block:

if (oSession.HTTPMethodIs("CONNECT") && oSession.HostnameIs("BuggySite.com"))
{
  oSession["x-OverrideSslProtocols"] = "ssl3";
  FiddlerApplication.Log.LogString("Legacy compat applied for inbound request to BuggySite.com");
}

Save the script and this block will force Fiddler to handshake with the server using only the SSLv3 protocol, which does not send TLS extensions, avoiding the problem.

To apply the same workaround in your .NET application without Fiddler, use the same trick: ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;

-Eric