Hosting a proxy in an ASP.NET or SharePoint application - limits of NTLM (double hop issue) and how to overcome

Hosting a proxy in an ASP.NET or SharePoint application - limits of NTLM (double hop issue) and how to overcome

ISSUE: If you host a proxy page (ASHX) in ASP.NET or in SharePoint, and the target server is a different machine, and the target server requires Windows Authentication, then you should know that NTLM has a restriction, where it cannot pass credentials on the 'second hop' from the proxy to the target server, unless it has an explicit username and password.

this is called the 'double hop NTLM issue' and is better documented elsewhere:
http://blogs.msdn.com/b/besidethepoint/archive/2010/05/09/double-hop-authentication-why-ntlm-fails-and-kerberos-works.aspx

note: if your application uses Negotiated (Kerberos) Authentication, and Kerberos is correctly setup, then my understanding is that Kerberos can handle the second hop from the proxy to the target server.

if you *must* use NTLM (for example, if you must deploy to an existing SharePoint Web App), then there are possible workarounds:

1. use explicit username/password, that are set on the request:

 System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)WebRequest.Create(uri);  
 var creds = new NetworkCredential(username, password);  
 req.Credentials = creds; //use explicit username + password, to enable NTLM to perform a second hop  

2. in SharePoint 2010, make the code which sends the request, run under Elevated Priviledges (SPSecurity.RunWithElevatedPrivileges) (see references at end of this post).  According to the referenced article, this should NOT work (it should hit the same issue with double hop & NTLM) but in my testing it DOES work, so it is worth a try.

The article referenced at the end of this post gives a set of Dos and Don'ts when using Elevated Priviledges.
There are also some steps needed, to make sure your ASHX page can access the server session.
For reference, here is some commented example source code:

   /// <summary>  
   /// Forwards requests to an ArcGIS Server REST resource.   
   /// This is necessary so that ArcGIS Server will respond to requests that contain longer queries (more than 2048 chars)  
   ///  
   /// ref: http://help.arcgis.com/en/webapi/javascript/arcgis/help/jshelp_start.htm#jshelp/ags_proxy.htm  
   /// </summary>  
   public partial class MyArcGisProxy : IHttpHandler, IReadOnlySessionState //ASHX implements IReadOnlySessionState in order to be able to read from session  
   {  
     /// <summary>  
     /// forward given request to the request's server.  
     /// </summary>  
     public void ProcessRequest(HttpContext context)  
     {  
       try  
       {  
         HttpResponse response = context.Response;  
         //SPSecurity.RunWithElevatedPrivileges()  
         //  
         //note: this is to circumvent this issue: "double hop issue NTLM" - http://blogs.msdn.com/b/besidethepoint/archive/2010/05/09/double-hop-authentication-why-ntlm-fails-and-kerberos-works.aspx  
         //  
         //note: this works even with impersonate=true in the web.config !  
         SPSecurity.RunWithElevatedPrivileges(delegate()  
         {  
           //note: this code will only work, if SPContext.Current is NOT null, which means the proxy URL *must* be under the SP site (e.g. .../sites/mySite/.../proxy.ashx)  
           //  
           //note: in order for RunWithElevatedPrivileges to work, we *must* create a SPSite  
           //ref: http://extreme-sharepoint.com/2012/05/30/impersonation-elevation-of-privileges/  
           //  
           //if we open the SPList programatically via URL, then we will need to dispose of the SPSite.  
           using (SPSite elevatedSite = new SPSite(SPContext.Current.Site.ID))  
           {  
             using (SPWeb elevatedWeb = elevatedSite.OpenWeb(SPContext.Current.Web.ID))  
             {  
               // Perform administrative actions by using the elevated site and web objects.   
               // elevatedWeb.CurrentUser.LoginName gives SHAREPOINTsystem   
               // WindowsIdentity.GetCurrent().Name gives Application pool Windows account(ContsoAdmin1)  
               // Hence, Both SharePoint Security context and Windows Security context are changed.  
               System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)WebRequest.Create(uri);  
               //the following approach works, WITHOUT RunWithElevatedPrivileges(), when the AppPool account is a DOMAIN account (as with SP)  
               //but only when impersonate is OFF in web.config (ASP.NET + SP).  
               //  
               //it also works, with RunWithElevatedPrivileges() (even when impersonation is ON in web.config).  
               req.Credentials = CredentialCache.DefaultCredentials;  
               req.ImpersonationLevel = TokenImpersonationLevel.Delegation;  
               //to turn off caching: req.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);  
               req.Method = context.Request.HttpMethod;  
               req.ServicePoint.Expect100Continue = false;  
               req.Referer = SPHttpUtility.HtmlEncode(context.Request.Headers["referer"]);  
               req.Method = "GET";  
               // Send the request to the server  
               System.Net.WebResponse serverResponse = null;  
               try  
               {  
                 serverResponse = req.GetResponse();  
               }  
               catch (System.Net.WebException webExc)  
               {  
                 response.StatusCode = 500;  
                 response.StatusDescription = webExc.Status.ToString();  
                 response.Write(webExc.ToString());  
                 response.Write(webExc.Response);  
                 response.End();  
                 return;  
               }  
               // Set up the response to the client  
               if (serverResponse != null)  
               {  
 ...  
 ...  
 ...  
 }  
 }); //end of elevated priviledges block  

3. move the proxy, to be on the target server. [UNTESTED]
my understanding, is that if you host the proxy on the target server, then this basically means the second hop is on the same box as the proxy, and so NTLM should be capable of performing a successful authentication.  note: I have not been able to test this option.  perhaps also using the AppPool (DefaultCredentials) would also be neccessary.

4. use the AppPool account to perform the request.

 //the following approach works,  
 //when Impersonation is OFF in web.config (ASP.NET + SP).  
 //OR when run inside SPSecurity.RunWithElevatedPrivileges()  
 req.Credentials = CredentialCache.DefaultCredentials;  
 req.ImpersonationLevel = TokenImpersonationLevel.Delegation;  

note: this only worked for me, when web.config was set to have impersonation = "false"
For some reason I don't understand, setting  impersonation = "false"
seems to get around the double hop NTLM issue, when the AppPool account is used.  Presumably this allows Kerberos Authentication to be used.
note: in SharePoint 2010, using RunWithElevatedPriviledges() allowed this to work, WITHOUT having to set impersonation = "false".

note: in case it helps anyone (or myself, in the future!) then here is a related issue I documented in StackOverflow, with a workaround for SharePoint 2010:
http://stackoverflow.com/questions/13292513/ashx-sends-httpwebrequest-with-impersonation-on-which-works-to-urls-on-same-ser

references:
re: SharePoint and Elevation of Priviledges (SPSecurity.RunWithElevatedPrivileges)
Impersonation in SharePoint : An Extreme Overview
http://extreme-sharepoint.com/2012/05/30/impersonation-elevation-of-privileges/

re: NTLM double hop issue
Double-hop authentication: Why NTLM fails and Kerberos works
http://blogs.msdn.com/b/besidethepoint/archive/2010/05/09/double-hop-authentication-why-ntlm-fails-and-kerberos-works.aspx