Instantiating the OrganizationServiceProxy for IFD enabled organizations.

Last week was an interesting week at work, highlights include running an application from my laptop against the live service.

It took me a while to get this working because I was using the wrong endpoint, D'oh, and the wrong type of credentials, double D'oh.

In any case, here it is for posterity:

public class IFDCrmService

    {
        const string ServiceUrl = "https://{0}/XRMServices/2011/Organization.svc";

        public static IOrganizationService GetIFDCrmService()
        {
            ClientCredentials credentials = new ClientCredentials();
            credentials.UserName.UserName = ConfigurationManager.AppSettings["UserName"];
            credentials.UserName.Password = ConfigurationManager.AppSettings["Password"];

            if (ConfigurationManager.AppSettings["FQDN"] == null) { throw new ArgumentNullException("FQDN key missing."); }
            string fqdn = ConfigurationManager.AppSettings["FQDN"]; 
        
            return new OrganizationServiceProxy(new Uri(string.Format(ServiceUrl, fqdn)), null, credentials, null);
        }
    }

The relevant part of the app.config is below:

<appSettings>
    <add key="UserName" value="dev\crmapppool" />
    <add key="Password" value="P@55w0rd1" />    
    <add key="FQDN" value="crmdev.dev.com"/>
 </appSettings> 

It's always a good idea to encrypt the passwords but I won't discuss this today here.

Integrating MS SharePoint with MS Dynamics CRM 2011/2013 - User Permissions

One of the things that seems to come up time and again on any integration of MS Dynamics CRM and SharePoint is the issue of user permissions.

Generally speaking it would be nice to be able to control access to SharePoint based upon the permissions the user has in MS Dynamics CRM. Alas, this is not possible without writing a bit code. (I've not investigated the Server to Server Integration yet as it seems to be available for online Dynamics CRM only)

The way I have done this, is by using an HttpModule deployed to the Ms SharePoint servers to check whether the user making the request to the MS SharePoint site has actually got access to the record in MS Dynamics CRM itself.

In our case this is pretty straight forward as we only store documents for a single entity, but there is nothing in principle to rule out an expansion to multiple entities.

Depending on the usage, caching will need to be a serious consideration, as performance could be impacted, but I have not thought about it too much yet.

The following assumptions have been made about the integration between MS Dynamics CRM and MS SharePoint:

1. A document library exists and is named with the entity schema.
2. Each entity record in MS Dynamics CRM has a single folder in MS SharePoint and this folder is named with the GUID of the record. 
3. Entity Records in MS Dynamics CRM are not shared.

This is the code for the module itself:

using log4net;
using Microsoft.Crm.Sdk.Messages;
using Microsoft.Xrm.Sdk.Client;
using Microsoft.Xrm.Sdk.Query;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Net;
using System.Security.Principal;
using System.ServiceModel.Description;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using System.Web;

namespace CRMUserPermissions
{
    public class CRMUserPermissions : IHttpModule
    {

        public static ConcurrentDictionary<string, Guid> userIds = new ConcurrentDictionary<string, Guid>();

        const string GuidPattern = @"(/|%2f)([A-F0-9]{8}(?:-[A-F0-9]{4}){3}-[A-F0-9]{12})";

        const string UserIdQuery = @"<fetch version='1.0' output-format='xml-platform' mapping='logical' distinct='false'>
  <entity name='systemuser'>   
    <attribute name='systemuserid' />
    <attribute name='domainname' />
    <filter type='and'>
      <condition attribute='domainname' operator='eq' value='{0}' />
    </filter> 
  </entity>
</fetch>";

        public void Dispose()
        {
        }

        public void Init(HttpApplication context)
        {
            context.PostAuthenticateRequest += new EventHandler(context_PostAuthenticateRequest);
        }

        void context_PostAuthenticateRequest(object sender, EventArgs e)
        {
            HttpApplication app = sender as HttpApplication;
            HttpContext context = app.Context;

            if (IsRequestRelevant(context))
            {
                try
                {

                    string user = HttpContext.Current.User.Identity.Name.Split('|').Last();

                    var service = CrmService.GetService();

                    string url = app.Request.Url.ToString();

                    if (!userIds.ContainsKey(user))
                    {
                        string query = string.Format(UserIdQuery, user);
                        var userId = service.RetrieveMultiple(new FetchExpression(query)).Entities.SingleOrDefault();
                        userIds.TryAdd(user, userId.Id);
                    }

                    var record = GetRecordInfo(url);

                    RetrievePrincipalAccessRequest princip = new RetrievePrincipalAccessRequest();
                    princip.Principal = new EntityReference("systemuser", userIds[user]);

                    princip.Target = new EntityReference(record.Item1, record.Item2);

                    var res = (RetrievePrincipalAccessResponse)service.Execute(princip);

                    if (res.AccessRights == AccessRights.None)
                    {
                        app.Response.StatusCode = 403;
                        app.Response.SubStatusCode = 1;
                        app.CompleteRequest();
                    }
   
                }
                catch (Exception)
                {
                    app.Response.StatusCode = 403;
                    app.Response.SubStatusCode = 1;
                    app.CompleteRequest();
                }
            }
        }
    }
}

A few comments are in order since I'm not including all methods.

IsRequestRelevant(context): This method checks that the user is authenticated and that the request is for documents relating to an entity we want to control access via this method.

CrmService.GetService(); This method just returns an OrganizationServiceProxy.
GetRecordInfo(url); This method works out the record guid and what type of entity it is.

It would probably be a better idea to get all, or some, users and cache them on receipt of the first query.

Depending on the system's usage profile different types of caching make more or less sense. For instance, if users tend to access multiple documents within a record in a relatively short time, say 10 minutes, then it makes sense to cache the records and the user's right to them but if users only tend to access a single document within a record, this would make less sense. Consideration needs to be given to memory pressures that caching will create if not using a separate caching layer, such as Redis.

The simplest way of deploying this httpModule is by installing the assembly in the GAC and then manually modifying the web.config of the relevant MS SharePoint site by adding the httpModule to the modules in configuration/system.webServer/modules:

<add name="CRMUserPermissions" type="CRMUserPermissions.CRMUserPermissions, CRMUserPermissions,,Version=1.0.0.0, Culture=neutral, PublicKeyToken=87b3480442bff091"></add>

I will post how to do this properly, i.e. by packaging it up in a MS SharePoint solution in an upcoming post.

ID1073: A CryptographicException occurred when attempting to decrypt the cookie using the ProtectedData API (see inner exception for details).

A few weeks back, the performance testers found this issue when doing some resilience testing:

ID1073: A CryptographicException occurred when attempting to decrypt the cookie using the ProtectedData API (see inner exception for details). If you are using IIS 7.5, this could be due to the loadUserProfile setting on the Application Pool being set to false.

In essence, if the user started his session on server 1, then that server was stopped, when he connected to server 2, the issue would occur. Annoyingly, this would require removing cookies, which our testers are not able to do, on their locked down machines.

The deployment uses an NLB flag was checked for this deployment, but this seemed to make no difference, so we decided to encrypt the cookies using a certificate rather than the machine key.

This is a bit annoying as there are few things to consider:

1. We're going to have to modify the MS Dynamics CRM web.config, which means that every new patch, update, etc.. might overwrite it.
2. Following from that we need a deployment script to automate it as much as possible.
3. We'll need to store a way to id the certificate to be used for the encryption somewhere easily accessible (We could hard code it but then when the certificate expires we'd been in trouble).

I decided to use the registry for 3.

This is the code that we've used to encrypt the cookies with a certificate.

using Microsoft.IdentityModel.Tokens;
using Microsoft.IdentityModel.Web;
using Microsoft.Win32;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading.Tasks;

namespace CRM.RSASessionCookie
{
    /// <summary>
    /// This class encrypts the session security token using the RSA key
    /// of the relying party's service certificate.
    /// </summary>
    public class RsaEncryptedSessionSecurityTokenHandler : SessionSecurityTokenHandler
    {
        static List<CookieTransform> transforms;

        static RsaEncryptedSessionSecurityTokenHandler()
        {            
            string certThumbprint = (string)Registry.GetValue(@"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM",
                "CookieCertificateThumbprint",
                null);

            if (!string.IsNullOrEmpty(certThumbprint))
            {
                X509Certificate2 serviceCertificate = CertificateUtil.GetCertificate(StoreName.My,
                                                         StoreLocation.LocalMachine, certThumbprint);

                if (serviceCertificate == null)
                {
                    throw new ApplicationException(string.Format("No certificate was found with thumbprint: {0}", certThumbprint));
                }

                transforms = new List<CookieTransform>() 
                         { 
                             new DeflateCookieTransform(), 
                             new RsaEncryptionCookieTransform(serviceCertificate),
                             new RsaSignatureCookieTransform(serviceCertificate),
                         };
            }
            else
            {
                throw new ApplicationException(
                @"Could not read Registry Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM\CookieCertificateThumbprint.\nPlease ensure that the key exists and that you have permission to read it.");
            }
        }

        public RsaEncryptedSessionSecurityTokenHandler()
            : base(transforms.AsReadOnly())
        {
        }
    }

    /// <summary>
    /// A utility class which helps to retrieve an x509 certificate
    /// </summary>
    public class CertificateUtil
    {
        /// <summary>
        /// Gets an X.509 certificate given the name, store location and the subject distinguished name of the X.509 certificate.
        /// </summary>
        /// <param name="name">Specifies the name of the X.509 certificate to open.</param>
        /// <param name="location">Specifies the location of the X.509 certificate store.</param>
        /// <param name="thumbprint">Subject distinguished name of the certificate to return.</param>
        /// <returns>The specific X.509 certificate.</returns>
        public static X509Certificate2 GetCertificate(StoreName name, StoreLocation location, string thumbprint)
        {
            X509Store store = null;
            X509Certificate2Collection certificates = null;           
            X509Certificate2 result = null;

            try
            {
                store = new X509Store(name, location);
                store.Open(OpenFlags.ReadOnly);
                //
                // Every time we call store.Certificates property, a new collection will be returned.
                //
                certificates = store.Certificates;

                for (int i = 0; i < certificates.Count; i++)
                {
                    X509Certificate2 cert = certificates[i];

                    if (cert.Thumbprint.Equals(thumbprint, StringComparison.InvariantCultureIgnoreCase))
                    {
                        result = new X509Certificate2(cert);
                        break;
                    }
                }                
            }
            catch (Exception ex)
            {
                throw new ApplicationException(string.Format("An issue occurred opening cert store: {0}\\{1}. Exception:{2}.", name, location, ex));
            }
            finally
            {
                if (certificates != null)
                {
                    for (int i = 0; i < certificates.Count; i++)
                    {
                        X509Certificate2 cert = certificates[i];
                        cert.Reset();
                    }
                }

                if (store != null)
                {
                    store.Close();
                }
            }

            return result;
        }
    }
}

Company standards dictate that this class should be deployed to GAC but it can be deployed to the CRM webpage bin folder instead.

This is the PowerShell function in our script that sets the certificate on the registry:

function SetCookieCertificateThumbprint
{
 param ([string]$value)
 $path = "hklm:\Software\microsoft\mscrm"
 $name = "CookieCertificateThumbprint"
 
 if( -not (Test-Path -Path $path -PathType Container) )
 {
  Write-Error "Cannot find MSCRM Registry Key: " + $path
 }
 else
 {
  $keys = Get-ItemProperty -Path $path

  if ($keys.$name -or $keys.$name -ne $value)
  {
   Set-ItemProperty -path $path -name $name -value $value 
  }
 }
}

I have not automated the rest, which is the really fiddly part, i.e. updating the web.config. Here are the relevant parts though:

Config Sections First:

<configSections>
    <!-- COMMENT:START CRM Titan 28973
   If you add any new section here , please ensure that section name is removed from help/web.config
End COMMENT:END-->
    <section name="crm.authentication" type="Microsoft.Crm.Authentication.AuthenticationSettingsConfigurationSectionHandler, Microsoft.Crm.Authentication, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    <section name="microsoft.identityModel" type="Microsoft.IdentityModel.Configuration.MicrosoftIdentityModelSection, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
  </configSections>

The actual token handler:

<microsoft.identityModel>
  <service>   
    <securityTokenHandlers>
      <!-- Remove and replace the default SessionSecurityTokenHandler with your own -->
      <remove type="Microsoft.IdentityModel.Tokens.SessionSecurityTokenHandler, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
      <add type="CRM.RSASessionCookie.RsaEncryptedSessionSecurityTokenHandler, CRM.RSASessionCookie, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d10ca3d28ba8fa6e" />
    </securityTokenHandlers>
  </service>
</microsoft.identityModel>

It should, hopefully be obvious that this will need to be done across all CRM servers that have the Web Server role.

After all this work, we thought we would be ok for our dev and test environments sharing a single ADFS server, as the cookies would be encrypted with the same certificate, but it turns out that this is not supported by MS Dynamics CRM 2013 :(

Using ARR (Reverse Proxy) with Microsoft Dynamics CRM 2015 - IFD

In this post I'll discuss how we have used the Application Request Routing module from IIS to expose Ms Dynamics CRM to the big bad world.

In this project I'm working on at the moment we want to allow authorized third parties to access to our application, which means that there has been a lot of discussions around what's the best way of doing this.

The Problem:

We want to allow third parties access to our MS Dynamics CRM 2015 system but these third parties might be mom and pop operations so federating with them is not an option. In reality, this is a fantasy conjured up by the PM, but we all go along with it, because sometimes it's important to know which battles to fight.

We also don't want to expose our MS Dynamics CRM 2015 servers to the big bad internet.

The Solution:

Use ARR to expose the MS Dynamics CRM 2015 website to the outside world, while keeping the servers happily inside the perimeter.

This is an architecture diagram of what we are trying to achieve.

I'm assuming that you have a working MS Dynamics CRM 2013/2015 installation configured for Internet Facing Deployment and a wildcard certificate for the appropriate domain available.

Install ARR.

Navigate to this page and click on install this extension, which will use the Windows Platform installer to install ARR, just follow the instructions therein. Alternatively you can do it manually, which I haven't tried.

Configure ARR

ARR is configured from the IIS Manager console.

So fire it up (windows key + r -> inetmgr)  and let's get started:

We first create a Server Farm for the MS Dynamics CRM Web Servers. If you have web and application CRM servers then you only need to add the web servers. We have full servers.

Ensure you add all servers, not just one server, unless you only have one server like myself in this environment :)

If you are only using ARR to expose MS Dynamics CRM to the outside world, then it's probably ok to click yes here.

 We can now configure our server farm.

We want to make sure that traffic is only directed to servers that are up and running so I normally set the organization's wsdl page as the health test URL. The server's page (using FQDN) should also work and in fact should be used if you have more than one organization.

The other thing that needs to be done is to set load balancing to be a based on Client affinity, in other words, a client will always hit the same server, unless that server is down. This is an optional step, that's only required, as far as I can tell, for reports.

We now need to check that the routing rule is correct for the current set up.

Pretty simple, as we're sending everything that hits this server onward to the MS Dynamics CRM Farm. However, the rule will, by default, forward it to http rather than https, so this needs to be changed.

The final step is to create the proxy websites. I'm going to assume that a wildcard certificate has been installed already for the server. These website will receive the traffic from the internet and forward it to the MS Dynamics CRM Web Servers.

When using IFD, a website is needed for each organization and another one for the auth website. You can also add one for the discovery service, but as far as my limited testing goes, it works without it.

At this point ARR is configured, now all that remains is to set up name resolution.

I normally use CNAME aliases for auth, disc and the various organizations to point to the ARR server:

stark.dev.local --> ARR Server (Organization)
auth.dev.local --> ARR Server  (auth)
disc.dev.local --> ARR Server   (discovery)

In my next post I will add ADFS to the reverse proxy, note that it's perfectly acceptable to use ARR for this purpose as well, but it also possible to use WAP, which provides pre-authentication.

MS Dynamics CRM - Blocking Direct Access to SharePoint

In this project I'm working on we've had a requirement to block direct access to SharePoint, in other words. Our users can only access their documents through CRM.

There are various ways, in which this can be achieved and today I will be discussing achieving this by leveraging ARR.

We have a similar architecture to this:

We have a Sharepoint site collection and every document is stored in this site collection, so our users would go https://sp.dev.local/sites/spsite/<crm_entity>/ to access documents for <crm_entity>

This is actually a somewhat irritating requirement because IE behaves differently than Firefox and Chrome. We're lucky enough not to have to support Opera and Safari as well, nothing wrong with these browsers, but 5 browsers would drive our testers crazy.

In any case, we've configured two farms, CRM and SHAREPOINT, so we need rules for those.

So for CRM we have:

<rule name="CRM Farm Forwarding Rule" enabled="true" patternSyntax="Wildcard" stopProcessing="false">
    <match url="*" />
    <conditions logicalGrouping="MatchAny" trackAllCaptures="true">
        <add input="{HTTP_HOST}" pattern="*crm.dev.local" />
    </conditions>
    <serverVariables>
    </serverVariables>
    <action type="Rewrite" url="https://CRM/{R:0}" />
</rule>

And for Sharepoint we have:

<rule name="Sharepoint Farm Forwarding Rule" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
    <match url="*" />
    <conditions logicalGrouping="MatchAny" trackAllCaptures="true">
        <add input="{HTTP_HOST}" pattern="*sp.dev.local" />
    </conditions>
    <serverVariables>
    </serverVariables>
    <action type="Rewrite" url="https://SHAREPOINT/{R:0}" />
</rule>

So far so good, this is where things start to get interesting. We want to block direct access to SP for IE browsers, which we can achieve like this:

 
<rule name="IE - Allow Access to SharePoint grid in CRM" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
    <match url="*sites/SPSITE/crmgrid*" />
    <conditions logicalGrouping="MatchAny" trackAllCaptures="false">
    </conditions>
    <serverVariables>
    </serverVariables>
    <action type="Rewrite" url="https://SHAREPOINT/{R:0}" />
</rule>    

and

<rule name="IE - Block Direct Access to SharePoint" stopProcessing="true">
    <match url="sites\/SPSITE" />
    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
        <add input="{HTTP_REFERER}" pattern="^/?$" negate="true" />
        <add input="{HTTP_USER_AGENT}" pattern="MSIE" />
    </conditions>
    <serverVariables>
    </serverVariables>
    <action type="CustomResponse" statusCode="403" subStatusCode="1" statusReason="IE" statusDescription="Direct Access to SHAREPOINT is not permitted" />
</rule>

The first rules allows traffic if it's trying to access SP through the List component, this allows the SharePoint CRM List component to work.

The second rule will block access for IE user agents ( i.e containing MSIE) and where the referer is not empty. It will stop processing if there is a match.

For some reason, I.E. blanks the referer when accessing documents in SharePoint from the list component but crucially fills it in if accessing documents directly from SharePoint.

Firefox and Chrome will have the correct referer, i.e. sp.dev.local/sites/spsite/crmgrid, so there is a single rule:

<rule name="Other Browsers - Block Direct Access to SharePoint" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
    <match url="*sites/SPSITE*" />
    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
        <add input="{HTTP_REFERER}" pattern="*dev.local/sites/SPSITE/crmgrid*" negate="true" />
        <add input="{HTTP_USER_AGENT}" pattern="*MSIE*" negate="true" />
    </conditions>
    <serverVariables>
    </serverVariables>
    <action type="CustomResponse" statusCode="403" subStatusCode="2" statusReason="Other" statusDescription="Direct Access to SHAREPOINT is not permitted" />
</rule>

Just need to make sure that the user agent is not from IE. The rule will stop processing if there is a match.

Rules.xml can be found below, with the rules in the correct order.

Thus effectively we do the following:

CRM -> Rewrite to  CRM Farm
SP - crm grid?  -> Rewrite to SP Farm
SP - IE and empty Referer -> Rewrite to SP Farm
SP - Other Browser and correct Referer -> Rewrite to SP Farm
SP -> Rewrite to Farm

The last rule is only needed if there are other sites in SP that the users might need to access.

<?xml version="1.0" encoding="UTF-8"?>
<appcmd>
    <CONFIG CONFIG.SECTION="system.webServer/rewrite/globalRules" path="MACHINE/WEBROOT/APPHOST" overrideMode="Inherit" locked="false">
        <system.webServer-rewrite-globalRules>
            <rule name="CRM Farm Forwarding Rule" enabled="true" patternSyntax="Wildcard" stopProcessing="false">
                <match url="*" />
                <conditions logicalGrouping="MatchAny" trackAllCaptures="true">
                    <add input="{HTTP_HOST}" pattern="*crm.dev.local" />
                </conditions>
                <serverVariables>
                </serverVariables>
                <action type="Rewrite" url="https://CRM/{R:0}" />
            </rule>
            <rule name="IE - Allow Access to SharePoint grid in CRM" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
                <match url="*sites/SPSITE/crmgrid*" />
                <conditions logicalGrouping="MatchAny" trackAllCaptures="false">
                </conditions>
                <serverVariables>
                </serverVariables>
                <action type="Rewrite" url="https://SHAREPOINT/{R:0}" />
            </rule>
            <rule name="IE - Block Direct Access to SharePoint" stopProcessing="true">
                <match url="sites\/SPSITE" />
                <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    <add input="{HTTP_REFERER}" pattern="^/?$" negate="true" />
                    <add input="{HTTP_USER_AGENT}" pattern="MSIE" />
                </conditions>
                <serverVariables>
                </serverVariables>
                <action type="CustomResponse" statusCode="403" subStatusCode="1" statusReason="IE" statusDescription="Direct Access to SHAREPOINT is not permitted" />
            </rule>
            <rule name="Other Browsers - Block Direct Access to SharePoint" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
                <match url="*sites/SPSITE*" />
                <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    <add input="{HTTP_REFERER}" pattern="*dev.local/sites/SPSITE/crmgrid*" negate="true" />
                    <add input="{HTTP_USER_AGENT}" pattern="*MSIE*" negate="true" />
                </conditions>
                <serverVariables>
                </serverVariables>
                <action type="CustomResponse" statusCode="403" subStatusCode="2" statusReason="Other" statusDescription="Direct Access to SHAREPOINT is not permitted" />
            </rule>
            <rule name="Sharepoint Farm Forwarding Rule" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
                <match url="*" />
                <conditions logicalGrouping="MatchAny" trackAllCaptures="true">
                    <add input="{HTTP_HOST}" pattern="*sp.dev.local" />
                </conditions>
                <serverVariables>
                </serverVariables>
                <action type="Rewrite" url="https://SHAREPOINT/{R:0}" />
            </rule>
        </system.webServer-rewrite-globalRules>
    </CONFIG>
</appcmd>

It can be exported with this command:

appcmd.exe list config -section:system.webServer/rewrite/globalRules -xml > rules.xml 

The imported with this command:

appcmd.exe set config /in < rules.xml

The authentication endpoint Kerberos was not found on the configured Secure Token Service! - Redux

So today I had an interesting moment of Deja vu today, where we started getting this issue again.

This time though, the configuration had already been done so it was a bit mystifying to say the least and to further add to the confusion, it worked on one server but not the other.

After a lot of blood, sweat and tears the issue was tracked down to the affected server being unable to communicate with the ADFS load balancer. In essence, our network guys had not opened the firewall for both servers but just one of them.

So, if you get this issue, it might simply be due to a good old fashioned network issue.

Removing HTTP Headers for an ARR/MS Dynamics CRM/Sharepoint 2013 system

We had a pen test carried out last week and one of the outcomes was that we were leaking information with our HTTP headers and they must be removed.

Our environment consists of a web layer (IIS using ARR) and then an app layer (MS Dynamics CRM and MS SharePoint)

Personally I think this is a bit of security through obscurity but needs must so here we go:

X-AspNet-Version Header:

MS Dynamics CRM 2013

In the web.config, <drive>:Program Files\Microsoft Dynamics CRM\CRMWeb  add enableVersionHeader="false" to the httpRuntime element, normally you'll end up with something like this:

<httpRuntime executionTimeout="300" maxRequestLength="32768" requestValidationMode="3.0" encoderType="Microsoft.Crm.CrmHttpEncoder, Microsoft.Crm" enableVersionHeader="false"/>

MS SharePoint 2013

In the web.config,  <drive>:inetpub\wwwroot\wss\VirtualDirectories\80\ add enableVersionHeader="false" to the httpRuntime element, normally you'll end up with something like this:

<httpRuntime maxRequestLength="51200" requestValidationMode="2.0" enableVersionHeader="False" />

X-Powered-By Header:

From IIS Manager -> Server -> HTTP Response Headers

Server Header:

The simplest way I found is to use URL Rewrite to blank this header, which works very well for our system as we're using ARR already so just need to do this one on the web layer ... from IIS Manager -> Url Rewrite -> Add Rule

Select Blank Outbound Rule

Fill in the details as below

Don't forget to click Apply when you've finished.

It's worth pointing out  that this will simply blank out the value of the Server Header, rather than remove it completely.

If you want to remove it completely you will need to install urlscan.

This approach can be used for all the other headers above I suppose.

X-Powered-By: ARR/2.5 Header

From a powershell console with elevated permissions go to C:\Windows\system32\inetsrv and run this command:

.\appcmd.exe set config -section:webFarms /"[name='serverfarmname'].applicationRequestRouting.protocol.arrResponseHeader:false" /commit:apphost

Configure MS Dynamics CRM 2011/2013/2015 to use multiple Report Servers (SSRS)

This week I've had quite a bit of fun turning the resilience up to 11 for our production environment of MS Dynamics CRM 2013.

In this post I will discuss how to configure Ms Dynamics CRM 2013 to use multiple SSRS servers, thus ensuring that reporting functionality is as resilient as the rest of the system.

In order to achieve this we need to make changes to AD, the SSRS configuration and finally MS Dynamics CRM. 

It's worth pointing out that this could well be overkill for your system, and to a certain extent it is for ours, but the whole architecture must be resilient is the diktat from above so ...

Pre-Requisites:

• 1 x Load Balancer (Distributing traffic to SSRS Servers on correct port, normally 80).
• 1 x VIP.
• 1 x DNS Record (For VIP above).
• 2+ x SSRS Servers in a scale out deployment.
• SSRS configured to use a domain account.
• Permissions to set SPNs on your domain and edit at least ssrs service account.

My Setup:

DNS record is: CRMReports.dev.local
SSRS Service Account:  dev\svc-ssrs

1. Active Directory

The first thing to do is to set up an Service Name Principal for the account that's running the SSRS service, which can be done with the following commands:

setspn -S HTTP/<VIP FQDN> <SSRS Service account>
setspn -S HTTP/<VIP Name> <SSRS Service account>

So in my case:

setspn -S HTTP/CRMReports.dev.local dev\svc-ssrs
setspn -S HTTP/CRMReports dev\svc-ssrs

The next thing is to ensure that the account is enabled for delegation, so from Active Directory Users and Computers console.

Note that the delegation tab will only appear after an SPN has been set up for that account.

2. SSRS

The first thing to do in the report server, is to enable Kerberos authentication, which can be done by editing the rsreportserver.config file. This file is normally found in this directory: C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\ReportServer\

Look for the Authentication section and enable Kerberos and Negotiate as below (I've commented out NTLM in case I wanted to go back)

<Authentication>
  <AuthenticationTypes>
   <RSWindowsKerberos/>
   <RSWindowsNegotiate/>
   <!--<RSWindowsNTLM/>-->
  </AuthenticationTypes>
  <RSWindowsExtendedProtectionLevel>Off</RSWindowsExtendedProtectionLevel>
  <RSWindowsExtendedProtectionScenario>Proxy</RSWindowsExtendedProtectionScenario>
  <EnableAuthPersistence>true</EnableAuthPersistence>
 </Authentication>

From the Reporting Services Configuration Manager, Click on Web Service URL and then click on Advanced

Click on Add as highlighted.

Enter the host header name, which will be the DNS Record for the VIP, in my case: crmreports.dev.local and click OK to accept.

Repeat this process for the Report Manager Url.

Once done repeat all steps on section 2 on the other report servers.

3. MS Dynamics CRM

The process is relatively simple and it involves editing the organization to point to the new report server dns, i.e. crmreports.dev.local in my case.

From the Deployment Manager, select organization and Disable your organization(s).

Click Edit Organization.

Set the new report server Url.

 Ensure that all checks are ok.

Congratulations, you now have a resilient reporting service for MS Dynamics CRM.

Unable to Navigate to external domain (auth endpoint) in MS Dynamics CRM 2013/2015 IFD

The standard practice for my company when deploying web servers is to use a host header, which probably made sense at some point but it surely made for an interesting Friday.

I'm tired, so I will just present the facts: If you are configuring IFD and can't get to the external domain endpoint, the problem might be that you have a host header for your https binding.

The external domain endpoint is normally: https://auth.adomain.com/FederationMetadata/2007-06/FederationMetadata.xml and is the last step on the configure IFD wizard.

Ensure that IIS is configured without a host header for the https binding:

OrganizationServiceContext performance in MS Dynamic CRM 2013

The OrganizationServiceContext has a SaveChanges method that essentially does what it says on tin, namely the changes on the objects back to the database.

I've never used this method, preferring to use the regular Update method  on the IOrganizationService instead, but last week I had a moment of doubt, what if it is faster, what if it's multi-threaded, so I decided to run some tests to see whether there was any performance difference, spoiler alert: There wasn't

The tests consisted of the update of a three fields on the account entity for 2000 accounts and were run from an idle CRM server to minimize the influence of network traffic.

Results shown below are for the average of the three runs I did, except for the parallel version, using Parallel.ForEach, where it shows the single run I did.

OrganizationServiceContext	IOrganizationService	IOrganizationService (Parallel)
126.7 s	130.1 s	49.6 s

My fears were unfounded and the SaveChanges method, while slightly faster in these tests, does not seem to be appreciatively faster.

The difference is less than 3% and the slowest run using the OrganizationServiceContext was basically the same as the fastest with IOrganizationService : ~129 s

I broke Ms Dynamics CRM 2013 today

So I have a custom workflow activity that has an output of a custom reference, something like this:

[Output("ClientRecord")]
[ReferenceTarget("new_clientrecord")]
public OutArgument<EntityReference> ClientRecord { get; set; }

And like idiot I did this:

ClientRecord.Set(executionContext, ClientRecordEntity);

where ClientRecordEntity is an Entity and not an EntityReference.

Running the Dialog that contained this workflow activity, resulted in a very, very long wait and after that, 503 for absolutely everybody using the dev environment.

A proud day, today

I broke CRM.

Limit Regarding Lookup in MS Dynamics CRM 2013

So Today we had an interesting requirement: We had a custom entity, called Paper, for which we wanted to limit the entities that could be selected for the regarding lookup.

I can't come up with a supported way of doing this so this is the unsupported way:

In essence, this will limit the Regarding to new_keyissue and new_goal.

You'll need to fire it on the onChange event like this, NEW.Paper.limitRegardingLookup.

if (typeof (NEW) == "undefined")
{ NEW = {}; }
NEW.Paper = {
    limitRegardingLookup: function () {
        
        var KeyIssueOTC = GetEntityTypeCode("new_keyissue");
        var goalOTC = GetEntityTypeCode("new_goal");
  
        var ObjectTypeCodeList = KeyIssueOTC + ", " + goalOTC;
        var LookupTypeNames = "new_keyissue:"+ KeyIssueOTC + ":Key Issues,new_goal:" + goalOTC + ":Goal";
        
 Xrm.Page.getControl("regardingobjectid").setFocus(true);
  
        document.getElementById("regardingobjectid_i").setAttribute("lookuptypes", ObjectTypeCodeList);
        document.getElementById("regardingobjectid_i").setAttribute("lookuptypenames", LookupTypeNames);
        document.getElementById("regardingobjectid_i").setAttribute("defaulttype", KeyIssueOTC);

    }
};

function GetEntityTypeCode(entityName) {
    try {
        var lookupService = new RemoteCommand("LookupService", "RetrieveTypeCode");
        lookupService.SetParameter("entityName", entityName);
        var result = lookupService.Execute();
        if (result.Success && typeof result.ReturnValue == "number") {
            return result.ReturnValue;
        }
        else {
            return null;
        }
    }
    catch (ex) {
        throw ex;
    }
}

List Entity relationships in CRM 2011/2013 - Brain Dump 6

Cleaning up my inbox today when I found this SQL query, in short a quick way of listing all the relationships for a particular entity:

SELECT distinct (rel.name),ent.name,
        Case [CascadeDelete]
                         when 0 then 'None'
                         when 1 then 'All'
                         when 2 then 'Referential'
                         when 3 then 'Restrict'
                        end as CascadeDelete
      ,Case [CascadeAssign]
                         when 0 then 'None'
                         when 1 then 'All'
                         when 2 then 'Referential'
                         when 3 then 'Restrict'
                        end as CascadeAssign
      ,Case [CascadeShare]
                         when 0 then 'None'
                         when 1 then 'All'
                         when 2 then 'Referential'
                         when 3 then 'Restrict'
                        end as CascadeShare
      ,Case [CascadeUnShare]
                         when 0 then 'None'
                         when 1 then 'All'
                         when 2 then 'Referential'
                         when 3 then 'Restrict'
                        end as CascadeUnShare
      ,Case [CascadeReparent]
                         when 0 then 'None'
                         when 1 then 'All'
                         when 2 then 'Referential'
                         when 3 then 'Restrict'
                        end as CascadeReparent
      ,Case [CascadeMerge]
                         when 0 then 'None'
                         when 1 then 'All'
                         when 2 then 'Referential'
                         when 3 then 'Restrict'
                        end as CascadeMerge
       from MetadataSchema.Relationship rel
      
join  MetadataSchema.Entity ent on rel.ReferencedEntityId = ent.entityid
where rel.name like '%<entityname>%'
order by ent.Name

And the results:

Update Entity from JavaScript in Ms Dynamics CRM 2011/2013 using OData endpoint

Posting this for future reference:

function updateEntity(entityName, id, entity ) {

    var url = oDataUrl + "/" + entityName + "Set(guid'" + id + "')";

    entityData = window.JSON.stringify(entity);

    return $.ajax({
        type: "POST",
        contentType: "application/json;charset=utf-8",
        datatype: "json",
        data: entityData,
        url: url,
        beforeSend: function (x) {
            x.setRequestHeader("Accept", "application/json");
            x.setRequestHeader("X-HTTP-Method", "MERGE")
        },
    });
}

entityName is the logical entity name.
id is the Id of the record that we want to update
entity is a object that contains that values that need changing.
 e.g.:

 account = {};
 account.Name = "New Name"; 

 oDataUrl is the Url of the OData endpoint for your organization

The authentication endpoint Kerberos was not found on the configured Secure Token Service!

We finally managed to overcome all issues and deployed the build to the OAT environment and it went: KABOOM!!!

Unable to get item. Exception: System.NotSupportedException: The authentication endpoint Kerberos was not found on the configured Secure Token Service! at Microsoft.Xrm.Sdk.Client.IssuerEndpointDictionary.GetIssuerEndpoint(TokenServiceCredentialType credentialType) at Microsoft.Xrm.Sdk.Client.AuthenticationCredentials.get_IssuerEndpoint() at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1.AuthenticateInternal(AuthenticationCredentials authenticationCredentials) at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1.AuthenticateFederationInternal(AuthenticationCredentials authenticationCredentials) at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1.Authenticate(AuthenticationCredentials authenticationCredentials) at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1.Authenticate(ClientCredentials clientCredentials) at Microsoft.Xrm.Sdk.Client.OrganizationServiceConfiguration.Authenticate(ClientCredentials clientCredentials) at Microsoft.Xrm.Sdk.Client.ServiceProxy`1.AuthenticateClaims() at Microsoft.Xrm.Sdk.Client.ServiceProxy`1.AuthenticateCore() at Microsoft.Xrm.Sdk.Client.ServiceProxy`1.Authenticate() at Microsoft.Xrm.Sdk.Client.ServiceProxy`1.ValidateAuthentication() at Microsoft.Xrm.Sdk.Client.ServiceContextInitializer`1.Initialize(ServiceProxy`1 proxy) at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.RetrieveMultipleCore(QueryBase query) at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.RetrieveMultiple(QueryBase query) at Consumer.ItemManager.RetrieveLastItem(String type) at
Consumer.ItemManager.RetrieveLastItem(String type) at
Consumer.Service.ConsumerService.Process[T](Config feed)

I thought that the Kerberos endpoint must not be enabled in ADFS, but it was and after a bit of investigating, it turns out that this is a known issue in MS Dynamics CRM 2011/13.

The interesting bit about this issue is that the front end was working fine, but trying to use the SDK in any way was not working.

The MEX endpoint that gets set when Claims Based Authentication is configured is like this:

https://adfs.domain.com/adfs/ls/mex

This is a bit of a problem, as it doesn't exist :(

The Working MEX endpoint is:

https://adfs.domain.com/adfs/services/trust/mex

Microsoft have kindly provided a PowerShell script to rectify this issue:

Save this as UpdateMEXEndpoint.ps1

Param (
     #optional params
     [string]$ConfigurationEntityName="FederationProvider",
     [string]$SettingName="ActiveMexEndpoint",
     [object]$SettingValue,
     [Guid]$Id
 )
 $RemoveSnapInWhenDone = $False
 
 if (-not (Get-PSSnapin -Name Microsoft.Crm.PowerShell -ErrorAction SilentlyContinue))
 {
     Add-PSSnapin Microsoft.Crm.PowerShell
     $RemoveSnapInWhenDone = $True
 }
 
 $Id=(Get-CrmAdvancedSetting -ConfigurationEntityName FederationProvider -Setting ActiveMexEndpoint).Attributes[0].Value
 
 $setting = New-Object "Microsoft.Xrm.Sdk.Deployment.ConfigurationEntity"
 $setting.LogicalName = $ConfigurationEntityName
 if($Id) { $setting.Id = $Id }
 
 $setting.Attributes = New-Object "Microsoft.Xrm.Sdk.Deployment.AttributeCollection"
 $keypair = New-Object "System.Collections.Generic.KeyValuePair[String, Object]" ($SettingName, $SettingValue)
 $setting.Attributes.Add($keypair)
 
 Set-CrmAdvancedSetting -Entity $setting
 
 if($RemoveSnapInWhenDone)
 {
     Remove-PSSnapin Microsoft.Crm.PowerShell
 }

This can then be used to modify the relevant setting:

UpdateMEXEndpoint.ps1 –SettingValue “https://<ADFS STSHOST>/adfs/services/trust/mex”

An alternative to use this script is updating the FederationProvider table in the MSCRM_Config database, but this is not supported.

Storing sensitive data (e.g. passwords) in MS Dynamics CRM 2011/2013

We need to integrate with a third party, who have decided that implementing a federated trust is too complex and thus have just given us a user name and password.

Since the integration is done via a couple of plug-ins and custom activities we've decided to store the password in Dynamics CRM, the only problem is that there is no out of the box way of storing the passwords that would allow a relatively simple automated deployment.

Sure, we can register plug-in and use the secure configuration but that means deploying the solution and then re-registering the plug-ins with the secure data but this wasn't suitable.

We ruled out symmetric encryption as we would just have the same problem but for the encryption key, so the obvious choice was asymmetric encryption, the problem is that asymmetric encryption is not really suitable for large amounts of data, so we settled on the recommend way of using asymmetric encryption to encrypt the encryption key of a symmetric encryption scheme.

The thing is, the .NET framework sort of includes this in the form of the EncryptedXml class, which can use a certificate to encrypt and decrypt Xml documents and we can use this for storing a password for instance.

A sample of how to use this class can be seen below:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Security.Cryptography.Xml;
using System.Text;
using System.Threading.Tasks;
using System.Web;
using System.Xml;
using System.Xml.Linq;
using log4net;

namespace Encryption
{
    public class Encryption
    {
        ILog logger;
        X509Certificate2 certificate;

        public Encryption(string certificateThumbprint) 
        {
            logger = LogManager.GetLogger("Encryption");
            certificate = GetCert(certificateThumbprint);
        }

        public string Encrypt(string plaintext)
        {
            XmlDocument Doc = new XmlDocument();
            Doc.LoadXml(string.Format("<sensitivedata>{0}</sensitivedata>", HttpUtility.HtmlEncode(plaintext)));
            Doc.PreserveWhitespace = true;
            XmlElement toEncrypt = Doc.GetElementsByTagName("sensitivedata")[0] as XmlElement;
            EncryptedXml eXml = new EncryptedXml();
            EncryptedData edElement = eXml.Encrypt(toEncrypt, certificate);
            EncryptedXml.ReplaceElement(toEncrypt, edElement, false);
            return Doc.OuterXml;
        }

        public string Decrypt(string encryptedtext)
        {
            XmlDocument Doc = new XmlDocument();
            Doc.LoadXml(encryptedtext);
            EncryptedXml exml = new EncryptedXml(Doc);
            exml.DecryptDocument();
            string plaintext = HttpUtility.HtmlDecode(XDocument.Parse(Doc.OuterXml).Element("sensitivedata").Value);
            return plaintext;
        }

        private X509Certificate2 GetCert(string thumbprint)
        {
            X509Certificate2 cert = null;
            X509Store store = new X509Store(StoreLocation.LocalMachine);
            store.Open(OpenFlags.ReadOnly);
            try
            {
                X509Certificate2Collection certCollection = store.Certificates;
                cert = certCollection.Cast<X509Certificate2>().Where(c => c.Thumbprint.Equals(thumbprint)).Single();
            }
            catch (Exception ex)
            {
                logger.ErrorFormat("An error occurred looking for certificate with thumbprint: {0}.\nException:{1}."
                    , thumbprint, ex);
            }
            finally
            {
                store.Close();
            }
   
            return cert;
        }
    }
}

The thing to note is that sensitive data to be encrypted needs to be valid xml data, in other words, if your sensitive data contains ampersands it will not be parsed, which is why I use the HttpUtility class to encode and decode the sensitive data.

 

 It does seem a little bit fiddly but it is quite almost all done by the framework, so it's really less code to maintain and the extra text that needs to be stored is not a consideration as it's a single record.

 

Furthermore, it would be trivial to modify and return a plaintext Xml Document for processing instead of the value of an element, but the value of an element is what I needed.

URLs fields in MS Dynamics CRM 2011 - MS Dynamics CRM 2011 annoyances part 1337

We had a requirement to let users see a URL field but not change the URL, the requirement, quite sensibly specified that the users should be able to click on the link.

I thought this was a textbook case for field security profiles, alas I was wrong. In MS Dynamics CRM 2011 if a URL field is read only, however this is achieved, the URL will not be clickable.

Despite the message, it does not work, note how the mouse icon does not change.

Fortunately, this has been rectified in MS Dynamics CRM 2013

Configure Federation Trust for Claim Based authentication in Ms Dynamics CRM 2013 using ADFS 3.0 ( Windows Server 2012 R2) part 3

In part 1, I described how to install and configure ADFS on a Windows 2012 R2 server. In part 2, I described how to configure Ms Dynamics CRM 2013 to use claim based authentication and in this final post of this series I configure a federation trust to allow users from a separate forest to access MS Dynamics CRM 2013.

 

In our case, this is due to an acquisition, so that we are hosting Dynamics CRM in a domain called dev.local and the company bought has their own domain called taleb.local.

 

The objective is to allow users from taleb.local to log on, with their taleb.local accounts into our instance of Dynamics CRM, so we could say that taleb.local is the Accounts domain and dev.local is the Dynamics Domain.

 

Pre-Requisites:

• Working ADFS servers on both domain/forests
• Name resolution working between domain/forests as well as inside forests, I have used stub zones but any way that works for you is good.
• TCP port 443 between ADFS servers and Dynamics Servers(or at least Front End Servers) open.
• Account with permissions to configure both ADFS servers.

 

In our case, due to the internal use, Public certificates are not in use, so the first step is to ensure that taleb.local machines trust the dev.local CA. This is out of scope for this post, see this link for details.

 

The ADFS server(s) in dev.local need to trust the taleb.local CA, which can be achieved by adding the certificate to the trusted store for the computer account.

 

On the Accounts domain's (taleb.local) ADFS server, add a Relying Party Trust for the dev.local ADFS endpoint:

 

When the Claim Rules window opens, Click add Rule and Add a Send LDAP Attribute as Claims Claim Rule. The only LDAP Attribute that we are after is UPN.

On the Dynamics domain's (dev.local) ADFS Server, Add a Claims Provider Trust for the Accounts domain (taleb.local) ADFS endpoint:

When the Claim Rules window opens, Click add Rule and Add a Pass Through or Filter an Incoming Claim Claim Rule. I think that we only really need UPN, but I've added Windows Account Name and Primary SID as well for good measure.

 

 

At this point, all that remains is for accounts from the Accounts domain (taleb.local) to be added to Dynamics CRM as users and given a role so that they can log in to Dynamics CRM.

 

Ensure that UPN is used to add a new user to Dynamics CRM, e.g. For a user with logon: taleb\NN, the username in Dynamics CRM would be: nn@taleb.local. Annoyingly, it doesn't retrieve first name, last name so these will need to be added manually. I'd love to know if it's actually possible (to get the first/last name to auto populate) by adding more claims but I've never been able to get it working.

 

I've added both domains to the trusted sites as our default configuration is for automatic logon with current username and password on trusted sites.

 

Navigating to https://allinone.dev.local/lospolloshermanos/main.aspx from a machine in the accounts domain (taleb.local) brings this screen up: 

Selecting sts1.taleb.local will redirect to https://allinone.dev.local/lospolloshermanos/main.aspx with the current account from the accounts domain (taleb.local).