Thursday 29 September 2011

The importance of good references

As part of the housekeeping service that I was talking about yesterday there is a call to the CrmService.Delete method.

Having gone through the pain described in my previous post,or not, as it doesn't tell anything about all the hair pulling,  code revisions, swearing, etc.. Anyway I ran into another issue: The DeleteMethod, that wasn't described in my previous post, is simply a wrap around for the CrmService.Delete method and whenever I passed the entity name and a guid I would get this error:
DeleteEntity web service method name is not valid
I checked the entity in the database and it was there. I checked the CRM trace logs and found nothing, which I thought it was a bit strange. I added a catch block for SOAP Exceptions and found that it wasn't a SOAP exception. I refreshed the web service references in the Visual Studio project, but to no avail. 

A colleague suggested removing the references and readding them and this fixed the issue, WTF???. This is the second such incident (CRM service reference issue that I have had) and have solved both the same way, i.e. by removing the references, web service references I should say, and re adding it, but never really understood what the cause of the issue was. At any rate, maybe it'll help somebody if they come across the same issue as me.

Wednesday 28 September 2011

Multithreading and anonymous Methods

A few weeks ago I started my first foray into multi threading. The idea was to speed up a housekeeping service by using several threads (all it does at the moment is call, or consume, a web service that deletes stuff from the database) and since we have four servers, I thought I could speed it up by forking four threads each one going to an application server.

All seemed to be going well until I actually tested properly, i.e. deleting hundreds of entities rather than just a few. Then something bizarre was happening. I kept getting an indexoutofrange exception on line 25, which I could not understand as the threadNumber is not anywhere else in the code (only relevant method shown below)

   1  public static void RunBatch(string entityName)
   2 {
   3     try
   4     {
   5         List<string> entityIds = GetEntityListToBeDeleted(entityName);
   6 
   7         if (entityIds.Count > 0)
   8         {
   9             //number of entities that each thread will delete.
  10             int entitiesperthread = entityIds.Count / nofThreads;
  11 
  12             for (int threadNumber = 0; threadNumber < nofThreads; threadNumber++)
  13             {
  14                 MRE[threadNumber] = new ManualResetEvent(false);
  15 
  16                                 myService = InstantiateService(threadNumber);
  17                                 
  18                 if (threadNumber != nofThreads - 1)
  19                 {
  20 
  21                     threads[threadNumber] = new Thread(() => DeleteMethod(threadNumber * entitiesperthread, (threadNumber + 1) * entitiesperthread - 1, MRE[threadNumber], threadName, myService, entityIds, entityName));                    
  22                 }
  23                 else
  24                 {
  25                     threads[threadNumber] = new Thread(() => DeleteMethod(threadNumber * entitiesperthread, entityIds.Count, MRE[threadNumber], threadName, myService, entityIds, entityName));
  26                 }
  27 
  28                 threads[threadNumber].Start();
  29                
  30             }
  31 
  32             WaitHandle.WaitAll(MRE);
  33 
  34         }
  35     }
  36     catch (Exception ex)
  37     {
  38      //TODO: Log it.   
  39     }
  40 
  41 }

It turns out that when you pass a value to an anonymous method, as I do to tell the thread what to do, the values are passed by reference, which means that the for loop continues running and will set threadNumber to 4, in my case, this happens very quickly, in fact quicker that the thread is actually started, so that we get the indexoutofrange exception as by the time the thread has started and gets a chance to call DeleteMethod, the for loop has already completed and set threadNumber to 4. A simple change gets us over the hurdle:

   1  public static void RunBatch(string entityName)
   2 {
   3     try
   4     {
   5       
   6         List<string> entityIds = GetEntityListToBeDeleted(entityName);
   7 
   8         if (entityIds.Count > 0)
   9         {
  10             //number of entities that each thread will delete.
  11             int entitiesperthread = entityIds.Count / nofThreads;
  12 
  13             for (int i = 0; i < nofThreads; i++)
  14             {
  15                 int threadNumber = i;
  16 
  17                 MRE[threadNumber] = new ManualResetEvent(false);
  18 
  19                 myService = InstantiateService(threadNumber);
  20                                 
  21                 if (threadNumber != nofThreads - 1)
  22                 {
  23 
  24                     threads[threadNumber] = new Thread(() => DeleteMethod(threadNumber * entitiesperthread, (threadNumber + 1) * entitiesperthread - 1, MRE[threadNumber], threadName, myService, entityIds, entityName));
  25                 }
  26                 else
  27                 {
  28                     threads[threadNumber] = new Thread(() => DeleteMethod(threadNumber * entitiesperthread, entityIds.Count, MRE[threadNumber], threadName, myService, entityIds, entityName));
  29                 }
  30 
  31                 threads[threadNumber].Start();
  32                
  33             }
  34 
  35             WaitHandle.WaitAll(MRE);
  36 
  37         }
  38     }
  39     catch (Exception ex)
  40     {
  41      //TODO: Log it.   
  42     }
  43 
  44 }

Now the for loop will finish, but because we are using a copy of the loop variable, rather than the variable itself, threadNumber will never be set to 4 and thus it will run fine.

What improvements can we expect by fully utilizing our app servers?
Well, we've gone from 6 deletions per minute for a single thread to 27 deletions per minute for 4 threads (1 per server). As an experiment I set two threads per server and we got an improvement  but only to 33 deletions per minute. I've not tried running three threads per server, but I can't see that we'll get a massive improvement out of that.

Friday 23 September 2011

ITIL Foundation certificate V3

I passed my V3 Foundation Certificate in IT Service Management exam, what more can I say?

Can I have those two and half days of my life back?

Having already done V2, I struggle to quantify any benefit I got out of this course.

At least management got their numbers, which is all that counts.

Sunday 18 September 2011

Construct FetchXml queries the easy way

We all are lazy, I mean, we all like to minimize energy output and what better way of doing that than letting others do the work for you.

An easy way of creating a FetchXml query is to use Advanced Find. (http://<hostname>/ORG/AdvancedFind/AdvFind.aspx)
  1. Create the query that you wish to use. Bear in mind that some attributes might be missing, you can always add them later manually to your code.
  2. Run Query, i.e. Press Find
  3. Type the following in the address bar:   javascript:prompt("q:", resultRender.FetchXml.value);
  4. Hit Enter
  5. You'll get a window with the query used. Grab it and use it in your code.
If you don't have an address bar, in step 3, just hit Ctrl + N.

Luckily, Microsoft has recognised the productivity gains that this (easily generated FetchXml queries) can generate and has included a button to do this in Dynamics CRM 2011.

Saturday 17 September 2011

grep in Windows (dos)

I sometimes wish that there was a grep command in windows, well, it turns out that there is, sort of, it's called find.

You can just pipe the output of a command to find and it will do pretty much the same as grep, e.g.
netstat -anp TCP | find  "5555"
TCP    0.0.0.0:5555          0.0.0.0:0              LISTENING
TCP    10.168.2.115:5555     10.168.2.115:39898    ESTABLISHED
TCP    10.168.2.115:39898    10.168.2.115:5555    ESTABLISHED
You can use the switch /v to get all lines that don't contain the string:
dir | find /V ".cer"
 Volume in drive C has no label.
 Volume Serial Number is B8E9-71D1

 Directory of C:\Dev\CACerts

05/09/2011  18:26    <DIR>          .
05/09/2011  18:26    <DIR>          ..
12/08/2011  18:52             3,298 tony.pfx
12/08/2011  19:19             3,274 tonytest.pfx
              2 File(s)         6,572 bytes
               2 Dir(s)   4,861,059,072 bytes free
It's possible to pipe the output multiple times. In the example below you'll see only established connections on port 5555, which may or may not be of any use but it's neat.
netstat -anp TCP | find "ESTABLISHED" | find "5555"
TCP    10.168.20.115:5555     10.168.20.115:41184    ESTABLISHED
TCP    10.168.20.115:41184    10.168.20.115:5555     ESTABLISHED
I guess you can use it as a sort of logical and capability, similarly using /V switch , you can use it as not and, e.g. for established and not in port 5555
netstat -anp TCP | find "ESTABLISHED" | find /V "5555"
 TCP    10.168.20.115:445      10.168.20.101:1974     ESTABLISHED
 TCP    10.168.20.115:3389     10.168.20.203:60815    ESTABLISHED

Friday 16 September 2011

SSL Testing with VBS (winhttprequest)

I have a bit of a love/hate relationship with VBS, perhaps because not all of our boxes have PowerShell installed and I'm forced to use VBS a lot of the time for testing web services and an other assorted things.

We have been testing a website that uses client certificate mapping in IIS, see step 8 on this post, with the added twist that we have no other way of authentication, it's client certificates or bust. We are  sort of forced to POST a soap request, as the main method that we are testing, has way too many parameters for a GET request.
In my example script, see below, I'm using a client certificate installed in the user's store, this is more common than a client certificate installed in the Local Computer's store. If this is the case change line 17 to "LOCAL_MACHINE\Personal\1234" or whatever your certificates's name is.

   1 Dim objwinhttp, strURL, strStatus, soap
   2 
   3 soap = "<soap:Envelope xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:SOAP-ENC='http://schemas.xmlsoap.org/soap/encoding/' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>"
   4 soap =  soap + "<soap:Body>"
   5 soap = soap + "<MyMethod xmlns='http://server.dev.com/MyMethod'><Centre>string</Centre>....manymoreparemetershere</MyMethod>
   6 soap = soap + "</soap:Body></soap:Envelope>"            
   7 
   8 strURL = "https://server.dev.com:5555/service.asmx" 
   9 
  10 set objwinhttp = CreateObject("WinHttp.WinHttpRequest.5.1")
  11 
  12 objwinhttp.Open "POST", strURL
  13 objwinhttp.setRequestHeader "Content-type","text/xml; charset=utf-8"
  14 objwinhttp.setRequestHeader "SOAPAction","http://dom.dev.com/MyMethod"
  15 objwinhttp.setRequestHeader "Content-Length",Len(soap)
  16 
  17 objwinhttp.SetClientCertificate "CURRENT_USER\MY\1234"
  18 
  19 objwinhttp.Send(soap)
  20 
  21 wscript.Echo("Response Code: " & objwinhttp.Status & " Response: " & objwinhttp.ResponseText)

Wednesday 14 September 2011

SSL woes, .NET framework to the rescue.

We have been struggling with some SSL issues and I remembered that last time we had similar issues, on a different project, the .NET built-in trace logging was a life saver, as it allowed to see what was going in, so I thought I would post this here.
All that is needed is to add the following to your app.config file and make sure that the account running the app has the right permissions:

<system.diagnostics>
        <trace autoflush="true" />
            <sources>
                <source name="System.Net" maxdatasize="1024">
                    <listeners>
                        <add name="mytracefile"/>
                    </listeners>
                </source>
              <source name="System.Net.Sockets" maxdatasize="1024">
                    <listeners>
                        <add name="mytracefile"/>
                    </listeners>
                </source>  
           </sources>
            <sharedListeners>
                <add name="mytracefile" type="System.Diagnostics.TextWriterTraceListener"
                  initializeData="c:\ssltrace.log" />
            </sharedListeners>
            <switches>
                <add name="System.Net" value="Verbose" />
              <add name="System.Net.Sockets" value="Verbose" />
            </switches>
</system.diagnostics>

Original source can be found here.

Note, that you can also have this on your web.config to see what happens on the other end, but note that this is of limited utility for diagnosing SSL issues, as it will only log anything once the SSL handshake has completed.

Monday 12 September 2011

Bespoke vs COTS

It seems that, lately, all that our sales people push are COTS packages, which I guess is fine. They are cheaper (possibly), quicker to deploy (maybe) and easier to maintain (not guaranteed), but unfortunately, require a significant level of understanding from the customer of what they are actually getting. Yes, they are customizable, some products even, extremely customizable, but that doesn't mean that you will be able to do everything that you want with it.  You will have to learn to compromise, if you want the application to do exactly what you want, then go bespoke and stop wasting everybody's time. We'd all be happier that way.

Sunday 11 September 2011

The dreaded fetchxml 5000 results limit.

Following on from my last post, I was trying to finish up a windows service to clear all the old calendars in our system, when I hit the dreaded 5000 limit in fetchxml queries.

In essence, by default, the most results an fetchxml query will return is 5000. I vaguely remembered this limit from a while ago when we had another deletion situation like this, of a custom entity but that is by the by. I had a look on-line and found that most solutions use a DataSet object, which did not seem like the right object for me this time, as all I wanted was a list of GUIDs, so I just used a List<string> object (calendars in the code example below) and pass each GUID, after it's been stripped of its wiggly brackets ({}), to the deletion method described in my previous post.

Code Example:
   1 bool finished = false;
   2 int fetchPage = 0;
   3 
   4 while (!finished)
   5      {
   6          string fetch = "<fetch version='1.0' mapping='logical' page='" + fetchPage.ToString() + "' count='5000'><entity name='calendar'><attribute name='calendarid'/><filter type='and'>"
   7              + "<condition attribute='createdon' operator='lt' value='" + deletionDate.ToString("yyyy-MM-dd") + "'/>"
   8              + "<condition attribute='modifiedon' operator='lt' value='" + deletionDate.ToString("yyyy-MM-dd") + "'/>"
   9              + "</filter></entity></fetch>";
  10 
  11          string result = crmService.Fetch(fetch);
  12 
  13          XmlDocument doc = new XmlDocument();
  14          doc.LoadXml(result);
  15 
  16          //We'll be finished when nomorerecords is 0. 
  17          if (Convert.ToInt32(doc.FirstChild.Attributes["morerecords"].Value) == 0)
  18          {
  19              finished = true;
  20          }
  21 
  22          XmlNode resultsetNode = doc.SelectSingleNode("//resultset");
  23 
  24          foreach (XmlNode xn in resultsetNode.ChildNodes)
  25          {
  26              //Not the most elegant solution, but I guess it should be quicker than a regex.
  27              calendars.Add(xn.FirstChild.InnerXml.Remove(0, 1).Replace("}", string.Empty));
  28          }
  29 
  30          fetchPage++;
  31      }

Note, that you can set TurnOffFetchThrottling in the registry(HKLM\Software\Microsoft\MSCRM) to 1, to get around this problem as well, but this might create problems performance problems, which is why Microsoft must have introduced this arbitrary limit.

Wednesday 7 September 2011

Why Microsoft, Why?

In one of our Dynamics CRM app we make heavy use of facility/equipment calendars. Ordinarily this would not be even worthy of mention, never mind a post, but Microsoft decided that it would be a bad idea to delete those calendars. Nominally deletion of calendars is supported in the SDK documentation, but in practice, things are a little bit different.

Anyway, the fact of the matter is that we have built up a rather large collection of calendars and calendar rules (90K & 550K last time I checked)  and this is slowing our users down. Some operations are slightly slower, to the point that they have not noticed, whereas others are simply painstakingly slow (A custom screen can take up to 1 minute to load), so I (re)used a very simple method to delete a calendar:

   1 [WebMethod]
   2 public bool Delete(string strEntityName, string strEntityGuid)
   3         {
   4             try
   5             {
   6                 CreateLog("Trace", "Delete:: Delete( string strEntityName, string strEntityGuid)",
   7                                    "1. IN. strEntityName: " + strEntityName +
   8                                    " - strEntityGuid: " + strEntityGuid + " .");
   9 
  10                 // Set up the CRM Service.
  11                 //CrmService lCrmService = new CrmService();
  12                 //lCrmService.Credentials = System.Net.CredentialCache.DefaultCredentials;
  13                 //lCrmService.Url = ConfigurationSettings.AppSettings["CRMServiceURL"];
  14 
  15                 Guid objEntityId = new Guid(strEntityGuid);
  16 
  17                 CreateLog("Trace", "Delete:: Delete( string strEntityName, string strEntityGuid)", "2. About to Call Delete.");
  18 
  19                 lCrmService.Delete(strEntityName, objEntityId);
  20 
  21                 CreateLog("Trace", "Delete :: Delete( string strEntityName, string strEntityGuid)", "3. Delete Complete. Returning True");
  22                 CreateLog("Audit", "Delete :: Delete( string strEntityName, string strEntityGuid)", "Deleted Call with ID: " + strEntityGuid + " .");
  23 
  24                 return true;
  25             }
  26             catch (Exception ex)
  27             {
  28                 CreateLog("Error", "Delete :: Delete( string strEntityName, string strEntityGuid)", "Error: " + ex.ToString() + " .");
  29                 return false;
  30             }
  31         }       

but alas I got the following error message, when I passed in a calendar:
Crm Exception: Message: The object you tried to delete is associated with another object and cannot be deleted., ErrorCode: -2147220953
Fair enough, I thought, I need to delete the calendarrules first, unfortunately I got this error message:
Crm Exception: Message: The 'Delete' method does not support entities of type 'calendarrule'., ErrorCode: -2147219456
No dice. The irritating part is that we can delete the calendars directly with a sql statement, a rather convoluted sql statement actually, but this is:
  1. unsupported
  2. would require a lot of testing
  3. probably definitely a bad idea
After a bit of digging around and a somewhat inspired guess, I have managed to delete calendars, the code, see below, needs to be tested and it's not production ready, you don't say, but it does delete calendars, at least the calendars linked to facilities/equipment, which is the ones we are interested about.

Source Code:
   1 /// <summary>
   2 /// Delete passed in Calendar
   3 /// </summary>
   4 /// <param name="strEntityGuid">Calendar id</param>
   5 /// <returns>true if successful</returns>
   6 [WebMethod]
   7 public bool TDelete(string strEntityGuid)
   8 {
   9     try
  10     {
  11         CreateLog("Trace", "DeleteEntity :: TDelete( string strEntityName, string strEntityGuid)",
  12                            "1. IN. strEntityGuid: " + strEntityGuid + " .");
  13 
  14 
  15         calendar acal = DeleteCalendarRules(strEntityGuid);
  16 
  17         if (acal != null)
  18         {
  19             //Now, we need to find the main calendar for the equipment this calendar (i.e. strEntityGuid) belongs to
  20             //and from that calendar we need to find the calendarrule that we need to delete
  21 
  22             //grab the equipmentid from the name of the calendar so that we can get the equipment this calendar belongs to, remember 
  23             //that calendars are named as equipmentid + date
  24             string equipmentid = System.Text.RegularExpressions.Regex.Match(acal.name.ToUpper(), @"[A-F0-9]{8}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{12}").Value;
  25 
  26             //grab the equipment to which calendar strEntityGuid "belongs"
  27             equipment myequip = (equipment)lCrmService.Retrieve(EntityName.equipment.ToString(), new Guid(equipmentid), new AllColumns());
  28 
  29             //grab the calendar for the above equipment. This is because there is a calendarrule reference to this calendar from strEntityGuid
  30             calendar equipmentcal = (calendar)lCrmService.Retrieve(EntityName.calendar.ToString(), myequip.calendarid.Value, new AllColumns());
  31 
  32             //grab the aforementioned calendarrule so that we can set it to null. Note that this should be unique.
  33             string fetch = "<fetch version='1.0' output-format='xml-platform' mapping='logical' distinct='false'><entity name='calendarrule'><attribute name='calendarruleid'/><filter type='and'><condition attribute='calendarid' operator='eq' value='" +
  34          myequip.calendarid.Value.ToString() + "'/><condition attribute='innercalendarid' operator='eq' value='" + strEntityGuid + "'/></filter></entity></fetch>";
  35 
  36             string result = lCrmService.Fetch(fetch);
  37 
  38             XmlDocument doc = new XmlDocument();
  39             doc.LoadXml(result);
  40             string guid = doc.SelectSingleNode("//result").FirstChild.InnerText;
  41 
  42             //loop through all calendar rules for the equipment's calendar 
  43             //and "drop", i.e. set to null, when it matches the result of the fetch query.
  44             //We update the calendar so that the calendarrule is "dropped", this is handled on the background by the platform.
  45             for (int i = 0; i < equipmentcal.calendarrules.Length; i++)
  46             {
  47                 calendarrule mycalrul = equipmentcal.calendarrules[i];
  48 
  49                 if (mycalrul.calendarruleid.Value == new Guid(guid))
  50                 {
  51                     equipmentcal.calendarrules[i] = null;
  52                     lCrmService.Update(equipmentcal);
  53                 }
  54             }
  55 
  56 
  57             TargetDeleteCalendar target = new TargetDeleteCalendar();
  58             target.EntityId = new Guid(strEntityGuid);
  59 
  60 
  61             DeleteRequest delete = new DeleteRequest();
  62             delete.Target = target;
  63 
  64             CreateLog("Trace", "DeleteEntity :: TDelete( string strEntityName, string strEntityGuid)", "2. About to Call Delete.");
  65 
  66             //Delete the bad mother
  67             DeleteResponse deleted = (DeleteResponse)lCrmService.Execute(delete);
  68 
  69             CreateLog("Trace", "DeleteEntity :: TDelete( string strEntityName, string strEntityGuid)", "3. Delete Complete. Returning True");
  70             CreateLog("Audit", "DeleteEntity :: TDelete( string strEntityName, string strEntityGuid)", "Deleted Callaback with ID: " + strEntityGuid + " .");
  71 
  72             return true;
  73         }
  74         else
  75         {
  76             return false;
  77         }
  78     }
  79     catch (Exception e)
  80     {
  81         CreateLog("Error", "DeleteEntity :: TDelete( string strEntityName, string strEntityGuid)", "Error: " + e.ToString() + " - source: " + e.Source + " .");
  82         return false;
  83     }
  84 }
  85 
  86 /// <summary>
  87 /// This doesn't actually delete the calendarrules associated with the passed in calendar.
  88 /// It simply sets its deletionstatecode to 2, so that the deletion service deletes it
  89 /// </summary>
  90 /// <param name="strEntityGuid">Calendar id</param>
  91 /// <returns>calendar entity</returns>
  92 private calendar DeleteCalendarRules(string strEntityGuid)
  93 {
  94     try
  95     {
  96         calendar calTBDeleted = (calendar)lCrmService.Retrieve(EntityName.calendar.ToString(), new Guid(strEntityGuid), new AllColumns());
  97 
  98         //this sets the calendar rules to null. I love this comment :)
  99         calTBDeleted.calendarrules = null;
 100 
 101         lCrmService.Update(calTBDeleted);
 102 
 103         return calTBDeleted;
 104     }
 105     catch (Exception e)
 106     {
 107         //TODO: log the exception don't be lazy
 108         return null;
 109 
 110     }
 111 }