Tuesday, 10 July 2018

Azure On-premise Backup on Windows 2008 R2

We have one server that runs Windows 2008 R2 still, too long to explain, and we've recentely decided to move our backups to Azure.

I had no issues on Windows 2016 or 2012 but the backups would not run on Windows 2008 R2 unless Backup Now was click, which is not exactly what one wants from a backup solution.

A scheduled task is created/amended every time you set backup schedule, which is reasonable enough, but the problem that I had was that the task was not working.

This is the task in question



Add Arguments (optional):

-command Import-Module MSOnlineBackup; Start-OBBackup -Name "2cdeaf83-dead-c0de-beef-c345bead15b1
When I tried to run this on powershell, I got this error:

Import-Module : The specified module 'MSOnlineBackup' was not loaded because no valid module file was found in any module directory.
I looked for the path where MSOnlineBackup was and found it here:

'C:\Program Files\Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup\MSOnlineBackup.psd1'

So, I added this path to the system path and ..... nothing happened, same error.

Desperate times, call for desperate measures, so I changed the arguments on the task to:
-command Import-Module 'C:\Program Files\Microsoft Azure Recovery Services Agent\bin\Modules\MSOnlineBackup\MSOnlineBackup.psd1'; Start-OBBackup -Name "2cdeaf83-dead-c0de-beef-c345bead15b1"

This works.

A word of caution though, any changes to the schedule, will result in the task going back to what it was and thus will not work, so I would suggest creating a new task and ignore the standard task.

I have not tried rebooting the server, which seems to be required for path changes, so use this if you can't reboot for a while???

Monday, 16 April 2018

Homemade Energy Bars

For those of you who know me, you know that I do cycle quite a bit, today I've decided to share the secret of my success to the cycling world by posting my recipe for homemade energy bars.

  • 300 g Porridge Oats
  • 280 g Peanut Butter
  • 250 g Lyle's Syrup

  • Mix together
  • Spread on a tray (approx 10" by 10")

The good thing about this recipe is that it's extremely flexible. You want to add protein powder, go right ahead, I would suggest that you sift it first but why not? Seeds? Nuts? sure. I've even added raw cocoa powder (carefully sifted, otherwise the bars will surprise you with chunks of cocoa powder)


Friday, 11 August 2017

Windows event log service Error 5: Access is Denied

So a few weeks back we had an issue where we would get this error

    Error 5: Access denied

If the event log service is down troubleshooting things can be a bit tricky, plus the SMTP service depends on the Event Log service, which was a problem for us as we use the SMTP service heavily.

In our case, we had an error that indicated that a dependent service had not started and thus this service could not start. This is where it gets interesting.

The Event Log service had no dependencies, at least as far as the services console would suggest and yet the error suggested that there were dependencies.

After a while we looked at this hive in the registry:


Lo and behold, there was a dependency

Namely DependOnService was populated with a non-existent service, which is why I guessed that it wasn't being displayed on the services console.

So we deleted the key and the service started.

Saturday, 24 December 2016

Travelling to London by train

I currently work for a very small software company, which is big change for me as this company has 3 orders of magnitude fewer employees than any other company I've worked for before. There are many changes for me, but the one I want to talk about today is how I approach expenses.

In previous jobs, I didn't care about expenses, if the company wanted me to go somewhere I would expect the company to pay the full cost, even if it meant silly money for a train ticket due to being asked to be in London next morning, but in my current job I almost feel that I'm paying for it personally, which means, among other things, that I try to minimize travel or look for cheaper alternatives.

A few weeks ago I had to go from Sheffield to London (~ 170 miles) to see a client for an urgent meeting in the morning with just two days notice, so I had several alternatives:

Means of Travel Cost To Company
Car to Customer site £105
Car To North London, then Tube £100
Train, then Tube £186

Had the meeting been around 13:00 I would've been able to catch an off-peak train so this is what the costs would be:

Means of Travel Cost To Company
Car to Customer site £105
Car To North London, then Tube £100
Train, then Tube £80

It is clear that the train companies use price sensitivity to the full, particularly when it comes to travel to London, but there is a way to fight back and that is to buy the train ticket in tranches, which is what I did:

Means of Travel Cost To Company
Sheffield To Leicester £34
Leicester To Market Harbourough £11.6
Market Harbourough to Luton £27
Luton to London St Pancras £27.5
Tube £5.6

Including booking fee, the grand total was £107, which is slightly more than drive but has two major advantages:
  • Arrival Time is significantly less dependent on external factors (namely traffic)
  • I can work on the train.
It's only fair to point out that some of these are day returns so it wouldn't work for all instances and also that booking tickets in advance can result in pretty reasonable prices too.

I made a point about travel to London, I think other cities like Leeds, Manchester and Birmingham also have peak time fares but the premium is not anywhere near as much as to London.

Finally, it would probably be trivial to optimize the cost of the tickets by using the available API or even national rail's website itself, given it's query friendly url format, not tried it though

Saturday, 10 December 2016

Different Approaches to problem solving - Typescript Regular Expression escaping.

One our developers left the company last week and one of the last things he did was to expand the password reset functionality so that managers can change the passwords of their direct reports on the system (most of our users don't have emails and we've not got SMS messaging working yet)

In any case, the password complexity rules are as follows:
  • The password length must be greater than or equal to 8
  • The password must contain one or more uppercase characters
  • The password must contain one or more lowercase characters
  • The password must contain one or more numeric values
This is the regex used client side to validate these rules.


This is the actual regular expression that the browser was seeing after transpiling:


There are no typos, the second group is missing the backward slash before the d,  while the third group has kept the backslash before the n, I don't understand why this is, but we can ignore this for the purposes of this little tale.

In any case, the intention was this:

The actual expression was this:

This would not have been an issue had it been tested properly or had our convention for development passwords deviated from variations on leet speak versions of the word password, which contained the required d.


So the new guy joins us and he has this issue assigned to him. He clearly struggles to understand the issue, although I make it clear to him that if he gets stuck he should not hesitate to talk to me, but he doesn't. In fact, in the first two days he only asks me about a single issue, which turns out to be a node issue and was actually preventing him from getting started.

In any case, at the end of his second day of work, he submits a pull request and goes offline. I look at the pull request and this is what he came up with:


This is exactly what is required and is far simpler than the original regular expression, while at the same time bypassing any potential escaping issues.

I'll state the obvious, which is:
Sometimes a head on approach is less likely to solve an issue than a side ways approach

If you know why the escaping issue (We're using Aurelia) leave a comment.

Saturday, 27 August 2016

Have I broken LinkedIn's Job Recommendation Engine?

I obviously haven't broken the engine, sorry about the click bait title but I have found that the job recommendation engine is actually pretty rubbish, it's only taken me around four years to find out.

Back in 2012 or thereabouts, I was working as a Dynamics CRM developer when I started using LinkedIn. LinkedIn has this feature, where emails containing job ads that match one's preferences are sent out, even if one hasn't set said preferences. By and large these jobs where relevant, sure, they would contain the odd random job, which as far as I could work out was only relevant geographically, i.e. they were in the city I live in or the odd job that related to actual Customer Relationship Management, but in general the emails contained relevant job ads.

This situation continued more or less unchanged for the next 3 years while I continued working as a Senior/Lead Dynamics CRM developer and then it wasn't much changed when I became a Solutions Architect, apart from the frequency of the emails, which markedly increased.

Sure, the emails did contain the odd job ad for old fashioned architects, you know the ones that design buildings and they would also contain job ads for other IT architects (Infrastructure, Networking, Cloud Technologies (AWS/Azure mostly)) and even for developer roles but they were mostly relevant.

About a year ago, I became the Head of IT of a small company and that's when LinkedIn's Job Recommendation Engine started going pear shaped.

It turns out that Heat of IT seems to be too vague a title for the engine to deal with so the emails just contain jobs that revert back to mostly geographical significance and I say mostly because  the emails now contain  job ads that are completely irrelevant, e.g. secretary 100 miles from home, see a list of f jobs contained in the last email:

Finance Analyst - Home town
Network Engineer - Home town
Planning Manager - Home town
Ruby Web Developer - 50 miles from home.
PHP Developer - Home town

I did get a few emails a couple of month back that contained a job listing for a Head of IT and another for a Head of Technology, but given that these emails seem to be coming weekly or more often, this is very poor.

What is strange is that when I first joined this company I used to get emails containing jobs for Head of just about everything but IT, I'm probably exaggerating a little bit here as this is all from memory until I started to twig that something had gone wrong with the emails.

Maybe, LinkedIn is trying to tell me something about my choice of career.

Thursday, 31 March 2016

The Pluralsight Mobile App Sucks

There is a request to allow offline storage of courses on the SD card but it's been active for over two years now.

This can't be an insurmountable problem as quite a few apps quite happily allow this:

In any case, yesterday after I could not update several apps for lack of space for about the umpteenth time, I thought, enough is enough and I endeavored to find a solution.

I first wrote a small script in PowerShell that called Handle.exe on a loop, to get the list of files being opened by the pluralsight desktop app and wrote that list of files to a file:

if (-not (test-path 'handle.exe'))
 write-host "Can't find handle.exe"

$files= @()

 $video =.\Handle.exe -p pluralsight | Select-String mp4
 if ($video)
  $path = $video.Line.Split(')')[1] 

  if (-not($files.Contains($path)))
   Write-Host $path
   Write-Host "Files Added So far: " + $($files.Count)
   $files > file.txt

This required me to click on each chapter (video) of the course to get it to be logged by handle, which while a little bit annoying I thought could be automated somehow with CodedUI or AutoHotKey but the former was not very reliable and I didn't have the patience for the latter.

The above script worked on the first course I tried but it was a bit hit and miss on the second and after a lot of playing about I realized that for small videos it would simply not work, not sure why that is, is it reading the file quicker than the loop loops?  I've not stopped to do the maths but it seems odd in any case.

The second part of the process was the script below, which moves the files to a new folder so that they can be copied safely across to the phone/tablet:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][ValidateScript({Test-Path $_})][string]$destination)

$files = Get-Content $sourceFile
$counter = 0 

foreach($file in $files)
 $dest = $(join-path $destination $counter) + ".mp4"
 Copy-Item $file.Trim() $dest

If the first script grabbed all files, then this script worked fine, but like I said, it didn't always do it, it seemed not to work for small files.

So, I did a bit more googleing, alas, it seems that there is no strace equivalent in Windows and then when I was about to give up*, I hit upon Process Monitor, which does exactly what I needed, in other words, and among other things, it can list the files that an application opens. In order to do this, all that is needed is the correct filter and a little bit more of PowerShell scripting.

Open Process Monitor -> Filter and set the following filters (in green):

In reality, it's probably enough to do the path ends with mp4 filter as no operation seems to generate a single entry for a file read, which means that some processing will be needed, but the QueryNetworkOpenInformationFile Operation seems to generate the fewer hits so I stuck with that.

Note, sometimes it simply doesn't register anything, in which case, just restart Process Monitor.

With this running, I can now go through all the chapters of the course that I want to watch on my mobile and I get something like this:

Now it's time to export the results to a CSV file, which you can do by simply pressing CTRL + S

Finally, it's time to process the file that I've just saved, for which I slightly modified the second PowerShell script:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][ValidateScript({Test-Path $_})][string]$destination)

$source = Import-CSV $sourceFile
$source.Path | % { if (!$files.Contains($_)){$files+=$_}}

$counter = 0 

foreach($file in $files)
 if ($counter -lt 10) { $prefix= "0" + $counter} else {$prefix=$counter}
 $dest = $(join-path $destination $prefix) + ".mp4"
 Copy-Item $file.Trim() $dest

The above script is fine, but I'd rather have a single file than loads of files, so I use ffmpeg to concatenate the clips:
param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][string]$destFile, [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$ffmpeg)

$source = Import-CSV $sourceFile
$source.Path | % { if (!$files.Contains("file '" + $_.Replace('\','\\') + "'")){$files+="file '" + $_.Replace('\','\\') + "'"}}

$fileList = (Get-Date).Ticks

Set-Content -Path $fileList -Value $files

$x= $ffmpeg + " -f concat -i $filelist -codec copy $destFile"

Invoke-Expression $x

Remove-Item $fileList
Remove-Item $sourceFile -Confirm
Using this technique is actually quite flexible I can create individual videos for each module if a single video for the whole course is too unwieldy.

Unfortunately, this last step doesn't appear to work too well on new pluralsight videos, which is annoying as there doesn't appear to be a free Video Player for Android that does all of these:
  • Play all files from a directory.
  • Play at up 1.5x speed.
  • Allow the video to fill the screen.
So I say that there is no free video player that does this at the moment for Android.

Saturday, 27 February 2016

Collating and Plotting Perfmon data

We've been using Perfmon to collect information about the various apps we have for quite a while now and it was always a massive pain to present the information in a meaningful manner.

The problem is that each file contains a day's worth of data, I realize that it's possible to keep a counter for longer but the comparison problem remains the same and we wanted to be able to compare performance counters on different days.

I thought I would try to use Python and the pandas library for this.

I installed the pandas library by installing Anaconda, in particular I'm using version 2.3 for Python 2.7, yes it's all a bit confusing and no, I'm not using Visual Studio but Spyder, which isn't' great but mostly works

The script below is very much tailor made to my current needs, so there are few quirks:
  1. It assumes all files will be full day files at 15 seconds intervals.
  2. It assumes that all files will have the same counters in the same order. It should be relatively trivial to change the script to take a config file with the counters needed.
  3. It calculates a total CPU usage of all processes except SQL Server by subtracting SQL server usage from total cpu usage.
There are several command line options that should, hopefully, be self explanatory

Without further ado here's the script:

Needs a bit more work - 
#way to limit the number of days to be plotted, 
#explore different plots perhaps for different counters
#Actually get the files from the servers

from os import listdir, mkdir
from os.path import isfile, join, exists
import pandas as pd
import matplotlib.pyplot as plt
import re
import sys

def RenameColumns (current, fileName):
    current = current[:-1]
    return current
def Process (allFiles, numberToPlot, category, currentFile):
    ##Add the date column. This will not work if different files start at different times and have different intervals
    df = pd.DataFrame(allFiles[0][1][allFiles[0][1].columns[0]])
    #Rename the column to date
    df.columns = ['date']
    #We only need the time, not the date.
    df['date'] = df.date.apply(lambda x:x.split(' ')[1])
    #Get the last file number
    lastFile = currentFile + numberToPlot
    if lastFile > len(allFiles):
        lastFile = len(allFiles)        
    #Process each file into a dataframe
    for fi in range(currentFile, lastFile):
        fileName = allFiles[fi][0]
        df = df.join(allFiles[fi][1][allFiles[fi][1].columns[counter+1]])
        df.columns = RenameColumns(list(df.columns), fileName)
        #This is not ideal but to provide adjacent values would have to loop through i guess.
  #I'm sure there are better ways of dealing with missing values than this
        df[fileName] = df[fileName].apply(lambda x: 0.0 if x ==' ' else x ).astype(float)
    #Return a tuple with name and Dataframe.
    return (allFiles[currentFile][0] +"-" + allFiles[lastFile-1][0] + "-" + category, df)       

def GetCPUUsageRest (allFiles, numberToPlot,numberOfCPUs,  currentFile):
     #Is this really needed...Test you say? ...    
    numberOfCPUs = float(numberOfCPUs)
     ##Add the date column
    df = pd.DataFrame(allFiles[0][1][allFiles[0][1].columns[0]])
    #Rename the column to date
    df.columns = ['date']
    #Get the time
    df['date'] = df.date.apply(lambda x:x.split(' ')[1])
    #Get the last file
    lastFile = currentFile + numberToPlot
    if lastFile > len(allFiles):
        lastFile = len(allFiles)        
    #Process each file into a dataframe
    for fi in range(currentFile, lastFile):
        fileName = allFiles[fi][0]
        #Hardcoding FTW because 
        total= pd.DataFrame(DataFrames[0][1])
        sql = pd.DataFrame(DataFrames[25][1])
        sql[fileName] = sql[fileName].apply (lambda x:x/numberOfCPUs)
        df = df.join(pd.DataFrame(total[fileName] - sql[fileName]))
        df.columns = RenameColumns(list(df.columns), fileName)
        #This is not ideal but to provide adjacent values would have to loop through i guess
        df[fileName] = df[fileName].apply(lambda x: 0.0 if x < 0 else x ).astype(float)
    #Return a tuple with file name and Dataframe
    return (allFiles[currentFile][0] +"-" + allFiles[lastFile-1][0] + "-" + "Rest-Process % CPU" , df)      
#These are the counters used. The files need to contain them all otherwise the plots will be all wrong
#I suppose that we could load a list of these from a config file to accomodate changes rather than this
#hard coding.    
counters.append('Processor(_Total)-% Processor Time')
counters.append('Memory-Available KBytes')
counters.append('.NET CLR Exceptions(_Global_)-# of Exceps Thrown  per  sec')
counters.append('.NET CLR Memory(_Global_)-% Time in GC')
counters.append('ASP.NET-Application Restarts')
counters.append('ASP.NET-Request Wait Time')
counters.append('ASP.NET-Requests Queued')
counters.append('ASP.NET-Request Execution Time')
counters.append('ASP.NET Applications(__Total__)-Requests per Sec')
counters.append('Memory-Pages per sec')
counters.append('PhysicalDisk(_Total)-% Disk Read Time')
counters.append('System-Processor Queue Length')
counters.append('Web Service(_Total)-Get Requests per sec')
counters.append('Web Service(_Total)-Post Requests per sec')
counters.append('Web Service(_Total)-Current Connections')
counters.append('SQLServer-Access Methods-Page Splits per sec')
counters.append('SQLServer-Buffer Manager-Buffer cache hit ratio')
counters.append('SQLServer-Buffer Manager-Page life expectancy')
counters.append('SQLServer-Buffer Manager-Checkpoint pages per sec')
counters.append('SQLServer-General Statistics-User Connections')
counters.append('SQLServer-General Statistics-Processes blocked')
counters.append('SQLServer-SQL Statistics-Batch Requests per sec')
counters.append('SQLServer-SQL Statistics-SQL Compilations per sec')
counters.append('SQLServer-SQL Statistics-SQL Re-Compilations per sec')
counters.append('Network Interface-Bytes Total per sec')
counters.append('SQLServer-Process % CPU')

#This is regex for dates in the format yyyyMMdd. It doesn't quite validate as it would happily allow a day of 39
regexDate = '(20\d{2}[0-2]\d{1}[0-3]\d{1})'
#Set plotting style

cmdargs = str(sys.argv)

if len (sys.argv) != 6 :
    print "This script takes five arguments"
    print "First Argument should be path where csv files to be processed are"
    print "Second Argument should be path where plots will be stored"
    print "Third Argument should be server name"
    print "Forth Argument should be number of Days to Plot"
    print "Fifth Argument should be number of CPUs"

path = sys.argv[1].rstrip('\\"')
dest = sys.argv[2]
server = sys.argv[3]
numberOfDaysToPlot = int(sys.argv[4])
numberOfCPUs = int(sys.argv[5])

if not exists(dest):

DataFrames = []
#Read the files for the input path. 
files = [f for f in listdir(path) if isfile(join(path,f)) and f.endswith('csv')]
for f in files:    
    name = re.search(regexDate, f).group()
    allFiles.append((name, pd.read_csv(join(path,f))))

remainingFiles = len(files)
#It looks like perfmon adds an extra column at the end of the file, which isn't a counter, hence the - 2
for counter in range(len(allFiles[0][1].columns)-2):
    currentFile = 0 
    while remainingFiles > 0:
        DataFrames.append(Process(allFiles, numberOfDaysToPlot, counters[counter], currentFile))
        #if we've reached the last counter, then we can look at the rest
        if counter == len(allFiles[0][1].columns)-3:
            DataFrames.append(GetCPUUsageRest(allFiles, numberOfDaysToPlot, numberOfCPUs, currentFile))
        remainingFiles = remainingFiles - numberOfDaysToPlot
        currentFile = currentFile + numberOfDaysToPlot
    remainingFiles = len(files)

CPUylim = ()     
#All plotting is done here.
for d in DataFrames:
    p=d[1].plot(x='date', title=d[0])
    #Get Y Axis Limit for Total CPU
    if d[0].find('Processor(_Total)-% Processor Time') != -1:
        CPUylim = p.get_ylim()        
    if d[0].find('SQLServer-Process % CPU') != -1:
        p.set_ylim( CPUylim)
    if d[0].find('Rest-Process % CPU') != -1:
    #These two set the x ticks and labels make sure that they roughly match up.
    p.set_xticks([750.0,1500.0,2250.0,3000.0,3750.0,4500.0, 5250.0])
    fig = p.get_figure()
    fig.savefig(join (dest, d[0] + ".png"))
A sample plot generated

Monday, 15 February 2016

Can't afford a certificate

Today somebody at work told me to log in to our gotomypc account and I manually typed the address: https://www.gotomypc.co.uk/

Clearly the UK is not an important enough market to merit a proper cert or a SAN.

This is the certificate for the .com website:

Wednesday, 3 February 2016

MSDN Subscription - A Very British Rip-Off

So a few days back I was considering whether we should take the plunge and fork out for an MSDN subscription for all our developers, there is five of us and given where we want to be it made sense.

This was a cost that had been budgeted for so, in theory, this should not be a problem, but as with all large purchases, and for a small firm, 5 MSDN subscriptions is a Large purchase, which meant that we had to get approval from above so I went to this site to confirm prices.

5 x $1199 even at silly exchange rates, should be less than £5K, so I thought, this would be an easy sell as it was less than we had budgeted, boy was I wrong.

If you are in the UK, you can see for yourself here, which shows you this:


The exchange rate used by Microsoft is $1 = £1.01, rather than $1 = £0.68, so close to 50% surcharge. 

To put it another way, if we converted back to dollars, the price would be ~ $1780 rather than $1199.

I really like to hear an explanation from Microsoft, it certainly can't be the taxes the have to pay in this country, as the sales will probably be billed to the Irish subsidiary.

For the record this is the exchange rate today, according to Google:

and for the really lazy: 

I know there are resellers out there that do a cheaper price, it still does not change the fact that Microsoft itself applies an almost 50% surcharge to UK developers.