Travelling to London by train

I currently work for a very small software company, which is big change for me as this company has 3 orders of magnitude fewer employees than any other company I've worked for before. There are many changes for me, but the one I want to talk about today is how I approach expenses.

In previous jobs, I didn't care about expenses, if the company wanted me to go somewhere I would expect the company to pay the full cost, even if it meant silly money for a train ticket due to being asked to be in London next morning, but in my current job I almost feel that I'm paying for it personally, which means, among other things, that I try to minimize travel or look for cheaper alternatives.

A few weeks ago I had to go from Sheffield to London (~ 170 miles) to see a client for an urgent meeting in the morning with just two days notice, so I had several alternatives:

Means of Travel	Cost To Company
Car to Customer site	£105
Car To North London, then Tube	£100
Train, then Tube	£186

Had the meeting been around 13:00 I would've been able to catch an off-peak train so this is what the costs would be:

Means of Travel	Cost To Company
Car to Customer site	£105
Car To North London, then Tube	£100
Train, then Tube	£80

It is clear that the train companies use price sensitivity to the full, particularly when it comes to travel to London, but there is a way to fight back and that is to buy the train ticket in tranches, which is what I did:

Means of Travel	Cost To Company
Sheffield To Leicester	£34
Leicester To Market Harbourough	£11.6
Market Harbourough to Luton	£27
Luton to London St Pancras	£27.5
Tube	£5.6

Including booking fee, the grand total was £107, which is slightly more than drive but has two major advantages:

• Arrival Time is significantly less dependent on external factors (namely traffic)
• I can work on the train.

It's only fair to point out that some of these are day returns so it wouldn't work for all instances and also that booking tickets in advance can result in pretty reasonable prices too.

I made a point about travel to London, I think other cities like Leeds, Manchester and Birmingham also have peak time fares but the premium is not anywhere near as much as to London.

Finally, it would probably be trivial to optimize the cost of the tickets by using the available API or even national rail's website itself, given it's query friendly url format, not tried it though

Different Approaches to problem solving - Typescript Regular Expression escaping.

One our developers left the company last week and one of the last things he did was to expand the password reset functionality so that managers can change the passwords of their direct reports on the system (most of our users don't have emails and we've not got SMS messaging working yet)

In any case, the password complexity rules are as follows:

• The password length must be greater than or equal to 8
• The password must contain one or more uppercase characters
• The password must contain one or more lowercase characters
• The password must contain one or more numeric values

This is the regex used client side to validate these rules.

"(?=^.{8,}$)(?=.*\d)(?![.\n])(?=.*[A-Z])(?=.*[a-z]).*$"

This is the actual regular expression that the browser was seeing after transpiling:

"(?=^.{8,}$)(?=.*d)(?![.\n])(?=.*[A-Z])(?=.*[a-z]).*$"

There are no typos, the second group is missing the backward slash before the d,  while the third group has kept the backslash before the n, I don't understand why this is, but we can ignore this for the purposes of this little tale.

In any case, the intention was this:

The actual expression was this:

This would not have been an issue had it been tested properly or had our convention for development passwords deviated from variations on leet speak versions of the word password, which contained the required d.

D'OH

So the new guy joins us and he has this issue assigned to him. He clearly struggles to understand the issue, although I make it clear to him that if he gets stuck he should not hesitate to talk to me, but he doesn't. In fact, in the first two days he only asks me about a single issue, which turns out to be a node issue and was actually preventing him from getting started.

In any case, at the end of his second day of work, he submits a pull request and goes offline. I look at the pull request and this is what he came up with:

"((?=.*[0-9])(?=.*[A-Z])(?=.*[a-z])).{8,}$"

This is exactly what is required and is far simpler than the original regular expression, while at the same time bypassing any potential escaping issues.

I'll state the obvious, which is:

Sometimes a head on approach is less likely to solve an issue than a side ways approach

If you know why the escaping issue (We're using Aurelia) leave a comment.

Have I broken LinkedIn's Job Recommendation Engine?

I obviously haven't broken the engine, sorry about the click bait title but I have found that the job recommendation engine is actually pretty rubbish, it's only taken me around four years to find out.

Back in 2012 or thereabouts, I was working as a Dynamics CRM developer when I started using LinkedIn. LinkedIn has this feature, where emails containing job ads that match one's preferences are sent out, even if one hasn't set said preferences. By and large these jobs where relevant, sure, they would contain the odd random job, which as far as I could work out was only relevant geographically, i.e. they were in the city I live in or the odd job that related to actual Customer Relationship Management, but in general the emails contained relevant job ads.

This situation continued more or less unchanged for the next 3 years while I continued working as a Senior/Lead Dynamics CRM developer and then it wasn't much changed when I became a Solutions Architect, apart from the frequency of the emails, which markedly increased.

Sure, the emails did contain the odd job ad for old fashioned architects, you know the ones that design buildings and they would also contain job ads for other IT architects (Infrastructure, Networking, Cloud Technologies (AWS/Azure mostly)) and even for developer roles but they were mostly relevant.

About a year ago, I became the Head of IT of a small company and that's when LinkedIn's Job Recommendation Engine started going pear shaped.

It turns out that Heat of IT seems to be too vague a title for the engine to deal with so the emails just contain jobs that revert back to mostly geographical significance and I say mostly because  the emails now contain  job ads that are completely irrelevant, e.g. secretary 100 miles from home, see a list of f jobs contained in the last email:

Finance Analyst - Home town

Network Engineer - Home town

Planning Manager - Home town

Ruby Web Developer - 50 miles from home.

PHP Developer - Home town

I did get a few emails a couple of month back that contained a job listing for a Head of IT and another for a Head of Technology, but given that these emails seem to be coming weekly or more often, this is very poor.

What is strange is that when I first joined this company I used to get emails containing jobs for Head of just about everything but IT, I'm probably exaggerating a little bit here as this is all from memory until I started to twig that something had gone wrong with the emails.

Maybe, LinkedIn is trying to tell me something about my choice of career.

The Pluralsight Mobile App Sucks

There is a request to allow offline storage of courses on the SD card but it's been active for over two years now.

This can't be an insurmountable problem as quite a few apps quite happily allow this:

In any case, yesterday after I could not update several apps for lack of space for about the umpteenth time, I thought, enough is enough and I endeavored to find a solution.

I first wrote a small script in PowerShell that called Handle.exe on a loop, to get the list of files being opened by the pluralsight desktop app and wrote that list of files to a file:

if (-not (test-path 'handle.exe'))
{
 write-host "Can't find handle.exe"
 break;
}

$files= @()

while($true) 
{
 $video =.\Handle.exe -p pluralsight | Select-String mp4
 
 if ($video)
 {
  $path = $video.Line.Split(')')[1] 

  if (-not($files.Contains($path)))
  {
   Write-Host $path
   Write-Host "Files Added So far: " + $($files.Count)
   $files+=$path
   $files > file.txt
  }
 }
}

This required me to click on each chapter (video) of the course to get it to be logged by handle, which while a little bit annoying I thought could be automated somehow with CodedUI or AutoHotKey but the former was not very reliable and I didn't have the patience for the latter.

The above script worked on the first course I tried but it was a bit hit and miss on the second and after a lot of playing about I realized that for small videos it would simply not work, not sure why that is, is it reading the file quicker than the loop loops?  I've not stopped to do the maths but it seems odd in any case.

The second part of the process was the script below, which moves the files to a new folder so that they can be copied safely across to the phone/tablet:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][ValidateScript({Test-Path $_})][string]$destination)

$files = Get-Content $sourceFile
$counter = 0 

foreach($file in $files)
{ 
 $dest = $(join-path $destination $counter) + ".mp4"
 Copy-Item $file.Trim() $dest
 $counter++
}

If the first script grabbed all files, then this script worked fine, but like I said, it didn't always do it, it seemed not to work for small files.

So, I did a bit more googleing, alas, it seems that there is no strace equivalent in Windows and then when I was about to give up*, I hit upon Process Monitor, which does exactly what I needed, in other words, and among other things, it can list the files that an application opens. In order to do this, all that is needed is the correct filter and a little bit more of PowerShell scripting.

Open Process Monitor -> Filter and set the following filters (in green):

In reality, it's probably enough to do the path ends with mp4 filter as no operation seems to generate a single entry for a file read, which means that some processing will be needed, but the QueryNetworkOpenInformationFile Operation seems to generate the fewer hits so I stuck with that.

Note, sometimes it simply doesn't register anything, in which case, just restart Process Monitor.

With this running, I can now go through all the chapters of the course that I want to watch on my mobile and I get something like this:

Now it's time to export the results to a CSV file, which you can do by simply pressing CTRL + S

Finally, it's time to process the file that I've just saved, for which I slightly modified the second PowerShell script:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][ValidateScript({Test-Path $_})][string]$destination)

$source = Import-CSV $sourceFile
$files=@()
$source.Path | % { if (!$files.Contains($_)){$files+=$_}}

$counter = 0 

foreach($file in $files)
{
 if ($counter -lt 10) { $prefix= "0" + $counter} else {$prefix=$counter}
 $dest = $(join-path $destination $prefix) + ".mp4"
 Copy-Item $file.Trim() $dest
 $counter++
}

The above script is fine, but I'd rather have a single file than loads of files, so I use ffmpeg to concatenate the clips:

param ( [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$sourceFile, [Parameter(Mandatory)][string]$destFile, [Parameter(Mandatory)] [ValidateScript({Test-Path $_})][string]$ffmpeg)

$source = Import-CSV $sourceFile
$files=@()
$source.Path | % { if (!$files.Contains("file '" + $_.Replace('\','\\') + "'")){$files+="file '" + $_.Replace('\','\\') + "'"}}

$fileList = (Get-Date).Ticks

Set-Content -Path $fileList -Value $files

$x= $ffmpeg + " -f concat -i $filelist -codec copy $destFile"

Invoke-Expression $x

Remove-Item $fileList
Remove-Item $sourceFile -Confirm

Using this technique is actually quite flexible I can create individual videos for each module if a single video for the whole course is too unwieldy.

Unfortunately, this last step doesn't appear to work too well on new pluralsight videos, which is annoying as there doesn't appear to be a free Video Player for Android that does all of these:

• Play all files from a directory.
• Play at up 1.5x speed.
• Allow the video to fill the screen.

So I say that there is no free video player that does this at the moment for Android.

Collating and Plotting Perfmon data

We've been using Perfmon to collect information about the various apps we have for quite a while now and it was always a massive pain to present the information in a meaningful manner.

The problem is that each file contains a day's worth of data, I realize that it's possible to keep a counter for longer but the comparison problem remains the same and we wanted to be able to compare performance counters on different days.

I thought I would try to use Python and the pandas library for this.

I installed the pandas library by installing Anaconda, in particular I'm using version 2.3 for Python 2.7, yes it's all a bit confusing and no, I'm not using Visual Studio but Spyder, which isn't' great but mostly works

The script below is very much tailor made to my current needs, so there are few quirks:

1. It assumes all files will be full day files at 15 seconds intervals.
2. It assumes that all files will have the same counters in the same order. It should be relatively trivial to change the script to take a config file with the counters needed.
3. It calculates a total CPU usage of all processes except SQL Server by subtracting SQL server usage from total cpu usage.

There are several command line options that should, hopefully, be self explanatory

Without further ado here's the script:

Needs a bit more work - 
#way to limit the number of days to be plotted, 
#explore different plots perhaps for different counters
#Actually get the files from the servers

from os import listdir, mkdir
from os.path import isfile, join, exists
import pandas as pd
import matplotlib.pyplot as plt
import re
import sys

def RenameColumns (current, fileName):
    current = current[:-1]
    current.append(fileName)
    return current
    
def Process (allFiles, numberToPlot, category, currentFile):
    ##Add the date column. This will not work if different files start at different times and have different intervals
    df = pd.DataFrame(allFiles[0][1][allFiles[0][1].columns[0]])
    #Rename the column to date
    df.columns = ['date']
    #We only need the time, not the date.
    df['date'] = df.date.apply(lambda x:x.split(' ')[1])
    #Get the last file number
    lastFile = currentFile + numberToPlot
    if lastFile > len(allFiles):
        lastFile = len(allFiles)        
    #Process each file into a dataframe
    for fi in range(currentFile, lastFile):
        fileName = allFiles[fi][0]
        df = df.join(allFiles[fi][1][allFiles[fi][1].columns[counter+1]])
        df.columns = RenameColumns(list(df.columns), fileName)
        #This is not ideal but to provide adjacent values would have to loop through i guess.
  #I'm sure there are better ways of dealing with missing values than this
        df[fileName] = df[fileName].apply(lambda x: 0.0 if x ==' ' else x ).astype(float)
    #Return a tuple with name and Dataframe.
    return (allFiles[currentFile][0] +"-" + allFiles[lastFile-1][0] + "-" + category, df)       


def GetCPUUsageRest (allFiles, numberToPlot,numberOfCPUs,  currentFile):
     #Is this really needed...Test you say? ...    
    numberOfCPUs = float(numberOfCPUs)
     ##Add the date column
    df = pd.DataFrame(allFiles[0][1][allFiles[0][1].columns[0]])
    #Rename the column to date
    df.columns = ['date']
    #Get the time
    df['date'] = df.date.apply(lambda x:x.split(' ')[1])
    #Get the last file
    lastFile = currentFile + numberToPlot
    if lastFile > len(allFiles):
        lastFile = len(allFiles)        
    #Process each file into a dataframe
    for fi in range(currentFile, lastFile):
        fileName = allFiles[fi][0]
        #Hardcoding FTW because 
        total= pd.DataFrame(DataFrames[0][1])
        sql = pd.DataFrame(DataFrames[25][1])
        sql[fileName] = sql[fileName].apply (lambda x:x/numberOfCPUs)
        df = df.join(pd.DataFrame(total[fileName] - sql[fileName]))
        df.columns = RenameColumns(list(df.columns), fileName)
        #This is not ideal but to provide adjacent values would have to loop through i guess
        df[fileName] = df[fileName].apply(lambda x: 0.0 if x < 0 else x ).astype(float)
    #Return a tuple with file name and Dataframe
    return (allFiles[currentFile][0] +"-" + allFiles[lastFile-1][0] + "-" + "Rest-Process % CPU" , df)      
    
#These are the counters used. The files need to contain them all otherwise the plots will be all wrong
#I suppose that we could load a list of these from a config file to accomodate changes rather than this
#hard coding.    
counters=[]
counters.append('Processor(_Total)-% Processor Time')
counters.append('Memory-Available KBytes')
counters.append('.NET CLR Exceptions(_Global_)-# of Exceps Thrown  per  sec')
counters.append('.NET CLR Memory(_Global_)-% Time in GC')
counters.append('ASP.NET-Application Restarts')
counters.append('ASP.NET-Request Wait Time')
counters.append('ASP.NET-Requests Queued')
counters.append('ASP.NET-Request Execution Time')
counters.append('ASP.NET Applications(__Total__)-Requests per Sec')
counters.append('Memory-Pages per sec')
counters.append('PhysicalDisk(_Total)-% Disk Read Time')
counters.append('System-Processor Queue Length')
counters.append('Web Service(_Total)-Get Requests per sec')
counters.append('Web Service(_Total)-Post Requests per sec')
counters.append('Web Service(_Total)-Current Connections')
counters.append('SQLServer-Access Methods-Page Splits per sec')
counters.append('SQLServer-Buffer Manager-Buffer cache hit ratio')
counters.append('SQLServer-Buffer Manager-Page life expectancy')
counters.append('SQLServer-Buffer Manager-Checkpoint pages per sec')
counters.append('SQLServer-General Statistics-User Connections')
counters.append('SQLServer-General Statistics-Processes blocked')
counters.append('SQLServer-SQL Statistics-Batch Requests per sec')
counters.append('SQLServer-SQL Statistics-SQL Compilations per sec')
counters.append('SQLServer-SQL Statistics-SQL Re-Compilations per sec')
counters.append('Network Interface-Bytes Total per sec')
counters.append('SQLServer-Process % CPU')

#This is regex for dates in the format yyyyMMdd. It doesn't quite validate as it would happily allow a day of 39
regexDate = '(20\d{2}[0-2]\d{1}[0-3]\d{1})'
#Set plotting style
plt.style.use('ggplot')

cmdargs = str(sys.argv)

if len (sys.argv) != 6 :
    print "This script takes five arguments"
    print "First Argument should be path where csv files to be processed are"
    print "Second Argument should be path where plots will be stored"
    print "Third Argument should be server name"
    print "Forth Argument should be number of Days to Plot"
    print "Fifth Argument should be number of CPUs"
    sys.exit(0)

path = sys.argv[1].rstrip('\\"')
dest = sys.argv[2]
server = sys.argv[3]
numberOfDaysToPlot = int(sys.argv[4])
numberOfCPUs = int(sys.argv[5])

    
if not exists(dest):
    mkdir(dest)

DataFrames = []
allFiles=[]
#Read the files for the input path. 
files = [f for f in listdir(path) if isfile(join(path,f)) and f.endswith('csv')]
for f in files:    
    name = re.search(regexDate, f).group()
    allFiles.append((name, pd.read_csv(join(path,f))))

remainingFiles = len(files)
#It looks like perfmon adds an extra column at the end of the file, which isn't a counter, hence the - 2
for counter in range(len(allFiles[0][1].columns)-2):
    currentFile = 0 
    while remainingFiles > 0:
        DataFrames.append(Process(allFiles, numberOfDaysToPlot, counters[counter], currentFile))
        #if we've reached the last counter, then we can look at the rest
        if counter == len(allFiles[0][1].columns)-3:
            DataFrames.append(GetCPUUsageRest(allFiles, numberOfDaysToPlot, numberOfCPUs, currentFile))
        remainingFiles = remainingFiles - numberOfDaysToPlot
        currentFile = currentFile + numberOfDaysToPlot
    remainingFiles = len(files)

CPUylim = ()     
#All plotting is done here.
for d in DataFrames:
    p=d[1].plot(x='date', title=d[0])
    #Get Y Axis Limit for Total CPU
    if d[0].find('Processor(_Total)-% Processor Time') != -1:
        CPUylim = p.get_ylim()        
    if d[0].find('SQLServer-Process % CPU') != -1:
        p.set_ylim( CPUylim)
    if d[0].find('Rest-Process % CPU') != -1:
        p.set_ylim(CPUylim)   
    #These two set the x ticks and labels make sure that they roughly match up.
    p.set_xticks([750.0,1500.0,2250.0,3000.0,3750.0,4500.0, 5250.0])
    p.set_xticklabels(['03:07:30','06:15:00','09:22:30','12:30:00','15:37:30','18:45:00','21:52:31'])
    fig = p.get_figure()
    fig.set_size_inches(21,13)
    fig.savefig(join (dest, d[0] + ".png"))
    plt.close()

A sample plot generated

Can't afford a certificate

Today somebody at work told me to log in to our gotomypc account and I manually typed the address: https://www.gotomypc.co.uk/

Clearly the UK is not an important enough market to merit a proper cert or a SAN.

This is the certificate for the .com website:

MSDN Subscription - A Very British Rip-Off

So a few days back I was considering whether we should take the plunge and fork out for an MSDN subscription for all our developers, there is five of us and given where we want to be it made sense.

This was a cost that had been budgeted for so, in theory, this should not be a problem, but as with all large purchases, and for a small firm, 5 MSDN subscriptions is a Large purchase, which meant that we had to get approval from above so I went to this site to confirm prices.

5 x $1199 even at silly exchange rates, should be less than £5K, so I thought, this would be an easy sell as it was less than we had budgeted, boy was I wrong.

If you are in the UK, you can see for yourself here, which shows you this:

What????? 

The exchange rate used by Microsoft is $1 = £1.01, rather than $1 = £0.68, so close to 50% surcharge. 

To put it another way, if we converted back to dollars, the price would be ~ $1780 rather than $1199.

I really like to hear an explanation from Microsoft, it certainly can't be the taxes the have to pay in this country, as the sales will probably be billed to the Irish subsidiary.

For the record this is the exchange rate today, according to Google:

and for the really lazy: 

I know there are resellers out there that do a cheaper price, it still does not change the fact that Microsoft itself applies an almost 50% surcharge to UK developers.

Migrating from Mercurial Kiln to Visual Studio Online (Git)

So we migrated all of our Mercury repositories, hosted in Kiln, to Visual Studio Online last week and this is the procedure we followed.

We only had 3 separate repositories and they are relatively small, so we deemed it unnecessary to try a more scalable approach. 

All the command line commands are run from a Developer Command Prompt for VS 2015.

1. Clone Mercury Repository to be migrated to a new directory.

hg clone <repo url>

2. Create new Harmony Repository

a. Navigate to https://<yourcompany>.kilnhg.com/Code/Repositories
b. Click on New Repository

It's very important that you select a Harmony Repository, otherwise this process will not work. Forking or Branching a Mercury only repository will not work either, as it simple creates a new Mercury repository.

3. Push to new Harmony Repository

 a. Edit  the .hg/hgrc file and change the path the new repository url
[paths]
default = https://<>.kilnhg.com/Code/Repositories/Group/Test_Migration
b. hg push  

Once this is has finished the Test_Migration repository is a Git & Mercury repository of the initial Mercury only repo, so it's relatively simple from this point onward.

4. Clone Harmony Repository as a Git Repository

git clone <https://<>.kilnhg.com/Code/Repositories/Group/Test_Migration.git>

5. Create new Repository in VSO

a. Navigate to Project
b. Click on Code

Once created you can click on Explorer which will take you to this page

6. Push to VSO

git remote remove origin
git remote add origin https://<yourcompany>.visualstudio.com/DefaultCollection/<project>/_git/<repo>
git push -u origin --all

Cloning Reddit to learn ASP.NET 5 - Part 6

In the last post in the series, we used a DTO to minimize the transfered data but we retrieved all that data from the database, shouldn't we have got just what we wanted?

In short the answer is yes, again the potential savings need to be carefully considered as the code might be more cluttered.

This is the query that gets run against the database:

SELECT [x].[SubRedditId], [x].[IsBanned], [x].[IsDefault], [x].[IsPublic], [x].[Name]
FROM [SubReddit] AS [x]
WHERE ([x].[IsDefault] = 1 AND [x].[IsPublic] = 1) AND [x].[IsBanned] = 0

You might be tempted to modify the Repository method like this:

public IEnumerable<SubRedditDTO> GetDefaultSubReddits()
{
            return ctx.SubReddits
                .Where(x => x.IsDefault && x.IsPublic && !x.IsBanned)
                .Cast<SubRedditDTO>();       
}

If you do this, you'll get this error: System.InvalidOperationException:

No coercion operator is defined between types 'Reddit.Models.SubReddit' and 'Reddit.Models.SubRedditDTO'.

We could use AutoMapper here but it seems a bit clunky when you're trying to reduce object size as the definition needs to be a bit like this:

          Mapper.Initialize(config =>
            {
                config.CreateMap<SubReddit, SubRedditDTO>()
                .ForSourceMember(x => x.IsBanned, o => o.Ignore());               
            });

In all honesty I've not read enough about AutoMapping, DTOs and ViewModels to be able to advise on best practices, but in this case it does seem a bit clunky, so instead I tried the following:

       public IEnumerable<SubRedditDTO> GetDefaultSubReddits()
        {
            return ctx.SubReddits
                .Where(x => x.IsDefault && x.IsPublic && !x.IsBanned)
                .Select(x => new SubRedditDTO() { Name=x.Name });
        }

This actually achieves what we wanted, at least in theory, which is reduce the load on the database. I realize that DB performance is a lot more nuanced than I'm presenting it here, but as a rule of thumb the less data retrieved from the database the better. This is the query that is executed against the database:

SELECT [x].[Name]
FROM [SubReddit] AS [x]
WHERE ([x].[IsDefault] = 1 AND [x].[IsPublic] = 1) AND [x].[IsBanned] = 0

This should be better from a DB performance point of view, but how much, it's hard to say.

The last thing I'll tackle in this post is the lack of links on the front page. You might be tempted to simply change the Index.cshtml page to this:

@{
    ViewData["Title"] = "Reddit";
}
@section scripts {
    <script src="~/lib/angular/angular.js"></script>
    <script src="~/js/redditapp.js"></script>
}
<div ng-app="redditApp">
    <div ng-controller="SubRedditListCtrl">
        <ul class="list-inline">
            <li ng-repeat="subReddit in subReddits">
                <a href="http://reddit.com/r/{{subReddit.Name}}">{{subReddit.Name}}</a>                      
            </li>
        </ul>
    </div>
    <hr />
</div>

Unfortunately, that will not always work as expected, all links will not have the subreddit until angular has substituted the value of subReddit.Name, instead we need to use a angular directive:

<a ng-href="http://reddit.com/r/{{subReddit.Name}}">{{subReddit.Name}}</a>  

Cloning Reddit to learn ASP.NET 5 - Part 5

In the last post we finally started to do something to the front end, namely the SubReddit bar on top, but the values were hard coded and they weren't even links.

Let's try to remedy that.

There are a few things we need to do:

• How do we identify Default SubReddits?
• How do we get and display said SubReddits?

The first one is easy, we add a new property to our SubReddit class, which now looks like this:

using System.Collections.Generic;

namespace Reddit.Models
{
    public class SubReddit
    {
        public int SubRedditId { get; set; }
        public string Name { get; set; }
        public bool IsPublic { get; set; }
        public List<Post> Posts { get; set; }
        public List<RedditUser> Users { get; set; }
        public bool IsBanned { get; set; }
        public bool IsDefault { get; set; }
    }
}

There are other ways of doing this, say, have a default SubReddit table an add the default SubReddits there, but we'll do it this way until I change my mind.

We need to run now through the Migrations rigmarole again, so from ..\src\Reddit run the following commands:

dnx ef Migrations add isDefaultSubReddit
dnx ef database update

I've manually seeded the SubReddit table with a few records, both default and not default (I will discuss how to seed it from code in a later post) and now we can edit redditApp.js to this:

(function () {

  "use strict";

  var redditApp = angular.module('redditApp', []);

redditApp.controller('SubRedditListCtrl', ['$scope', '$http', function($scope, $http) {
  $http.get("/api/SubReddit").then(function(data){
  $scope.subReddits =data.data ;
})}
])
  
})();

In order to get around issues with minification, we inject the dependencies as strings which will not be minified.

There are several issues with this approach as we're getting more data than we need in two senses:

• We're getting all SubReddits, we could filter them but that would mean retrieving all the data from the server and then filtering them, which is as inefficient as it sounds. 
• Furthermore, we're retrieving all properties of the SubReddit objects when we only need the name.

Let's deal with the first of these issues. We add a new method called GetDefaultSubReddits to the Repository, make sure you add it to the Interface as well:

        public IEnumerable<SubReddit> GetDefaultReddits()
        {
            return ctx.SubReddits.Where(x => x.IsDefault && x.IsPublic && !x.IsBanned);
        }

And we also add a call to this method from the Controller:

.        [HttpGet("[action]")]
        public IEnumerable<SubReddit> GetDefault()
        {
            return repository.GetDefaultSubReddits();
        }

Note that I'm using an action here, which is effectively the name of the method
Now we modify the redditApp.js

(function () {

  "use strict";

  var redditApp = angular.module('redditApp', []);

redditApp.controller('SubRedditListCtrl', ['$scope', '$http', function($scope, $http) {
  $http.get("/api/SubReddit/GetDefault").then(function(data){
  $scope.subReddits =data.data ;
})}
])
  
})();

At this point we've solved issue number one but we're still sending quite a bit of unnecessary data up and down the wire so let's build a DTO that contains just the data that we need, which in this case is simply the Name. There are other ways of achieving this, for instance we could use an OData query to simply retrieve the necessary data.

This is the SubRedditDTO

namespace Reddit.Models
{
    public class SubRedditDTO
    {
        public string Name { get; set; }
    }
}

Modified GetDefault Method, I'm not sure whether it's a better idea to transform to DTO object here or on the Repository. I guess it depends on usage, in other words, if the Repository method will be used somewhere else it makes sense to do the transformation here.

        [HttpGet("[action]")]
        public IEnumerable<SubRedditDTO> GetDefault()
        {
            return repository.GetDefaultSubReddits().Select(x => new SubRedditDTO() { Name = x.Name });
        }

This is the reponse from the server now

[{"Name":"Gadgets"},{"Name":"Sports"},{"Name":"Gaming"},{"Name":"Pics"},{"Name":"WorldNews"},{"Name":"Videos"},{"Name":"AskReddit"},{"Name":"Aww"}]

This is before, when we where not using a DTO.

[{"SubRedditId":3,"Name":"Gadgets","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":4,"Name":"Sports","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":5,"Name":"Gaming","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":6,"Name":"Pics","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":7,"Name":"WorldNews","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":8,"Name":"Videos","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":9,"Name":"AskReddit","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true},{"SubRedditId":10,"Name":"Aww","IsPublic":true,"Posts":null,"Users":null,"IsBanned":false,"IsDefault":true}]

This might seem like a paltry gain but little savings like this can quickly add up.

Cloning Reddit to learn ASP.NET 5 - Part 4

In the previous post we ended up again with a broken, although partially this time, application let's try to fix it.

The problem was that it could not resolve the service type for IRedditRepository as we were not telling how to inject the dependency properly, which is done on the ConfigureServices method on Startup.cs:

public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddMvc();
            services.AddEntityFramework()
                .AddSqlServer()
                .AddDbContext<RedditContext>();

            services.AddScoped<IRedditRepository, RedditRepository>();
        }

AddScoped will use the same instance for each request, could have used AddSingleton or AddInstance.

We try running this again and we see this:

So it's working but there is nothing there, we'll have a look at this later on.

Now let's try to look at the homepage, which at the moment looks like this.

This doesn't look like Reddit, so let's do a bit of editing.

Delete the Contact.cshtml and About.cshtml files, in the Home folder

Edit Index.cshtml so that it looks like this:

@{
    ViewData["Title"] = "Reddit";
}

This is the _Layout.cshtml file, which can be thought of as a master page or template page. I've removed pretty much everything from it that might be rendered.

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>@ViewData["Title"] - Reddit</title>

    <environment names="Development">
        <link rel="stylesheet" href="~/lib/bootstrap/dist/css/bootstrap.css" />
        <link rel="stylesheet" href="~/css/site.css" />
    </environment>
    <environment names="Staging,Production">
        <link rel="stylesheet" href="https://ajax.aspnetcdn.com/ajax/bootstrap/3.3.5/css/bootstrap.min.css"
              asp-fallback-href="~/lib/bootstrap/dist/css/bootstrap.min.css"
              asp-fallback-test-class="sr-only" asp-fallback-test-property="position" asp-fallback-test-value="absolute" />
        <link rel="stylesheet" href="~/css/site.min.css" asp-append-version="true" />
    </environment>
</head>
<body>
    <div>
        @RenderBody()       
    </div>

    <environment names="Development">
        <script src="~/lib/jquery/dist/jquery.js"></script>
        <script src="~/lib/bootstrap/dist/js/bootstrap.js"></script>
        <script src="~/js/site.js" asp-append-version="true"></script>
    </environment>
    <environment names="Staging,Production">
        <script src="https://ajax.aspnetcdn.com/ajax/jquery/jquery-2.1.4.min.js"
                asp-fallback-src="~/lib/jquery/dist/jquery.min.js"
                asp-fallback-test="window.jQuery">
        </script>
        <script src="https://ajax.aspnetcdn.com/ajax/bootstrap/3.3.5/bootstrap.min.js"
                asp-fallback-src="~/lib/bootstrap/dist/js/bootstrap.min.js"
                asp-fallback-test="window.jQuery && window.jQuery.fn && window.jQuery.fn.modal">
        </script>
        <script src="~/js/site.min.js" asp-append-version="true"></script>
    </environment>

    @RenderSection("scripts", required: false)
</body>
</html>

The end result of all this should be an empty page.

I've left RenderBody and RenderSection method calls intact as they will come in handy later. In essence, they are used to merge in the content from the actual pages and to load the scripts needed respectively.

Since this is a modern app I'm going to use AngularJS, which is probably past it's prime now but there you go.

The simplest thing to do this is by using the manage bower packages:

This will display this page:

This is the result on the bower.json file, which should be noted is not shown by visual studio but it's there on the src\Reddit folder:

{
  "name": "ASP.NET",
  "private": true,
  "dependencies": {
    "bootstrap": "3.3.5",
    "jquery": "2.1.4",
    "jquery-validation": "1.14.0",
    "jquery-validation-unobtrusive": "3.2.4",
    "angular": "1.4.6"
  }
}

So let's add Angular to Reddit

I've added a new file called redditApp.js to wwwroot/js, which contains the following:

(function () {

  "use strict";

  var redditApp = angular.module('redditApp', []);

redditApp.controller('SubRedditListCtrl', function($scope) {
  $scope.subReddits = [
    {'name': 'Gadgets'},    
    {'name': 'Sports'},
    {'name': 'Gaming'},
    {'name': 'Pics'},
  ];
});
  
})();

I've hard coded a few subreddits to have something to display but we will soon get them from the database. In essence, the above is a self executing function containing an extremely simple angular controller that has an array of objects.

I have modified the Index.cshtml like this:

@{
    ViewData["Title"] = "Reddit";
}
@section scripts {

    <script src="~/lib/angular/angular.js"></script>
    <script src="~/js/redditapp.js"></script>
}
<div ng-app="redditApp">
    <div ng-controller="SubRedditListCtrl">
        <ul class="list-inline">
            <li ng-repeat="subReddit in subReddits">
                <span>{{subReddit.name}}</span>
            </li>
        </ul>
    </div>
</div>

Note the angular directives? ng-app, ng-controller and ng-repeat and also note the angular bingind {{subReddit.name}}

This is what Reddit looks like now:

Cloning Reddit to learn ASP.NET 5 - Part 3

In the last post we applied a database migration but nothing occurred.

The reason for this was that the RedditContext class was not tracking any of the objects nor where the relationships specified on the OnModelCreating.

So I've modified the RedditContext class

using Microsoft.Data.Entity;

namespace Reddit.Models
{
    public class RedditContext : DbContext
    {
        public RedditContext()
        {
            //Database.EnsureCreated();
        }

        public DbSet<RedditUser> RedditUsers { get; set; }
        public DbSet<UserDetails> UserDetails { get; set; }
        public DbSet<SubReddit> SubReddits { get; set; }
        public DbSet<Post> Posts { get; set; }
        public DbSet<Comment> Comments { get; set; }
        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            var connection = Startup.Configuration["Data:RedditContextConnection"];
            optionsBuilder.UseSqlServer(connection);
            base.OnConfiguring(optionsBuilder);
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<RedditUser>()
                .HasKey(x => x.RedditUserId);

            modelBuilder.Entity<UserDetails>()
                .HasKey(x => x.UserId);

            modelBuilder.Entity<RedditUser>()
                .HasOne(x => x.UserDetails)
                .WithOne(x => x.User);

            base.OnModelCreating(modelBuilder);
        }
    }
}

In order to recover from this mistake, we want to apply the new migration, but the Core migration contains nothing so I will first unapply it

dnx ef database update Start
dnx ef migrations remove
dnx ef migrations add Core
dnx ef database update

This time it worked .... hurray

At this point, it's worth considering how will this be deployed to a test environment and the answer is pretty simple. A migration script can be created with the following command:

dnx ef migrations script

This creates a Sql script that can be used to create the database.

It's time to have another look at our repository

Let's start with the IRedditRepository, where we now have a few methods and this time we'll try to implement some of them.

using System.Collections.Generic;

namespace Reddit.Models
{
    public interface IRedditRepository
    {
        UserDetails GetUserDetails();
        IEnumerable<SubReddit> GetSubscribedSubReddits(RedditUser user);
        IEnumerable<SubReddit> GetSubReddits();
        IEnumerable<Post> GetSubRedditPosts(SubReddit subreddit);
        IEnumerable<Comment> GetPostComments(Post post);
    }
}

using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Linq;

namespace Reddit.Models
{
    public class RedditRepository : IRedditRepository
    {
        private readonly RedditContext ctx;
        private readonly ILogger<RedditRepository> logger;

        public RedditRepository(RedditContext ctx, ILogger<RedditRepository> logger)
        {
            this.ctx = ctx;
            this.logger = logger;
        }

        public IEnumerable<Comment> GetPostComments(Post post)
        {
            return ctx.Posts
               .Where(x => x.PostId == post.PostId)
               .Single()
               .Comments;
        }

        public IEnumerable<Post> GetSubRedditPosts(SubReddit subreddit)
        {
            return ctx.SubReddits
                .Where(x => x.SubRedditId == subreddit.SubRedditId)
                .Single()
                .Posts;
        }

        public IEnumerable<SubReddit> GetSubReddits()
        {
            return ctx.SubReddits.AsEnumerable();
        }

        public IEnumerable<SubReddit> GetSubscribedSubReddits(RedditUser user)
        {
            throw new NotImplementedException();
        }

        public UserDetails GetUserDetails()
        {
            throw new NotImplementedException();
        }
    }
}

Straight away we see that there are going to be problems as we lack paging capabilities, but we'll add those later

In any case, we'll need to expose these method somehow, there are potentially two options here depending on what we're trying to accomplish: 

We could use these methods from inside a Controller or We could create an API and we're definitely going for the latter as it's way cooler

Create a new folder called Api inside the Controllers folder and add a new  Web API controller class, called SubRedditController

using System.Collections.Generic;
using Microsoft.AspNet.Mvc;
using Reddit.Models;

// For more information on enabling Web API for empty projects, visit http://go.microsoft.com/fwlink/?LinkID=397860

namespace Reddit.Controllers.Api
{
    [Route("api/[controller]")]
    public class SubRedditController : Controller
    {
        private readonly IRedditRepository repository;

        public SubRedditController(IRedditRepository repository)
        {
            this.repository = repository;
        }

        // GET: api/values
        [HttpGet]
        public IEnumerable<SubReddit> Get()
        {
            return repository.GetSubReddits();
        }

    }
}

We can now build and debug the project. This allow us to navigate to http://localhost:<portnumber>/api/SubReddit

Unfortunately this does not work:

We'll look at the solution to this problem in the next post.