MOOC done right - Embedded Systems - Shape the World

A few years back I signed up to edX and at time there weren't that many courses available so after the initial excitement of being able to do MIT courses died down, my account was dormant for quite a while, more on this later, until last year I decided to try to learn Python and rather than using LTCTHW or Code Academy, I thought I would try a Python course instead.

A few weeks after I got an email alerting me to other courses that might be of interest and in this email was one that sounded really interesting:

Embedded Systems - Shape the World - UTAustinX -  UT.6.02x

I signed up immediately.

Although the course can be completed without an embedded system, it is, of course, recommended that one is used. Buying instructions for the recommend kit (TI Tiva LaunchPad and some bits and bobs) are provided and not just for the USA. This I found a really nice detail, as I can imagine that probably the majority of people taking the course were not in the USA and it's really one of the many examples of the kind of involvement and enthusiasm that the staff exudes.

My main problem with MOOCs so far has been a combination of lack of motivation, I find it hard to follow through on a topic that, while interesting, might not be applicable for work, current or future, or even daily life and this is not to say that I only learn stuff that's useful, I don't, my head is filled with objectively useless facts, such as the Band Gap of GaAs being ~ 1.42eV (I did not look it up) or Juan Manuel Fangio having won almost half of all the F1 races he participated in or one of my favourite words, biblioclasm. 

The other reason, and this is probably, the main reason, is lack of time. A lot of the courses suggest a 10-15 hour weekly commitment, this might not sound like much, and in fairness it isn't, most of the time, but some times it is and this is where the Embedded Systems - Shape the World course gets it completely, and absolutely right. The first seven lessons were available when the course started, and the most of rest of content was made available shortly afterwards, so that effectively two weeks after the course started 85+% of the course materials and labs were available.

This is completely at odds with the way that most courses release their material, which is done on an almost weekly basis, with homework(s) due almost on a weekly basis. I find this terribly disappointing. What is the point of doing a course online when you have to basically do it at a pace set for you?  I appreciate that I'm doing it from the comfort of my home but even so this is very likely a major contributory factor to the really poor completion rates in MOOCs. Although I'm not doing the courses for the grades, it's always motivating to be able to get a good grade and the course runs on a very tight schedule a busy week at work or trip can prevent keeping up with the course schedule.

I don't have a CS degree and I've had a interest in low-level programming for a while now, but I've never really taken the time to explore this in any detail as I've always found it pretty daunting, but in the course concepts ranging from CPU instructions and registers to  interrupts and device drivers are explained in a simple and accessible manner. 

In fact, it is explained in such a manner that it's made me lose some of the awe for the guys and gals doing the low-level stuff. I realize that this is silly, as the drivers that are part of the course are extremely simply but it feels as if a veil has been lifted and beneath it, the truth has been revealed.

Instructions on how to get started, install IDE, drivers and TeXas software are provided and I found them easy enough to follow. Those of you out there without a Windows machine might grumble at the lack of direct support, there are instructions on how to the install from virtualization software from a Mac. I guess the authors assume that if you're using Linux you don't need any help getting a hypervisor running :)

All labs have a skeleton project, for Keil's µvision IDE and a simulation board, which allows the code to be tested before deploying it to the physical board. I often found that working code in the simulation board would fail when deployed to the physical board. This annoyed me a bit at first, but in reality this is no different from the good old:

Well, it works on my machine.

Generally speaking the code was not robust enough and it needed tweaking. I imagine that there are probably more sophisticated simulators available, but the cost is likely to be prohibitive. This is not unlike apps for the myriad Android phones out there.

One thing that was quite surprising at first, although it makes sense since we're are so close to the bare metal, is the way everything is referred to by memory address. For instance, if an application needs to use bits 0 and 1 of Port E, this required knowing the exact address of these bits. Thankfully, these were provided on the skeleton projects but they can also be looked up on the spec sheets. This is, incidentally, an invaluable skill due to large catalogue of systems and components out there.

This is a very simple program, that flashes an LED that's connected to bit 1 of Port E, based on whether a switch connected to bit 0 of PORT E is pressed, and I think it illustrates the point above. Also note, how the delay function counts, effectively, the number of cycles.

#define GPIO_PORTE_DATA_R       (*((volatile unsigned long *)0x400243FC))

int main(void){ 
 while(1){    
 Delay1ms(100);
 if( GPIO_PORTE_DATA_R&0x01){
  //Let's flip it (Bit 1)
  GPIO_PORTE_DATA_R^=0x02;
 }
 else{
  //Lusisti satis,edisti satis,atque bibisti,tempus abire tibi est
  GPIO_PORTE_DATA_R |=0x02;   
 }
  }

void Delay1ms(unsigned long msec){
        long cyclesPerMs = 15933;
 long counter = cyclesPerMs * msec;
 while (counter > 0){
  counter--;
 }
}
}

I've removed the initialization routine, which in essence, gets the physical board running and activates the various ports as needed. I found quite a few of my issues on the physical board where down to issues in the initialization routine, so it's by no means trivial.

The gradual increase in complexity of the physical circuits that needed to be built was very finely tuned. Chapter 8, finally required a circuit to be build, not just using the TI Tiva LaunchPad and it was really nerve racking, I don't really know why, as I think it was only £20 worth of kit, but there were diagrams available and enough warnings regarding what to do and more importantly what not to do, that I built the circuit without any issues. This, actually, ended up becoming the most enjoyable activity of the course, the actual building of the circuits. 

One of the hardest labs, was chapter 10, where a Finite State Machine is used to model a simplified traffic light system for an intersection. I actually really enjoyed this lab even if it took quite a bit of pen and paper to get the design right in the first place. Also, one of my better pictures :). The switches (yellow parts in the middle of the breadboard) model a sensor that detects cars or pedestrians.

Chapter 10 - Modeled  Intersection's Traffic Light System

Chapter 10 - Real System. 

This is not a great picture, but it shows the TI Tiva LaunchPad interfacing with the Nokia 5110 screen. In Chapter 14, an ADC was used for to measure distances. This works by moving the slide potentiometer, measuring the changes in voltage, which can then be converted to a distance, after suitable calibration.

Chapter 14 - Measuring Gauge

In the penultimate chapter of the course, all that was learned during the course is put together on a sort of final project: a games console. This involved designing the game and then putting together the circuit to build it. Although it might sound daunting as usual there was a lot of help in the form of skeleton code. The hardware part, was relatively simple in that it consisted in putting all that had been learned previously to use. An interesting exercise, you can see the results, not mine, I was too late, here.

The last chapter of the course, involved the Internet of Things, which I have to confess, I haven't done yet, as I've procrastinated on getting the WiFi booster pack for the Launchpad and this brings me to another issue with most other courses: The graders. 

In other courses, I've done these became inactive when the course becomes inactive and, to be fair, it's the same with this course, but there is a massive difference, the graders in this course work by checking a hash (The hash is computed by the grader software that is run in the local machine) and thus, it is entirely possible to check that your programs work as intended regardless of the course status, this is very welcomed novelty for me and I don't know why this is not the case for more courses.

I should point out that the last chapter does require access to a server, which to be fair could've been mocked to allow offline access. The server is still up 2+ weeks after the course ended.

This post has gone on for far longer than I originally intended and I haven't even talked about Electronics, which is a pretty important part of the course, but I  will stop here.

I would like to end the post by thanking the staff on the course and particularly Dr Valvano and Dr Yerraballi, for making this course very accessible, really enjoyable and tremendously educational.

I really hope that a more advanced course is made available by Dr Valvano and Dr Yerraballi, at some point in the near future.

This post doesn't do the course justice, go on, just go and take it, you will enjoy it.

Cashless society N=1

A couple of weeks ago I read an article regarding a new law in Denmark that will effectively make cash not legal tender anymore. In other words, business will not be obliged to accept cash as payment.

To me this seems fairly sensible, I really hate paying by cash, in fact I occasionally find myself struggling to remember the pin number of my debit card as I use it so rarely, so I thought I would try to analyze my cash use. The simplest way of doing this is by looking at cash withdrawals from my bank account.

I decided to have a look at what data I could use from my banks online site and was a little bit disappointed to find that they only provide the last 12 months, I could request older data but it would be printed so I decided to stick to 12 months.

This is the raw data.

As I suspected, there seems to be a decrease in the frequency of cash withdrawals, but the linear fit is pretty poor due to the various outliers.

I did a little bit of thinking and I realized that the March and September outliers are due to leaving dos from people at work and the June one was due to buying some stuff for my girlfriend at Vintage Fair. 

I decided to plot the data again but this time without the outliers.

The trend line, just a linear fit, shows a much better fit than the raw data as is to be expected.

I can easily see this trend holding true, i.e. I will continue to visit the ATM less often, due to the increasing acceptance of contactless payments and its limit being raised to £30 in September.

Gas and Electricity Consumption in a 1920s mid-terrace house in the North of England.

Last week I was going through some old pen drives to see if there was actually anything worth keeping and I found a lot of old energy consumption measurements I took back at our old house, so I thought I would share them here.

The house was a small mid terrace house, with central heating and a gas cooker, built after the First World War. I started taking the measurements after I decided that leaving my gaming PC on 24/7 wasn't a great idea, I should've taken a few measurements with it on, but there you go. We only heated the house to a relatively low temperature, i.e. ~ 18° C

Unfortunately, I don't have measurements of outside temperature so I cannot correlate energy use to outside temperature, but the data was gathered to try to get a better understanding of how much gas and electricity we were using at the time.

Without further ado here are the charts:

It's hard to see electricity consumption in the above chart, so here it is:

Estimate costs below. I will not rant about the rather ludicrous way Gas and Electricity is priced in this country.

Electricity on its own again:

Housework

Over the past three and half months my lovely girlfriend has been suffering from terrible eczema, so she's basically not been helping much with the household chores, which is why I've been looking at how long I've spend doing housework everyday.

I have been measured how long I spent doing work and what sort of work it was. The measurements almost certainly an underestimate as I've probably missed little tasks here and there, e.g. cleaning up a spill or any such ad hoc task.

I've broken down the tasks in three categories: Cooking, Cleaning and Miscellaneous.

The first two are self explanatory, although it's worth mentioning that when it comes to cooking I have only counted the preparation time, except for when the cooking time was so short that it would not really allow me to do anything else, in practice this only has significant a bearing in Sundays figures as we normally have pancakes, french toast or eggs Benedict for breakfast.

Miscellaneous is anything that is not cooking or cleaning, e.g. tidying up, putting the washing away, etc..

We live in a, small?, two bedroom apartment (about 60 m²) and have no children or pets, don't eat out much (twice a month since the data keeping started) and mostly cook from scratch, so without further ado, these are the figures:

Total:

As a percentage:

We can all be above average (almost)

I was reading a very interesting article on behavioural psychology today and its effect with relation to Peak oil, climate change etc. I really liked the article, it does a pretty good job of explaining skepticism towards change, it is a must read for certain. I could not help but notice a common enough error with statistics.
To illustrate a point about human optimism particularly with regards to our own abilities, the author quotes research showing how people think that they are better drivers than average (88%) or that professors think that they are better than average (94%), which is obviously impossible as "By definition, almost half of those surveyed are 'overly optimistic'" or in other words 38% of drivers surveyed and 94% of professors must be wrong. Interestingly, it is possible for the surveyed people to be right, although very unlikely and here is why.

The arithmetic mean or average of a distribution is simply the sum of all values of the distribution divided by the number of items in the distribution. Say that we quantify driver ability on a scale of 0 to 1000, where 1000 is the driving skills of Michael Schummacher coupled with the law abiding ethos of a 50 year old German protestant pastor and 0 is the law abiding ethos of an Italian teenager coupled with the driving skills of a 100 year old arthritic, half blind grandmother.

We take a random selection of 10 people and measure their driving skills. These are their results:

1. 501
2. 500
3. 504
4. 502
5. 503
6. 508
7. 516
8. 490
9. 491
10. 532

The average of the above distribution is 504.7 and thus we have 80% above average. what is going on here, I hear you ask?

Nothing much, just good old fashioned confusion, here is another distribution, this one is the annual salaries of ten friends who have met for a dinner party.

1. £50100
2. £50000
3. £50400
4. £50200
5. £50300
6. £50800
7. £51600
8. £49000
9. £49100
10. £532000

On this distribution 90% are below average, the average is close to £95k.

The author confused arithmetic mean with the median, which is described in Wikipedia as: a median is described as the numeric value separating the higher half of a sample, a population, or a probability distribution, from the lower half.

As I have shown it is possible to have almost everybody above or below average it is impossible to have more than 50% of the drivers above the median, by definition.

This post was just a little bit of facetiousness, his point still stands even if he conveyed it poorly. People think that they are above average and above the median.

Incidentally, the median for the first distribution is: 503 and for the second: £50300.

Developers Vs Developers

I have been meaning to write this post for ages as this is something that I have encountered time and time again during my career.

We have an integration layer on our application with our telephony system. A third party wrote this integration layer, in essence they have a web services that we communicate to and we expose a web service so that they can communicate with us, so far so simple, this is a staple of enterprise development and as I said I have had to deal with situations like this many times, both inside and outside the company, and if you've not done it inside your company and think that none of the problems that you are having would occur if you didn't have to deal with the idiots at Third Part Ltd., let me tell you that you will just have to deal with the idiots at the Other Department.

At any rate, integration was working fine when somebody realized that the spec called for SSL/TLS usage rather than using clear text. In theory this requirement made sense when the various sides of the integration equation where hosted by two different companies, but a change of architecture had been made, which meant that they weren't any longer, so using a secure website for an internal web interface that contained little more than phone numbers and UUIDs seemed like overkill, but the spec is the spec, agile be dammed.

So both Web Services and client apps on either side of the integration equation where reconfigured to use SSL/TLS and this is were the problems started and as it's normally the case in these situations, we started blaming each other. 

Furthermore, a supplier customer relationship had been allowed to develop. This is a relationship in which the supplier has the technical know how, or believes he does, and the customer doesn't, or is at least believed not to have it by the supplier. Needless to say that this wasn't the case as we shall see, but for various personnel reasons, i.e. developers on our company leaving, this relationship had taken hold, which meant that our evidence carried less weight as it were, because they were already predisposed to assuming that they were talking to yet another developer who didn't have a clue about how the integration was supposed to be working, which wasn't true but it was true that they knew their system and the history, and while I knew ours but could not comment on historical decisions as this had landed on my lap without handover from somebody that left the company suddenly, not so suddenly but I digress.

We both went back to basis to prove our respective thesis, i.e. it was the other party's fault, because history is written by the victors, you know how this will turn out, but I will continue anyway. The first thing that we discovered was another disagreement in the spec, regarding authentication, which I remedied after a lot of hair pulling.

After that, I found that our side was working fine, furthermore, I used Wireshark to prove that nothing was coming through over the wire on port 443, while stuff was going through on port 80 from the client PC that hosted the client app, which meant that failures on their side were not due to our Web Service, this despite the fact that the app was throwing a 404 error and pointed me to this link.

I mentioned the supplier customer relationship above, because it helps to explain why this evidence, our Wireshark evidence was ignored, they knew what they were doing we didn't so anything coming from our side was tainted.

To further compound the confusion, the client app would sometimes work when tested by them, which made us extremely suspicious and not work for us, which surely made them more suspicious of our general competence. It was working for them and they were dealing with a bunch of idiots so they probably were doing something stupid.

At this point they were willing to discuss the code that they were using for the first time and we realized that the source of the issue was their code, to be fair it was the usage of our proxy server, when it should not have been used, there were other issues but they are not important.

So I asked them to add this to the system.net element of the client's app.config:

<defaultProxy  enabled="false"/>

Lo and behold, everything started to work fine.

Not sure what the lesson is here, I guess if there is one, it's that appearances do matter even in a supposedly analytical field like this one.

This link seems tangentially related.

On Quality

A few months back I hit upon an idea for a rather laborious scheme that would make me not that much money: Selling hard drive magnets on Ebay.

There were approximately 50 or so old hard drives at work, and when I say old, I do mean old. None of them could accommodate more than 18.6 GB of storage with an Ultra2 Wide SCSI interface.

Nobody wanted to take the time to dispose of them properly, which included the unenvious tasks of filling all the paper work or indeed running the disks through the data erasure program. I figured if the disks were not working nobody had to worry about this.

I set about taking apart the first disk and I hit my first road block: Torx headed screws. I had never had the need to use a Torx screwdriver so I decided to go out and buy a set of Torx screwdrivers.

I did a bit of research online and I wasn't surprised to find such a large disparity in prices: from as low as a few pounds to around one hundred pounds. I was sure that I didn't need to spend £100 on a set of Torx screwdrivers, but how low should I go?

As is my wont, I procrastinated and resolved to do more research on the topic. However, that weekend I stumbled upon a set of Torx screwdrivers at a discount store for £2.99, so I thought that I might as well buy them there and then.

I was fully aware that at this price the quality of the set would leave a lot to be desired but still I managed to suppress this knowledge long enough for me to dismantle 1.5 hard drives, which is when I hit the quality limit of my £2.99 set of Torx screwdrivers.

As I said above, I was not expecting them to last a lifetime and they were a spur of the moment, almost an impulse buy, so it wasn't unexpected, however this left me with a bit of a problem.

I know that if one buys the cheapest, in this day an age with a proliferation of  peddlers of cheap junk, one gets poor quality, but is the converse true?  In other words, does one get quality by buying expensive stuff? Perhaps more importantly, how much should I have spent on the set given the used I was planning to give it?

It is very easy to determine the quality of items at the bottom of the price scale: They are rubbish, but one does know what one is paying for. However, once we leave the safety of the cheaper items, then it becomes a lot harder to ascertain how much better something is or put another way Do you know what you are paying for when you buy an expensive item?

Take the Apple iPod shuffle, which can be obtained for £35 from Amazon. Storage capacity is 2 GB, it has no screen, it's tiny and has some sort of clip mechanism. For a similar price, £36, it is possible to buy a Sansa Clip+ with 8GB of storage, an expansion slot, screen, FM radio and also a clip mechanism. Yes, it's slightly bigger but hardly noticeable and voice commands can be added with RockBox firmware, so are you sacrificing 6 GB of storage for voice command?

The reality is that to a great extent you are paying for the Apple brand, with its design and quasi-religious following, which means that if you don't really care about design and don't think much of Apple as brand then you would be wasting money by going down the iPod shuffle route.

Is there are a similar quasi-religious following for say Stanley tools?, I would rather imagine that this unlikely to be the case. In fact, from talking to some of my relatives, who work or have worked in construction, they seem to buy tools from different brands mostly through experience. In other words, they tend to favour a brand because it was worked for them in the past and negative experience have a much more lasting effect that positive ones:

I spent loads of money on a expensive diamond tipped drill bit set from Black & Decker and it was rubbish. Since then I've always gone with Bosch drill bit sets and power tools.

In truth it might have been the other way round, the point still stands though, a negative experience is a lot more likely to be remembered, than a positive one as the positive one, this case, simply means having a reliable tool every day for a long time.

Whenever I find myself thinking about quality, I always imagine myself going back to that Arcadia of the consumer on the days prior to consumerism, whenever they happen to have occurred. In reality I have to admit that there have always been different quality levels on the products available to the consumer and while the bewildering price ranges that can be found for most products these days, makes buying the right item really tricky and by the right item I mean an item whose quality is commensurate with the price paid, it is simply naive to think that it easier through choice.

It was only easier because there was no choice, in other words, if you were a worker you could just afford the cheapest stuff and that is what you bought. It's only a modern dilemma that we have this paradox of choice, which makes discerning how much of your money goes on quality and how much goes on to pay for the brand premium almost impossible.

To a certain extent this is ameliorated by the various product reviews, but product reviews are no panacea as it is just as likely that the service is reviewed, which can be helpful, but it's hardly relevant to the product's quality or lack thereof. Furthermore, a large number of reviews describe personal preference and are normally added very early on, i.e. when no issues are found or the product was found to be defective, so they tend to be very Manichean.

There are dedicated people who seem to take reviewing very seriously, a sort of amateur Which? (Consumer Reports if you are in the  US) but sadly they are, very much, the minority and if you're not contemplating buying something that they have already bought, then you are out of luck.

So what to do?

News is bad for you – and giving up reading it will make you happier

A few months back I read this article about how News is bad for you and I thought I would give the advise a try to see what would happen. I would give up the News, sort of. Would it result in any kind of improvement in my day to day? Would it make me more .. or less ..?

The first thing to point out is that I haven't completely cut myself out of the news cycle, my radio alarm is still set at 07:00 every morning, with the (electronic) dial set to Radio 4, so I do get some news, not many as I'm normally out of the door by the time the sports section start, so this is normally about 25 minutes or fewer.

Furthermore, I don't actually go out of my way to avoid the news, but since I don't own a television, this means that at most, I'm only going to catch the odd glimpse of the news from a TV left on somewhere or a newspaper laying about. In essence, what I have done is stopped browsing news websites, mostly the BBC and various other newspapers here and abroad. 

So, Gave I had an epiphany? Has my (mild) depression lifted? my productivity increased 2 fold? 3 fold? 10 fold? Has anything changed at all?

Perhaps, unsurprisingly, there is very little that has changed in my life. I think this is probably down to two main facts:

1. Rolf Dobelli is mostly right in his assertion that we don't really need the news.
2. I avoid local news like the plague.

I admit that I have not completely cut myself off from the News, so perhaps we do need the News, but he is correct that there is hardly a news item that, has affected my life in any meaningful way, and I have benefited from knowing it as early as possible or most of them at all.

It is worthwhile mentioning black swans events and opportunity cost here. The former, as it would seem that only such events would be worthwhile knowing about as soon as possible and opportunity cost because all the time, mainly, spent consuming the news in the vain hope of being ready for the black swan event, which might never come and even if it does, will it compensate for all the other things that could have been done with that time (money)?

On the aftermath of the Boston bombing, somebody wrote a blog post/article on the best way of having the most accurate information about the bombing or any such event. Their suggestion was to turn yourself off from the electronic world, go out with your mates to the park or something and then read all about it on the morning paper. 

Mr Dobelli would probably argue that even reading about on the morning paper would be a waste of time, which is probably true.

I think number 2 is the key to why I feel very little change in my life without a constant stream of news.

There really is no easy way of saying this, but Local news are simply evil. They tend to concentrate, overwhelmingly, on crime and because by their very own definition, they are local, it does not even allow us to dismiss them as something that would/could not happen here, as it has happened. Furthermore, since we are notoriously bad at probability, reading them is very likely to make us anxious even though nothing, or very little, has changed about the probability of being the victim of a crime. Yet, reading about it is likely to have made us more likely to believe that crime is worse than what it actually is and even, and I'm going on a limb here, before we read the story about the horrific crime.

The murder rate in the UK is 1.2 per 100000 inhabitants or 12 per million inhabitants. The local news in my area cover approximately 1000000 people (some local media cover smaller areas, of course), which means that on average there will be 12 murders a year, or put another way: 1 per month. Not enough to be a constant worry, but enough to be a constant reminder. Never mind the fact that most of the crime is essentially criminals killing each other. Yes, there are cases were there are random acts on innocent people, but just because you can easily recall an example does not mean it's common, in fact, it's quite the opposite. Media coverage tends to be inversely proportional to frequency of an event. This is one of the reasons why the attacks on the London bombings on July 7^th 2005, got the coverage they got. 

The one positive effect, that this voluntary withdrawal from news sites has had in my life, is related to my somewhat complicated relationship with sports, which I'm not going to go into detail here, but suffice to say that not knowing anything sport news has left me without those little moments of joy when the results went my way or those loooong periods of annoyance, frustration, irritation and helplessness when they didn't (I am of course exaggerating a little here for effect).

If you consume local news, I do recommend that you stop, for everybody else you can probably carry on as you were, but know that being au fait with the latest is unlikely to be of much use for anything unless your job depends on it, in which case what the hell are you doing reading this blog?

Developers vs Testers

Last week I had the usual argument between developers and testers. (Note, I did not actually write the code that was the source of the issue, but I am getting ahead of myself.)

Tester: Why did you close the ticket as working to spec?

Me: Occam's razor. 

Tester looks baffled at me

Me: Wikipedia

Tester looks even more baffled

Me: Look it up in Wikipedia

Tester left admitting defeat.

That's how it should've played out but alas it wasn't.

The issue raised was that the invoices related to an account did not have the correct number, for various reasons too involved to explain here, we have separate auto numbering counters for different invoice types, thus it's perfectly valid to have 30 invoices and the 31st be numbered 1 if it's the first invoice of this type, e.g. 30 for consultancy services and 1 for accountancy services.

Unfortunately, the auditing system in MS dynamics CRM 2011 is not perfect so deleting invoices, which is actually a custom entity, leaves a record that they were deleted, by whom and at what time, but crucially it does not say which invoice was deleted.

I responded to the tester, after checking the audit view, that he must've deleted the invoices and that explained the incorrect number and closed the ticket.

The tester was not happy, they thought it was a real issue that needed looking at and this is where things normally take a turn for the worse, because we, as developers, assume that the tester did something wrong. 

We don't just assume that the tester did something wrong, of course, we check our code (✔), we run a unit test (✔), we might actually prove to the tester that the functionality that the tester raised a defect about working properly (✔), so why is this insignificant little tester wasting my time rather than accepting that they did something wrong?

It's pretty simple really, they need an explanation that satisfies them. 

Nobody likes admitting that they are wrong and in cases like this, where the issue, at least to the tester, is intermittent, they would like the developer to prevent it from happening again.

At this points my options were:

1. Get into a ticket passing ping pong match (which almost certainly would've escalated to ..)
2. Escalate to PM and/or TM
3. Reproduce the bug.

So I did, and it turned out that my explanation was only half right. Yes, they had deleted some invoices but there were genuine holes in my explanation.

Is there anything that can be learned from this episode?

1. Don't be lazy.
2. Don't assume that the tester made an error
3. If you are sure that the tester made an error, then see #1 and do replicate the error that they made, who knows they might help you uncover some bugs in the system.