ExpectedException Failures in MSTest?

This morning I got an automated email alerting me of a failed build.

Excerpt from Jenkins

Results               Top Level Tests
-------               ---------------
Failed                SecurePasswordTest.EncryptDecryptFailure
Failed                SecurePasswordTest.EncryptDecryptFailureSecureString
Passed               SecurePasswordTest.EncryptDecryptSuccess
Passed               SecurePasswordTest.EncryptDecryptSuccessSecureString
2/4 test(s) Passed, 2 Failed

This is one of the failing tests:

[TestMethod]
[ExpectedException(typeof(System.Security.Cryptography.CryptographicException))]
public void EncryptDecryptFailureSecureString()
{
    var password = DateTime.Now.Ticks.ToString();
    var encryptedPasswordRandom = Convert.ToBase64String(Encoding.ASCII.GetBytes(password));
    var encrypted = SecurePassword.Encrypt(SecurePassword.ToSecureString(password));
    var decrypted = SecurePassword.ToInsecureString(SecurePassword.DecryptString(encryptedPasswordRandom));
}

I ran the tests from my machine and it worked fine. I tried from the server and it worked ....

At this point I remembered that a second build server had been brought online, as some tests runs were taking 2-3 hours, in any case. I checked the other server and I could reproduce the issue, in other words, the test failed there, but why?

Build Server #2 did not have Visual Studio 2013 Shell installed, after I installed it, it started working correctly.

It is rather strange that this happened in the first place and I'm not sure that this is actually related to the ExpectedException attribute, the type of exception or what exactly.

At any rate, hopefully it helps somebody out some day.

TIL - DeploymentItem files not copied in MSTest

I have a set of unit tests that need an xml file to work  and I despite me using the DeploymentItem attribute the test was not working.

Turns out that DeploymentItem will only copy from the build directory, so I set the Copy to Output Directory property of the file to Copy always: