codeflood logo

Damn you VSUnit!

I ran into quite an unusual quirk with VSUnit today. (I've taken to calling Visual Studio Unit Testing VSUnit for brevity.) I've been doing some Sitecore dev using Test Driven Development (TDD) which I have had great success with in the past when working in a purely NUnit space. But with VSUnit I like the ability to run my tests straight inside the IDE (please don't point me at :) ). And also the integrated code coverage (please don't point me at NCover). Yeah, I know there are things I can use with NUnit to give me these capabilities, but the VSUnit things requires no additional setup. It's there out of the box.

So enough about the good points of VSUnit...onto the source of this rant. So, there are 2 buttons in the toolbar which allow me to either run the tests in the current context, or all tests in the project.

VSUnit toolbar

The tests in the current context depend where your cursor is. If your cursor is in the body of a test method, then only that single test will be run. If it's in a test class file outside a method, then all the tests for that class will be run. But when I have several tests and test classes, I don't want to execute each test class separately, I want to make use of that "run all tests in project" button.

So when I click to execute all tests in the project, what order did VS run them in? Almost random apparently. Hmm. I was expecting it to behave like NUnit, where if I select to run all tests they will run in some kind of predictable order. At least all the tests for a test class are executed one after another in NUnit. VSUnit runs them completely out of order. It might execute Class1.Test1 then Class1.Test2 then Class2.Test1 then back to Class1.Test3...Completely out of order. OK, I can see the point that unit tests should be isolated and atomic and should be able to be run in any order. And I agree. Test 3 should not rely on Test 1 being executed before it. But what about when I'm doing class initialisation and cleanup? I'll get back to that in a second as it ties into my next rant.

So what happens if I need to perform class initialisation and cleanup. This is the perfect place to create the Sitecore items which are required by your tests, and then destroy them to return the environment to a known good state when our tests are complete. As anyone who has done this before, you'll know that item creation in Sitecore is not takes time. So we don't really want to be creating and destroying items for each test. Now in NUnit, because everything runs in an order as I expect, I know that the class initialise will execute, then the classes tests, then the cleanup, then onto the next test class. Not so with VSUnit. It will call the classes initialise, then some of the tests, then it will branch off to another test. And if that test requires initialisation it will call that, then the test, etc. Note, we haven't called any cleanup yet. VSUnit waits until all tests have completed before calling any of the cleanup methods. So in the case above where I create Sitecore items during initialisation and clean them up in cleanup, my items don't get cleaned up before the next test class runs, so when it initialises it's initialising into a dirty environment.

In the end I had to move my Class Init and Cleanup to Test Init and Cleanup. Looks like I might have to reassess the use of those other NUnit extensions for VS...


If I'm not allowed to point at, can I point at ReSharper that I very much prefer over anything else? :-)

Alistair Deneys

Of course you can point out ReSharper Alexey :) . Thanks. Like I said at the end of my post, I think it's time I reassessed the NUnit VS integration tools again. Perhaps I'll start with that one. If it's good enough for Alexey, it's good enough for me!

According to what we've seen here we hypothesize that VSUnit runs your tests in more than one thread, so you can't know in what order these tests will run. This does force you to keep your tests atomized which is a good thing - no telling when you'll need to take a test and put it in another fixture, and if it's not independent you'll have extra work to set up its preconditions.
I do have several gripes about VSUnit (running deployed tests by accident, the ExpectedException attribute...) but its overall a solid framework.

Alistair Deneys

Thanks for that Doron. I wasn't aware of the possible multithreading of VSUnit tests. Makes sense so your tests are run faster. My issues were really brought to light due to the nature of my tests. They are not unit tests but integration as they are testing code which I have written to run as part of the third party system (Sitecore). I require a known content structure in Sitecore before I run my set of tests and want to cleanup the environment when I'm done. Actually I'm going to be doing a post soon on how I do this, so stay tuned.

Leave a comment

All fields are required.