The kinds of tests we write

When starting work on version v3 of our software PriceOn, we decided to take unit testing seriously. This was basically our CTO's idea, and I am very happy that we managed to adopt it as a default coding practice in virtually no time. The reason it went so smoothly for us is just that all three of us backend developers were highly motivated to master it, and we had some previous experiences trying it and failing when not taking things seriously.

What we deliver is a service API reachable via HTTP. This API is consumed by our frontend clients: website and mobile apps for iOS and Android. We code in Microsoft's C# - some would say non typical choice for a startup, but we decided to stick with the thing we know best, as there are far more important challenges than picking 'cool' language. So let me just explain some points related to how/why we perform testing.

Simple integration tests that ping service via HTTP

These primarily test that the  wiring is correct and service does not crash on typical request. Just a simple service ping, which checks that server returns 200 OK, no business logic testing here.

[Theory, NinjectData]
public void FollowCardApidPing(ICanSave<Card> repo, FollowCardReq req, Card card)
{
    req.UserId = MockedUserId;
    repo.Save(card.CardId, card);
    AssertPostTakesAndReturns<FollowCardReq, FollowCardRes>(UrlRegistry.FollowCard.Mk(new { userId = MockedUserId, cardId = card.CardId }), req);
} 

Some details:

•  we use custom NinjectData attribute which resolves test parameters by first looking at Ninject kernel specifically configured for tests, and if that fails, creating something with AutoFixture.
• AssertPostTakesAndReturns is a method of base class integration tests derive from. This class Launches in-process instance of ServiceStack which hosts our services, so that we can interact with them via http, and that is what this Assert method does.
• Currently, this is integration test just in a sense that it launches http server and tests everything via http interface. All problematic dependencies within service are replaced with in-memory implementations. We may consider changing them to real ones sometime in the future when reliable performance of 3rd party software starts to weigh in.

High-level unit tests that fake only external dependencies

These are the majority of tests we write. We test service logic by trying to mock out as little dependencies as possible. Similar to Vertical Slice Testing by @serialseb. The unit tests get System Under Test (SUT) in fully wired up state, with only external dependencies such as Database, Timer, Filesystem faked. As in production, unit tests resolve dependencies from IOC container which is just slightly tweaked from production configuration to inject lightweight implementations for external services.
The most prevalent external dependency is a database. How do we deal with it? We have IRepository interface, and then we have InMemoryRepository : IRepository. For each test, we seed this InMemoryRepository with relevant data, and inject into SUT.

[Theory, NinjectData(With.NullUser)]
public void QueryingShops_SingleSellerExists_ReturnsThatSeller(ProductAdminService sut, ShopsReq req)
{
    const string seller = "Mercadona";
    var repo = SetupData(new[] {Product.CreateGoodProduct().WithSeller(seller)}, p => p.Id);
    sut.ProductRepo = repo;   
   
    var res = sut.Get(req);
   
    res.Items.Select(u=> u.Seller).Should().BeEquivalentTo(new[]{seller});
}

• As with integration tests, we resolve our dependencies from Ninject and AutoFixture, with just external dependencies faked. In this case, sut is taken from Ninject, and req is some randomly generated request made by AutoFixture. We may have to tune req according to test case, but in this case it is empty object, so nothing to be done there.
• Our InMemoryRepository with data for the test is injected into the SUT. This is more stable than faking response from repository directly.
• Repository is injected into sut via property setter. As we are resolving sut from IOC container, we already have default repository implementation, but we have to swap it with the one containing our predefined data.

"Real" unit tests in specific places with nontrivial business logic

We write these just in the places we feel logic is not trivial and may tend to change. This usually happens when we have to evolve API method to support more interesting scenarios, and it practically boils down to extracting domain specific code into isolated easily testable units. For easiest testability, it is very nice to isolate complex logic from all dependencies.

[Fact]
public void RegularPrice_IsPriceThatRepeatsMostDays()
{
    var date = new DateTime(2000, 1, 1);

    var sut = new Offer();
    sut.Prices.Add(date, new PriceMark(){Price = 1, ValidFrom = date, ValidTo = date.AddDays(10)});
    sut.Prices.Add(date.AddDays(11), new PriceMark() { Price = 2, ValidFrom = date.AddDays(11), ValidTo = date.AddDays(13) });
    sut.Prices.Add(date.AddDays(14), new PriceMark() { Price = 2, ValidFrom = date.AddDays(14), ValidTo = date.AddDays(16) });

sut.RegularPrice.ShouldBeEquivalentTo(1);
}

• This is fairly obvious code, our SUT does not require any dependencies, just plain business logic. Breeze to test.

User input validation

A very well understood concept in computer programming is that of 'validation'. In its basic form it's just a predicate which determines whether data conforms to some predetermined rules. Combined with a principle that problems should be identified as early as possible, this leads to the most common application: having a method call, we first validate input arguments before proceeding with any business logic (checking preconditions). If validation doesn't pass, we return some hints as to what went wrong. Sometimes the information we provide event suffices to painlessly fix the issue.

Now, validation is also very widespread in HCI, where computer validates user input. As with all thing human however, this gets a little bit more tricky. You see, as a user, I'm not interested in bureaucracy in the slightest. It annoys me when I'm told "your application form is invalid". I expect software to guide me as I try to express my intentions. And while validation errors are a form of guidance, they tend to occur too late and tend to suggest user failure. You'd usually want someone who is willing to provide as much help as possible instead of accepting/rejecting your application and referencing some article no. in "rules".

Anyway, looks like this exploration-based model will dominate the transaction-based model in HCI. And the  simple "form validation" is falling apart here. At least, it should be way more granular, and ideally there should be no "validation errors". You just want your computer usage to be a pleasant experience, without any error popups. It seems often to be much better to just ignore incorrect input, rather than scream loudly "I don't understand what you mean, fool". Quite the opposite to what we do with validation in our code for API.

CQRS and ReadModel

While CQRS seems to be very useful tool for dealing with complex domains, I kinda got burned by it more times than I would like to admit. I'm talking about traditional interpretation, CQRS+ES+DDD ala Greg Young. The biggest frustration I'm facing is how it claims that readmodel can be somewhat an afterthought, yet how it still remains the crucial part, and unless you take a good care of it, you gonna have a bad time.

CQRS lets you defer decisions, not avoid them. You may think that choosing the reporting database is exactly the kind of decision CQRS lets you avoid. Well, not entirely. It allows you to offload the decision to someone else who is responsible for reporting. But still it's gonna be you for small one-man projects. And you'd better damn know well your database, as there are tons of nasty surprises awaiting with these beasts.

It is completely unsuitable for CRUD scenarios. If there is no complex processing of data coming in, using CQRS is certainly overkill. If your commands map 1:1 to your events map 1:1 to your read model, it is just common sense then this is a nonsense. Whoever says there is no overhead in doing this and you'd better just do it everywhere has no clue or is lying.

Naive implementations of read model only take you so far. There are some people around boasting of how CQRS+ES allowed them to get rid of the database. So this is really neat point with ES, that your primary data model (Events) is very simple. You can put serialized events anywhere you want, no need to tie  yourself to any particular DB. On the reporting side however, this is entirely different story. Unless you are writing most primitive and thus probably useless app, just storing your (demormalized) data in files is not gonna work. Where is fulltext search? Where are those adhoc queries business always bugs you about? Or even, where is the list of users sorted by date of registration? Pretty tricky to implement when all you have is partitioning by id, eh?

So, while I believe this is the way we will be building software in the future, the fact remains, CQRS just lets you ask the right questions at the right time, but never be fooled that this makes the answers somehow less important.

Defensive programming, YU so popular?

This term can have a lot of meanings. Everyone likes to attribute all the good thing to  a term they like while saying the bad things are misinterpretations or don't belong to that term altogether (Agile, anyone?). Usually when I hear people mentioning defensive coding, this is along the following lines, and this is what I take it for:

Defensive coding is doing everything possible to not make a fool of yourself.

Looks like people believe this is good thing, but to me it looks potentially very devious. This is like curing symptoms, not a problem. Covering your ass may be good for you in the short term, but definitely not for your team.

While defensive coding may contain often useful practices such as validating data coming from the external system, other more questionable practices tend to be associated to it ("check everything for null", "log all random things you can think of" ). To me, these are just Good practices, and Bad practices (in some context ofc.) - and defensive programming seems to contain practices from both of those sets.

Now there is another term, Offensive programming, which basically says, when you fail, scream as loud as you can, so that everyone else knows. This looks much more appealing to me. It looks like programming model Erlang advocates, where if worker fails, it does not attempt any tricks to recover, it just exits. It is responsibility of that worker's supervisor to decide what should be done in face of this failure.

There is but one place where defensive coding makes sense. And that place is most important of all in software. It is Production. Here, crashing the system is the worst thing that could possibly happen. It is often much worse than corrupting data, hiding the fact that something was not accomplished, etc. This is lying to your user. But if you lie rarely enough so that user doesn't notice it, this is better than telling this software is full of bugs by crashing and thus interrupting workflow.

So, defensive coding is bad for you (in the long term). It is bad for your team. It seems to be good for your business. Yet, I feel it is a sorry state of software development, where writing crap, and then defending against the crap you and your colleagues wrote is regarded as proper way of coding. Offensive coding raises the bar higher, and I want to believe it will be a way of doing things sometime in the future.

Sand castle in software

Consider what it means to write software following Open-Closed Principle (OCP).  You never modify existing code. If you need new functionality, you write a new code. Or maybe delete unused software units, to keep things clean.

In traditional statically-typed OO language, you would create class once, and never modify it again. What is the most important thing of that class? A name. You'd have to think really hard how to name this class, or else it will be the source of daily WTFs. I guess, that would be extremely hard.

The obvious benefit is that you never break anything that depends on your code. This enables you to write new code without fear that you may break something - which is very serious problem in software. This is my sand castle, a code that is as flexible as language allows it.

Now look at typical software. Each line of code is modified 10s if not 100s of times. Often changes have cascading effects. Code is full of unexpected behavior where names do not reflect the actual behavior, or even intent.

Why is it so hard to write code following OCP? I think, it is because we do not know what we are doing. Software we are supposed to create is beyond our comprehension. And so, we try creating random things, until something works as we thought it should. We may call this TDD, where we code to examples, and not full specification. On the other hand, there can be no specification other than code itself. The set of what -> how mappings is so big and with so many interesting options, there is no way we can choose The One, The Right option.

Decision making

I often hear this, that decision making capabilities have high influence on one's success. While this is true in  sense... I like to distinguish between 2 types of thinking behind decision-maker's mask.

Bull

The Bull has a strong personality. He sees the red flag, and he will hit it, no matter at what cost. He makes the decision, and then copes with the consequences. He does not think of the consequences upfront. He is strong enough to stand for his decisions, and commit to them. Once he has committed to a decision, he will not recognize he was wrong, no matter what it takes.

If faced with a hard problem, he will make a random decision. He will not admit he has no idea what the consequences will be. If there is a tree in his way, he doesn't give it a second thought before battering into it.

When asking a Bull to solve your problem, you will see his continuous efforts towards the goal. You will see total commitment and restless determination. Ultimately, despite all the complications, he will successfully deliver what you had asked. But you may be wondering, if the problem is indeed as hard as he was trying it to look like.

Butterfly

The Butterfly is a free thinker. He loves exploring the world. Decisions he makes come to him naturally, so as to best fit the problem into their world view. He likes to often change his opinion, and so he also hates to commit to his decisions. If he realizes he was wrong, he has no problem to admit it - he doesn't take this personally, he is just glad he now has better understanding of the world.

If faced with a hard problem, he will bounce off it like he had hit a tree. He will be helpless. The only way for him to proceed is to take time, investigate what the problem looks like, and only then, once he if familiar enough with the problem, he is comfortable with making a decision.

When asking a Butterfly to solve you his problem, you may not see progress for long periods of time. His reluctance to commit to anything may lead up to you considering to cancel the contract. But then suddenly he will do this all in a very short time. You may be wondering what he could be capable of accomplishing if he wasn't such a lazybones.

Coin

Our coin of 2 sides, each bad, is this:

1. Bull will commit to a bad decision and will not abandon it
2. Butterfly will not commit to any decision, and will frequently change his mind

Estimate: Guess and Commitment

Guess

So this is a typical situation. Manager comes to you, and says "Good day today, we've just got this new cool feature request... blabla ... This is extremely important feature, our bottom line greatly depends on how quickly we can deliver it. How much will it take you?".

 Being a nice day indeed, with a fresh head in the morning, excited about the great news, you feel noone can stop you. So you say "two weeks" (Thinking you could pull this off in a couple of all-nighters, but of course I will be clever and take reserve to cope with all those risks which of course will not affect Me, the Superhero. Just for the ease of mind.).

 "Great, so two weeks it is", says the manager rubbing his hands. "Please do your best. This is very important. It would be great if you could do this in one week. Now, the details...". And so, now you take a look at all the details. Needless to say, they are not what you expected at all.

 By this time, you start realizing how stupid you were. You barely know the requirements yet, having spent only a few moments to consider them. And still, you have managed to fully commit to do this, in a week (well, 2 weeks actually, but you know the expectation is 1 week, 2 being the worst case). You know you should have asked for time to consider and clarify requirements. But at the moment, you thought you fully understand what business means with this feature, and there is no need to clarify anything. Turns out you were wrong. They have very different view on what this feature means. And this view is at odds with your current architecture. And noone wants to hear about any compromises you are offering.

 And so, bit by bit, your motivation starts dropping. You start remembering how awfully you are underpaid and how your manager had no moral right to ask you to do this oh-so-important feature virtually for free, in the first place. To the level that you take full two months to complete this goddamn feature. It is buggy as hell. But of course it's not your fault. It's the fault of your manager, who was so stupid not to understand how you had to compromise your system's architecture to fit this abomination in. "How could anyone be so short-sighted", you wonder.

Commitment

 The next time you are much more careful. You are asked to estimate this apparently trivial feature. You ask for time to consider requirements. You clarify everything you can. Yes, this is indeed as easy as it looks, you conclude. Having spent 2 hrs estimating, you finally say "this will take me 4hrs, no more". Manager wonders "What took you so long to estimate this? I thought this was a no-brainer you should be able to do in a few hours".

 And indeed, by the 2nd hour, you are already done. Of course, the manager is not happy again. He accuses you of how was that extremely wasteful of you to spend 2 hrs estimating a feature, when you could have actually gotten it done by that time.

You are extremely frustrated by his assertion. Hadn't you put effort in understanding the problem, you'd probably had slipped on one of those edge cases. How can anyone be so blind?

Estimate

The two examples above are edge cases of what estimate could mean. The first is your rough guess. The second is your full commitment. They are very different things, and yet, when one is asked to give estimate, these two are often used interchangeably. One time, you just pull a number off the top of your head. Next time, you are very confident you have all cases covered; you know every little step that has to be made.

It is no wonder managers tend to look at estimates as commitments, while developers would like to take them as guesses. In ideal case, estimate should be a sweet spot somewhere in between. It is a probability distribution. It is your task as a developer to work with manager and find this sweet spot, so that you didn't spend too much time considering all the unlikely cases, and still have a reasonable prediction business can rely on.

As a developer, I am very guilty of doing this vile thing, being very inconsistent with my estimates. One time they are pure guesses based on my mood, the other time they are inevitable fact, based on deep knowledge of the problem. Managers hate that. They like to look at the numbers you give them in spreadsheet, and be confident those numbers are correct. The worst thing that could happen is when your "1" is sometime "2". Oh no, wait, it's more like it's sometimes "20".

In reality, you shouldn't give all your estimates with the same level of confidence. That would be wasteful. You need to find the sweet spot for each problem. And be very explicit where it lies. If you had seen this problem before and you know how to solve it, you should say so. Everyone will be happy. On the other hand, if you have no idea, you should tell your manager this could take years. He will not be happy and ask you to be more specific. You then take a few hours researching the problem, find a blog post describing solution to similar problem, and now you are confident this will take 1-2 weeks with prob. 90%. Manager is happy with such estimate, he asks you to stop researching and start coding.

Epilogue


Estimates are necessary when you want to have predictable results. In that case, we should ask how much predictable these results should be, and base our estimates accordingly. Estimate is probability distribution. It will probably be normal-distribution shaped for non-trivial problems. That means, there will be non-zero probability the task could take 100 years to complete.

If you think that's a nonsense, think about NP-completeness. Let's say the problem you need to solve is in NP. And you need efficient solution, for a general case. It is not known at the time whether the problem is in P. But who knows, maybe in 100 years someone will discover it is indeed in P. So what you do? You go to business and say "Here is and approximation, I am confident it will fit our case". If they say "No, no, we need it exactly as we specified. Continue searching. And please hurry, this thing is taking you too long" - I'm sorry for you mate. As someone said, "change company, or change company".

As you minimize risk, you minimize failure. But so you minimize success. The catch is: the more experience you have, the more you are minimizing failure rather than success when minimizing risk. Why? Because minimizing risk means making choices. The more experience you have, the better choices you can make. It doesn't mean experience will lead you to better choices always. But it is a vector, your North Star. It's like playing chess if you know nothing about openings vs someone who knows his openings. It is very unlikely you will avoid all opening traps and pull it off just because of your "lucky star", "genius mind", or anything of that sort.

As we see in software world today, we are strongly lacking in experience to minimize failure efficiently, while at the same time not to limit our success chances severely. Many successful software companies just made a random thing that worked. There are as many companies who made a random thing that failed. While we are at this level, there is no need for estimates, they just waste time. Build your MVP in a day, if people like it, start iterating based on feedback. and here you have it, a random product that works.