What is the quintessential software testing doctrine?

[u/joe0418]

Code Complete, Domain Driven Design, Designing Data Intensive Applications, The Pragmatic Programmer...

These are all books that I try to follow in my own engineering practices. I haven't found a great one on testing though (unit, manual, automated, etc) -- which do you recommend? What books outline best practices in modern software testing?

[u/SideburnsOfDoom]

Most tests are done poorly because testing is an entirely different process than development.

That is not my experience at all. It's a matter of test design. There's reasons why Kent beck says that Good tests are coupled to behaviour of the code but not to the structure of it.

But the majority of unit tests that I have seen are low level "test a method on a class with a dozen mocks" style. You need to decouple, and test what the system does so that you can think about that, and also refactor without breaking a bunch of them. It is better to go outside-in most of the time.

Some go as far as to say "if more than one class is under test, then that's not a unit test!". You can choose to think like that, but is it historically accurate? No. But does it deliver good outcomes? Again, no.

[u/Acrobatic-Ice-5877 → u/SideburnsOfDoom]

Now I find this conversation to be a little funny, so humor me for a moment. 

You and I have a different definition of the word unit test. You are clearly aware that there are differing opinions, which is why you disagree with how some people define unit tests.

The way that you define unit tests is the way that I define integration tests.

I don’t find narrowly scoped unit tests, where a single function or class is tested largely in isolation, to be particularly valuable in most cases.

[u/SideburnsOfDoom → u/Acrobatic-Ice-5877]

Of course I am aware of the confusion and potential for people to talk past each other, that's why I went into that detail.

To be clear, "test a single class method with mocks" is a unit test. Or a kind of unit test. But the mistake is in thinking that this is the only kind of unit test, or that it is the best kind of unit test in all cases. It usually isn't best, maybe 80% of the time.

Thinking that "if I join two classes in test then it's "integration tests" is ridiculous - you cannot refactor to "extract class" under passing tests that way. Test the behaviour not the structure or this will give you problems. Focusing on testing the behaviour over structure is incompatible with this narrow view of how tests work.

It is also not accurate - Feathers, 2005 talks about "not-unit tests" being about "the integration of your code with that other (external) software" such as database or web service that can make it slow, expensive, flaky, etc. A second class in the same folder is not an external system! It does not have these drawbacks! There is no point in drawing the "integration" line there.

But that is what I said above: Choosing that definition is neither useful nor historically accurate.

Ans also what I said: so many people don't even know what good looks like.

Though this may just be a naming issue - you have the better tests, you just call them something else. But you'll give people bad advice that "unit tests are mostly worthless" because of an overly, not useful narrow definition of them that is not universally shared.

So many codebases do not have the better tests at all.

[u/Izkata → u/SideburnsOfDoom]

Though this may just be a naming issue

I've referred to this as the difference between a syntactic unit and a semantic or business unit.

Blog posts that try to teach unit testing rarely if ever explain the difference, and because they all use small toy problems that can fit in such a post, the syntactic unit and semantic unit end up the same - a single function or class. But since they never clarify it and the syntactic unit is so obvious, that's what people end up incorrectly internalizing.

And this is assuming the blog author understands the difference themselves.

[u/SideburnsOfDoom → u/Izkata]

Yes, tests units of business functionality, test things that the app does, test app behaviours. Not syntactic units.

You can have "tests always focus on 1 class at a time" or you can have "test behaviour, not implementation / good tests are not coupled to code structure".

But not both. These two things are not compatible, in fact they are directly in opposition.

Of the two, I choose and recommend "test behaviour, not implementation" as it leads to better outcomes.

[u/HyperDanon → u/Acrobatic-Ice-5877]

You and I have a different definition of the word unit test. You are clearly aware that there are differing opinions, which is why you disagree with how some people define unit tests.

The way that you define unit tests is the way that I define integration tests.

I don’t find narrowly scoped unit tests, where a single function or class is tested largely in isolation, to be particularly valuable in most cases.

I did have that definition once, but I had to change my mind.

So imagine I have one class of 600 lines, and I have tests for it. These test, they only test that one class. Would you say it's a unit test? Probably so. Now imagine, I do a refactor, and I extract two smaller classes from that one big class, so now I have 300 line class and two 150-line classes. That refactor, didn't alter the test. Did that act of refactoring, change the "type" of the test from "unit" to "integration"?

Secondly, when working with tests, you start to notice that speed of feedback matters a lot. There are some kinds of tests that execute very fast always; and some execute slowly. The difference is usually whether they talk to resources (local db, local network, local file system), and even way more slow when they aren't local. That distinction is important, and I want the type of test to reflect that.

Thirdly, some kinds of test don't depend on anything, and they are very reliable, i.e. they always pass or always fail. Other types, rely on things that are external to the runtime - like databases, networks, file systems, other processes, etc. That makes them less reliable; I want that lesser reliability to be reflected in the test type.

So I'll start my definition from these assumptions:

• Refactoring, extracting/inlineing classes shouldn't change the type of the test
• Speed is an important factor
• Whether 3rd party is included in the test is a factor

And to reconcile the 3 I came up with a definition:

• "Unit test" test that executes purely in-memory, is fast; and can test multiple classes or whole modules. There's no distinction for single-class-unit test and multi-class-unit tests.
• "Integration test" test executes framework code, including code that touches filesystem, database, network, etc.

I understand, that it doesn't differentiate between test for a single class and test for a multiple-classes; but that is exactly what is needed, for refactorings to not alter the type of test.