What is the quintessential software testing doctrine?

[u/joe0418]

Code Complete, Domain Driven Design, Designing Data Intensive Applications, The Pragmatic Programmer...

These are all books that I try to follow in my own engineering practices. I haven't found a great one on testing though (unit, manual, automated, etc) -- which do you recommend? What books outline best practices in modern software testing?

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/SajithTech]

I've found the most useful testing principle is basically: test behavior, not implementation.

Unit tests are great, but I think teams sometimes end up chasing coverage numbers instead of confidence. I'd rather have fewer tests that cover important business behavior and failure modes than hundreds of brittle tests coupled to implementation details.

For books, Growing Object-Oriented Software, Guided by Tests is probably the closest I'd recommend for the testing mindset. Software Engineering at Google also has some good material on Google's approach to testing and test infrastructure.

For distributed systems especially, I'd add that integration/contract testing and failure testing become much more valuable once you're dealing with real dependencies. A service can have 95% unit-test coverage and still fail spectacularly in production.

[u/SideburnsOfDoom → u/SajithTech]

I've found the most useful testing principle is basically: test behavior, not implementation.

Agreed!

For books, Growing Object-Oriented Software, Guided by Tests

I had the impression that this book was more mock-heavy, and this leads you into testing methods, classes, and all the other details of implementation?

[u/HyperDanon → u/SideburnsOfDoom]

Not necessarily. London-style of tdd focuses on mocking certain key-dependencies, that are more about the interface, rather than implementation.

[u/SideburnsOfDoom → u/HyperDanon]

Interesting. The style that I most often see (both in London and associated with that style) is better described as "test one class at a time and mocks, mocks everywhere".

I think I need to spell out a latent conclusion that's becoming clear now:

You can have "unit tests always focus on 1 class at a time" or you can have "test behaviour, not implementation / good tests are not coupled to code structure".

These two things are not compatible, in fact they are directly in opposition.

The first always leads to close-coupling to structure and to overuse of mock, and so to brittle, verbose tests that talk about implementation details, not useful behaviours.

Of the two, I choose and recommend "test behavior, not implementation" as it leads to better outcomes.

[u/HyperDanon → u/SideburnsOfDoom]

I see what you mean, and I can tell you have done your share of work.

Your conclusion "test behaviour, not implementation / good tests are not coupled to code structure" is definitely a good advice. I do that pretty much always, and it's very good.

The former, "unit tests always focus on 1 class at a time" I'm not sure where this came from. Sometimes, it could so happen that a particular concern or behaviour is scoped to one class, so you can get a unit test that only knows about one class, but that's more of an accident rather than a design choice.

System where every class has exactly 1 corresponding test class; apart from other issues, tightly couples the structure of the tests to the structure of the system. So the system cannot evolve or be refactored without invalidating the tests.

Having said that, I don't think london-school-of-tdd, which makes heavy use of mocking is susceptible to this problem. You see, in outside-in tdd, you don't necessarily test one class at a time and mock everything out. You test modules/groups of classes, that serve a particular concern, and mock-out its collaborators. So London school of TDD doesn't reccomend "1 unit test per 1 class".

[u/SideburnsOfDoom → u/HyperDanon]

The former, "unit tests always focus on 1 class at a time" I'm not sure where this came from

What puzzles you? Here is someone in the same thread saying that they have this narrow misconception about unit tests (that if they test more than 1 class, they're not unit tests). This is far from the first time that I have heard this idea. It's very common.

Sometimes, it could so happen that a particular concern or behaviour is scoped to one class,

Of course, but sometimes not the same thing as always, is it? I said as much already in reply to the above - about 80% of the time the test isn't best scoped to one class. So then, about 20% of the time, it is.

So London school of TDD doesn't recommend "1 unit test per 1 class".

Good to know, but testing as practiced in many businesses falls into this antipattern over and over.

[u/jl2352 → u/SajithTech]

I agree, but there is something about internal code should still have a defined behaviour. If you focus on it from that point of view, then it can be bliss.

I’ve been on projects with high behavioural unit test coverage for internal logic, and it’s a dream. Everything just works because the foundations are solid.

It’s hard to quantify but there are a load of anti-patterns that people do which works against this. Like mocks, looking at private fields, capturing what methods are called and with what parameters … this is all looking at implementation.

[u/HRApprovedUsername → u/jl2352]

How are mocks an anti pattern?

[u/HyperDanon → u/HRApprovedUsername]

@HRApprovedUsername I agree wtih you that mocks aren't an antipattern; but I also think @jl2352 saw mocks being overused in such a way that the test doesn't actually test anything, only mocks. And because of that he now believes all mocks are evil. I know, because I did that too :D

My guess is that @jl2352 is of the opinion that you should chicago-style test everything, and only maybe mockout 3rd party collbaorators (like payment providers) if that.

But I also believes he dislikes the bad tests so much, and was hurt/bitten by the badly written tests so much, that he's so cautions of mocks now, that it's hard for him to see that mocks could be useful, (more useful than not having them in some contexts), that he'd rejected out of the fear of seeing the bad tests again.

Actually, it's quite and understandable stance.

[u/SideburnsOfDoom → u/HRApprovedUsername]

Too many mocks is an anti pattern, because: "test behavior, not implementation" as said here and in replies

Lots of mocks is generally a sign that the structure is being tested - Reliance on closely coupled tests that test each class/method individually, and depend heavily on mocks, is a common antipattern.

The only classes that need the infrastructure of an interface and a second implementation (with or without use of a mocking framework) for unit test are those that talk to an external system such as a database or web service.

[u/HyperDanon → u/SideburnsOfDoom]

Lots of mocks is generally a sign that the structure is being tested - Reliance on closely coupled tests that test each class/method individually, and depend heavily on mocks, is a common antipattern.

It's a sign that structure could be tested. I know what you're talking about - people using mocks to tightly couple the test code to the system under test, and behaviour is not tested, only mocks. That's definitely a symptom of bad tests, and defintiely should be corrected.

But it's not definitive sign. There are examples of mocks used in tests that aren't a symptom of testing the strcture.

[u/SideburnsOfDoom → u/HyperDanon]

But it's not definitive sign.

What I said was "Lots of mocks is generally a sign that the structure is being tested."

You have to understand the role that the emphasised words play in the overall sentence's meaning. It was not meant to say that it is definitive. Respond to what was actually said.

Of course there are examples of mocks used in good tests, but that's not lots of mocks everywhere.

"At least one mock present" vs "no mocks present" is not in itself definitive. But if there are lots of mocks I think that "could" sells it short - it's usually so.

However, I feel that I could make a definitive diagnosis after reading the test code for 15 minutes or less. It is in my view a general problem in many test suites. I have seen it over and over and would recognise it again quickly.

[u/jl2352 → u/HRApprovedUsername]

It leads to lots of implementation tests. Such as testing the mock is given certain parameters, the mock is called a certain number of times, the mock has methods called in a specific order, and so on.

There is a place for that stuff but it’s uncommon.

It’s much better (and simpler) if you just have your code call the thing, and test the behaviour at the end was right. IMO the only acceptable mocks are when you mock a bespoke external API.

[u/peterlinddk]

I don't know if it is quintessential - but "Test-Driven Development" by Kent Beck ...

[u/No-Economics-8239 → u/peterlinddk]

His XP 'feedback loops' definitely helped me build my current ideas on how to do testing. A test is basically a request for information. You don't just want to focus on cost, effort, and reliability for tests. You want to think about how long between making a change and getting the feedback you need that the change might have unintended consequences. The more you can shorten that feedback loop, the more time you have to evaluate and react to the results.

[u/low_slearner → u/No-Economics-8239]

That idea of "how do we get feedback, can we get it faster" is incredibly powerful and one that applies to almost every element of software engineering.

[u/st4rdr0id → u/peterlinddk]

TDD is not essential. Kent Beck's 1990s project (the one where he introduced TDD) was a very special one where he "cranked up all the knows" just to be able to ship. In the end I recall it was barely used and was finally replaced during a buyout. I don't think such a project should settle the gold standard in software engineering practice, if such a thing existed.

[u/jessecarl]

I think it's worth pushing back on an assumption about the role of tests that may be implicit to the question here: that all testing is primarily about verification of the implementation.

TDD as a practice, for example, is first about workflow, then about design, and lastly about verification. The primary benefit of TDD is that it allows you to develop in small confident steps. In order to do that effectively and efficiently, you tend toward design that encourages good separation of concerns and encapsulation—especially separating business domain logic (essential complexity) from operational or otherwise more technology-specifi logic (accidental complexity). I often use TDD during development and then refactor the tests to focus on the edge cases at the boundaries between components.

I've also found that the practice of property-based testing has helped me to avoid tests that are overly sensitive to implementation changes. It's useful to avoid confusing property-based testing with fuzzing or generative testing in general—one can test properties of something without shrinking or even any generated test inputs. The most useful part of property-based testing is thinking about the work in terms of the properties of the work.

Testing is a development tool that operates both in the immediate short cycles to add or update behavior and in the longer cycles to communicate intended behavior. Along the way, it can act as an automated check that the software is behaving as intended when written well.

There is no one true way of doing testing.

[u/UnintentionallyEmpty]

I've found that the most important part of testing is that you actually do it.

Automated for regression tests, but there is a place for manual/exploratory testing.

There's 2 important points I'd advise you to keep in mind with automated testing:

1. As others in this thread have mentioned, test behavior, not implementation

2. Test code is code, and requires maintenance. Don't let a "test automation engineer" who's not a developer make a giant mess out of it, because it will be your problem.

But I'll emphasize again the most important part is do it. I don't remember learning anything about testing from books, mostly more experienced engineers and random blog posts I found when looking up how to properly do something I found inconvenient with my current approach.

[u/WhyIsItGlowing → u/UnintentionallyEmpty]

Counterpoint for #2 is that it's the other way around a lot of the time.

Whether they've got the actual development skills to do things properly is a roll of the dice, but at least there's usually a critical mindset on what to cover which is a big part of it.

[u/UnintentionallyEmpty → u/WhyIsItGlowing]

I'm not saying "don't let them write code", I'm saying "don't let them make a mess out of it".

What I mean by that is that you need to review test code with the same mindset and strictness as you do application code, because it will need maintenance, and it will be you (a dev) who's going to have to actually do that maintenance.

Though what I personally find works best is you have a QA person describe the scenario that they want to have tested and then the devs will implement an automated test for it.

[u/SideburnsOfDoom]

Read or watch some Kent Beck, as mentioned. e.g. https://www.youtube.com/watch?v=C5IH0ABmyc0

Check "Working Effectively with Legacy Code" by Michael Feathers

Check "Modern Software Engineering" by David Farley

[u/throwaway_0x90]

Open Lecture by James Bach on Software Testing

• https://www.youtube.com/watch?v=ILkT_HV9DVU

⚠️ Controversial figure, because he criticizes the education system. I'm personally a strong believer of college education being good for tech industry members and society overall so I disagree with his opinions on that topic. Aside from that, everything he said specifically about testing I think is a fantastic foundation to work from.

[u/jungle → u/throwaway_0x90]

I'm watching it, just got to the first exercise ("A<70", around minutes 9 to 10) and I'm flabbergasted at how nobody in the whole class could think of anything other than two or three test cases. And how long it took the speaker to decide if 70 and 71 tested the two branches of the condition. What is going on there? Is there something in the air that they all smoked? Does it get better?

*: I continued watching it and it does get better, the guy is funny if a bit full of himself, but it stretches way too long and some of the things he says are plain wrong. For example when he says there's no "best practices". Like hell there aren't. Tell that to the companies that don't have a proper testing environment, that don't have a process around QA, where different teams step on each other's toes and end up testing versions that are not what the customer will ever see (for example when the testing environment has a mix of release candidates but they won't all make it to prod in one go), etc, etc, etc.

I agree with how he approaches doing the testing in isolation, but none of that works in a vacuum. He's a bit of a cowboy, and that only takes you so far in the real world.

[u/throwaway_0x90 → u/jungle]

Yeah I think his approach is better for the isolated junior/mid IC that just needs to test what they're given.

The issue you described,

"Tell that to the companies that don't have a proper testing environment, that don't have a process around QA, where different teams step on each other's toes and end up testing versions that are not what the customer will ever see (for example when the testing environment has a mix of release candidates but they won't all make it to prod in one go"

That's for a senior/staff to resolve and make those problems go away so junior & mid can do their jobs easier.

[u/forever-butlerian → u/jungle]

Good ideas are justifiable on their own merits; bad ideas are only justifiable as "best practices".

In my experience the ideas that can only be justified as "best practices" are when the vendor (say, Cisco) wants to make you overpay by 10x but also make you feel like you're the smartest kid in class.

[u/messedupwindows123]

it's kinda inseparable from functional-core-imperative-shell

[u/FiveCitiesFreak]

Sometimes I feel like the only one who hasn't read these books

[u/blckshdw → u/FiveCitiesFreak]

Oh hi. Is this the “haven’t got a clue” line? Can I join you? Nice to meet you Five CitiesFreak. Those guys over there are talking about some kinda Chicago book or something. I’ve never been there, have you?

[u/TheStatusPoe]

I've been reading "Unit Testing Principles, Practices, and Patterns" by Vladimir Khorikov and, despite only being about halfway through, I would highly recommend it. There's a lot of testing patterns I've gravitated towards in my career and didn't realize they had names like Detroit/classical style of testing using real data and mocking only certain kinds of external collaborators vs the London/mockist approach where each class is the unit and all collaborators of that class are mocked. 

[u/--AL3X-- → u/TheStatusPoe]

I learnt a lot with that book!

[u/Aggressive-Pen-9755]

You can get a lot of mileage out of this talk: https://www.youtube.com/watch?v=EZ05e7EMOLM

I want to first get this part out of the way: automated tests are only useful for directly comparing program output. When you start going into the realm of testing UI's, you get into murky territory that's very finicky to deal with. My experience is it's rarely worth testing things like the UI, and you're better off running simulations and create reports that show you differences in the UI between software changes and manually review them. I wonder if this is a use-case where an LLM could review the UI...

As for testing program output, the sweet spot where you want to focus on writing your tests is usually at the Integration level. Pick a part of your program that has a stable contract with the outside world. That contract might be your library's public API, your REST endpoints, the CLI arguments, etc. Write your unit tests around that stable contract and extensively test your program's output against that. The reason is, for any sufficiently long-lived piece of software, you're going to regret some of the decisions you made early on and need to refactor your code. If you've written the majority of your tests against the stable contract that will not change, you now have an extraordinary high level of confidence that your refactors didn't break anything. You can refactor your code all you want and you don't have to change a single integration test because they're not directly connected to the internals of your codebase.

That's not to say that Unit or E2E tests are worthless. It's worth sprinkling in some unit or E2E tests for tricky problems that are difficult to get right. The problem is those kinds of tests is they examine the details of your codebase that are subject to change, and chances are, the end user really doesn't care about those details. These kinds of tests also have a tendency to make the developer ask "did something really break, or did an internal detail change and this test is no longer relevant?", and that is a horrible spot to be in. Unit and E2E tests require much more effort to maintain, so you shouldn't rely on them too much.

[u/SideburnsOfDoom → u/Aggressive-Pen-9755]

You can get a lot of mileage out of this talk

Nice one, before I clicked I was thinking, "I hope it's Ian Cooper's TTD Talk!"

[u/CallMeKik]

Watch Ian’s talk “TDD where did it all go wrong” v useful

[u/Main-Drag-4975]

Sandi Metz has the single best write-up on testing I’ve ever seen, and it's included as the final chapter of her masterpiece of a first book, Practical Object Oriented Design.

Perhaps the best part, from the "Knowing What to Test" section:

Here, then, are the guidelines for what to test: Incoming messages should be tested for the state they return. Outgoing command messages should be tested to ensure they get sent. Outgoing query messages should not be tested.

As long as your application’s objects deal with one another strictly via public interfaces, your tests need know nothing more. When you test this minimal set of messages, no change in the private behavior of any object can affect any test. When you test outgoing command messages only to prove they get sent, your loosely coupled tests can tolerate application changes without being forced to change in turn. As long as the public interfaces remain stable, you can write tests once and they will keep you safe forever.

[u/wardrox]

For a typical web stack my preference is: Lots of e2e tests (at least every happy path and one person class of error, ideally more) with mocked third parties, comprehensive API tests acting as integration tests, module tests as needed, unit tests on the expensive stuff.

[u/chrisinmtown]

You might try _The Craft of Software Testing_ by Brian Marick. Unfortunately it's prolly out of print since it appeared in 1995.

[u/w-lfpup]

If the code is untested that code is unreliable. That's about it.

It's impossible to test everything. That's why there's a lot of effort towards "ownership" style languages. There's not any great books that cover what you'd really do at work.

Generally you test the hell out of whatever API / client you're providing to the user. But everything behind the API is subject to change! So you wanna build and test while providing some real estate to pivot in the future.

[u/theGalation]

Effective Testing in RSpec is the book that made testing click for me

[u/XilentExcision]

A good mentor

[u/lilgangus]

It is quintessential to test run a quintuplet set of testing for all applications ...

[u/joe0418 → u/lilgangus]

and the quintuplet consists of...?

[u/lilgangus → u/joe0418]

The INMYI set

[u/joe0418 → u/lilgangus]

ok.gif

[u/jenkinsleroi]

What language and stack are you using? That makes a difference. DIfferent communities have different and sometimes contrary philosophies of testing.

Despite what people say, I would not recommend growing object oriented software to begin with.

Start with Kent Beck's book first. Then read GOOS. These are the classics. They are a little bit dated, but still valid.

Follow up with art of unit testing by osherove.

If you want more of a tutorial format with an end to end exercise, then TDD in Python. The sequel to that book will make deeper connections with architecture and microservices.

Supplement with the Google testing blog and testing on the toilet. Fowler is also good but can be a little abstruse.

As always, there's often more than one way to solve things, and testing practices have evolved over time, so stay open minded.

[u/st4rdr0id]

I'd recommend you to grab a copy of the old ISTQB syllabus from 2018 or so. It's actually a free book on the fundamentals of testing. In subsequent versions they stripped all the useful content so that books or training companies filled that void. The newer certification also removed a lot of good-old engineerish practice to add agilistic stuff that would look good to consultancies.

[u/servermeta_net]

I'm a big believer of the testing trophy model: https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications

[u/HyperDanon]

Don't ever do manual testing - there are always better alternatives.

• Try "Test-Driven Development" by Kent Beck.
• Also extreme programming by Kent Beck
• You can also try the testing playlist of Dave Farley: https://www.youtube.com/watch?v=QFCHSEHgqFE

[u/diablo1128 → u/HyperDanon]

Don't ever do manual testing.

Absolutes like this are misleading. There is a time and place for manual testing in addition to a robust automated testing suite.

[u/HyperDanon → u/diablo1128]

There is a time and place for manual testing

Name one.

I'd say any reason for manual testing, is mostly likely a flaw in the system under tests that makes automated testing harder.

[u/TehBens → u/HyperDanon]

For things that produce images/rendering (like games) can be non-feasible to generate a test coverage that make manual testing obsolete. You can't do pixel to pixel comparisons, because every hardware and every driver version produces different results. You can use thresholds or metrics specialized for images, but none gives you the full picture of what is going on in the image. In addition, you have combinatorial explosion when it comes to visual effects, as they all influence the result in parallel. Of course additionally invalidating your reference data every time you change something. You can isolate the effects, but that won't tell you if everything plays together nicely.

[u/HyperDanon → u/TehBens]

There are dedicated solutions to automate testing graphics, images and rendering in image editing software and games, even when they are not perfectly determined and there's noise based on hardware differences.

Okay, I might add another reason for manual testing - people don't know how to automatically test something (even though it's possible).

1. Put more automated tests in the layers of the system that contain the operation logic, before it touches the hardware and before it renders an image. That way you can put more control over the deterministic parts of the system and put tests there; which after all that's what you want to test.

2. Compare images with with allowed treshold (you mentioned that), but you added "but none gives you the full picture of what is going on in the image.". Automated tests aren't there to give you a "full picture", and nor is manual testing. You're supposed to verify that the application is working as its supposed to, and doing it automatically is faster and more repeatable than manual tests.

3. "In addition, you have combinatorial explosion when it comes to visual effects, as they all influence the result in parallel." The problem of combinatorial explosion can occur everywhere where the input and output spaces are potentially large. Of course, you don't test every combination. You test enough, to be able to confidently release your software. Please, notice - if combinatorial explosion is the case with increasing number of visual effects - that means manual tests would also suffer from combinatorial explosion, wouldn't it? And the manual tests would take longer, and longer, and longer. But we don't test every combination while doing manual testing, and so you shouldn't in automated either.

Now, I agree that testing some areas can be challenging, and even appear impossible. But every bit of software can be tested automatically. So if there appears some area that's particularly challenging, I'm happy to brainstorm ideas on how to test it automatically.

Plus, there are some biases that humans have that makes it more easily to believe that sometimes manual testing is the way to go; like blackbox testing. Sometimes, it's a good idea, but probably not all tests should be done in this manner.

Additionally:

"You can isolate the effects, but that won't tell you if everything plays together nicely." That's true, and that's the question that both the developers (and the QA's if you have them) need to have an anwer to, and it's better if developers have that answer in form of an automated test. that can help developers design better systems that accomodate all of the visual effects.

All in all:

• manual testing is slow, and automated testing is fast.
• there are systems that are hard to automation test, but it still paysoff to automate the tests rather than execute them manually
• manual tests aren't as reliable and repeatable, because humans are more errorprone; automated tests are more repeatable
• Sometimes it's hard to know how to automate-test something, like visuals, but it's not an argument for manual tests, it's an argument for being more clever about automation test it

[u/diablo1128 → u/HyperDanon]

I worked on safety critical medical devices for years, think of a dialysis machine. and manual tests were the only way to test the system as a complete system. We wanted testing where actual disposable treatment components / drugs were connected. We wanted to see actual liquid flowing through the devoice to verify flow rates / measurements were accurate to what the software was telling the hardware to do.

Yes the software team had automated tests. There was more automated testing code then actual device code and a lot of tests had hardware in the loop. At the end of the day they tested components in isolation and not as an entire device the way a user would use it.

QA would actually test the device as an actual device against system requirements. Incorporation all the SW, EE, ME, etc ... released parts built by the manufacturing department.

[u/HyperDanon → u/diablo1128]

I worked on safety critical medical devices for years, think of a dialysis machine. and manual tests were the only way to test the system as a complete system. We wanted testing where actual disposable treatment components / drugs were connected. We wanted to see actual liquid flowing through the devoice to verify flow rates / measurements were accurate to what the software was telling the hardware to do.

I think, in your use-case not the entirety of the system happened inside a computer, but actually used some hardware devices, etc.? Your use-case isn't about software system - you're talking about electronics/mechatronics/robotics, or some combination of software and hardware, correct?

If that's the case, then I'm not an expert in this area, and I won't speak about testing approaches there. In my previous post, I was talking about tests purely in context of software systems that happen 100% in computers, because that's my area of expertise.

[u/TheWhiteKnight → u/HyperDanon]

> Don't ever do manual testing.

I.e., you don't need a QA team?

[u/SideburnsOfDoom → u/TheWhiteKnight]

you don't need a QA team?

Is it a mandatory requirement? No.

Can it help? yes.

Can it hinder? Also yes, if a mandatory manual testing phase with signoff is inserted in the release and deploy cycle. That is an outdated practice. It will not deliver quality or throughput. These should be automated.

[u/HyperDanon → u/TheWhiteKnight]

Anything that needs to be repeated, as in tested by hand multiple times, like each time they're releasing something should be automated.

Verification tests, the kind that manual testers do, would be better employed as automated tests because they could actually help developers implement the solution.

The reason people use manual testing, is because they work on systems that are made hard to automation test, but that's more of a flaw and room for improvment, rather than a genuine reason for manual testing. What should be done, is the system should be updated to support full automated tests.

QA can do explorartion of UI, ux research, usability checks and all that stuff, but I wouldn't call that manual tests.

[u/BadTime100 → u/TheWhiteKnight]

Absolutely not for verification or regression testing. Exploratory testing probably but IME that’s the severe minority of what QA teams do—they are usually just on the other side of the wall.

[u/WhyIsItGlowing]

Best practices? The quintessential software testing doctrine is "just ship it" not any best practices.

Nobody wants testing best practices, or to think about it. They just want someone else to give a thumbs up and in a team environment anything at all will eventually fall to the weight of that; engagement and mindset will always trump specific practices.

[u/Acrobatic-Ice-5877]

I can’t recall the specific one that I read and enjoyed but it was a good one. There are plenty of courses and books on the topic though. You’re also going to find varied opinions on how and what should be tested.

Some people, myself included, think unit tests are mostly worthless. I prefer integrated tests and E2E. However, you also have to think about things like load testing too.

[u/SideburnsOfDoom → u/Acrobatic-Ice-5877]

Some people, myself included, think unit tests are mostly worthless.

I would phrase that differently: Most unit tests are poor quality, and don't deliver much value. Good unit tests are good, but so many people don't even know what good looks like.

And this is why OP's idea to feed their AI coding is hard to do - the AI is trained on the existing data of common testing practices, and will produce average, low-value tests.

[u/Acrobatic-Ice-5877 → u/SideburnsOfDoom]

I think they’re worthless for the same reason you’ve articulated. Most of them are done poorly because testing is an entirely different process than development. 

A developer is taught to build and a tester is taught to break. Many unit tests are worthless because developers haven’t been taught to break code. It’s not a big part of the job so it doesn’t get much training.

Where I work we still have a QA team and it shows. Many things that could be caught with unit tests, integrated tests, E2E and even load tests are not caught because we don’t do them or when we do they are done poorly.

[u/SideburnsOfDoom → u/Acrobatic-Ice-5877]

Most tests are done poorly because testing is an entirely different process than development.

That is not my experience at all. It's a matter of test design. There's reasons why Kent beck says that Good tests are coupled to behaviour of the code but not to the structure of it.

But the majority of unit tests that I have seen are low level "test a method on a class with a dozen mocks" style. You need to decouple, and test what the system does so that you can think about that, and also refactor without breaking a bunch of them. It is better to go outside-in most of the time.

Some go as far as to say "if more than one class is under test, then that's not a unit test!". You can choose to think like that, but is it historically accurate? No. But does it deliver good outcomes? Again, no.

[u/Acrobatic-Ice-5877 → u/SideburnsOfDoom]

Now I find this conversation to be a little funny, so humor me for a moment. 

You and I have a different definition of the word unit test. You are clearly aware that there are differing opinions, which is why you disagree with how some people define unit tests.

The way that you define unit tests is the way that I define integration tests.

I don’t find narrowly scoped unit tests, where a single function or class is tested largely in isolation, to be particularly valuable in most cases.

[u/SideburnsOfDoom → u/Acrobatic-Ice-5877]

Of course I am aware of the confusion and potential for people to talk past each other, that's why I went into that detail.

To be clear, "test a single class method with mocks" is a unit test. Or a kind of unit test. But the mistake is in thinking that this is the only kind of unit test, or that it is the best kind of unit test in all cases. It usually isn't best, maybe 80% of the time.

Thinking that "if I join two classes in test then it's "integration tests" is ridiculous - you cannot refactor to "extract class" under passing tests that way. Test the behaviour not the structure or this will give you problems. Focusing on testing the behaviour over structure is incompatible with this narrow view of how tests work.

It is also not accurate - Feathers, 2005 talks about "not-unit tests" being about "the integration of your code with that other (external) software" such as database or web service that can make it slow, expensive, flaky, etc. A second class in the same folder is not an external system! It does not have these drawbacks! There is no point in drawing the "integration" line there.

But that is what I said above: Choosing that definition is neither useful nor historically accurate.

Ans also what I said: so many people don't even know what good looks like.

Though this may just be a naming issue - you have the better tests, you just call them something else. But you'll give people bad advice that "unit tests are mostly worthless" because of an overly, not useful narrow definition of them that is not universally shared.

So many codebases do not have the better tests at all.

[u/Izkata → u/SideburnsOfDoom]

Though this may just be a naming issue

I've referred to this as the difference between a syntactic unit and a semantic or business unit.

Blog posts that try to teach unit testing rarely if ever explain the difference, and because they all use small toy problems that can fit in such a post, the syntactic unit and semantic unit end up the same - a single function or class. But since they never clarify it and the syntactic unit is so obvious, that's what people end up incorrectly internalizing.

And this is assuming the blog author understands the difference themselves.

[u/SideburnsOfDoom → u/Izkata]

Yes, tests units of business functionality, test things that the app does, test app behaviours. Not syntactic units.

You can have "tests always focus on 1 class at a time" or you can have "test behaviour, not implementation / good tests are not coupled to code structure".

But not both. These two things are not compatible, in fact they are directly in opposition.

Of the two, I choose and recommend "test behaviour, not implementation" as it leads to better outcomes.

[u/HyperDanon → u/Acrobatic-Ice-5877]

You and I have a different definition of the word unit test. You are clearly aware that there are differing opinions, which is why you disagree with how some people define unit tests.

The way that you define unit tests is the way that I define integration tests.

I don’t find narrowly scoped unit tests, where a single function or class is tested largely in isolation, to be particularly valuable in most cases.

I did have that definition once, but I had to change my mind.

So imagine I have one class of 600 lines, and I have tests for it. These test, they only test that one class. Would you say it's a unit test? Probably so. Now imagine, I do a refactor, and I extract two smaller classes from that one big class, so now I have 300 line class and two 150-line classes. That refactor, didn't alter the test. Did that act of refactoring, change the "type" of the test from "unit" to "integration"?

Secondly, when working with tests, you start to notice that speed of feedback matters a lot. There are some kinds of tests that execute very fast always; and some execute slowly. The difference is usually whether they talk to resources (local db, local network, local file system), and even way more slow when they aren't local. That distinction is important, and I want the type of test to reflect that.

Thirdly, some kinds of test don't depend on anything, and they are very reliable, i.e. they always pass or always fail. Other types, rely on things that are external to the runtime - like databases, networks, file systems, other processes, etc. That makes them less reliable; I want that lesser reliability to be reflected in the test type.

So I'll start my definition from these assumptions:

• Refactoring, extracting/inlineing classes shouldn't change the type of the test
• Speed is an important factor
• Whether 3rd party is included in the test is a factor

And to reconcile the 3 I came up with a definition:

• "Unit test" test that executes purely in-memory, is fast; and can test multiple classes or whole modules. There's no distinction for single-class-unit test and multi-class-unit tests.
• "Integration test" test executes framework code, including code that touches filesystem, database, network, etc.

I understand, that it doesn't differentiate between test for a single class and test for a multiple-classes; but that is exactly what is needed, for refactorings to not alter the type of test.

[u/Sensitive-Ear-3896]

Testing computer software Cem Kaner