Acceptance tests, integration tests, component tests. Developers discuss test strategy with the help of these phrases but sometimes the definitions remain unclear. Before specifying eg. a component test, we should first figure out what exactly we mean by a component. This applies to other test levels too: to avoid misunderstandings, first specify the context and the boundaries of the system component we plan to test.
Consider also the environment where the test runs, not just the functionality we want to verify. Sometimes it’s makes sense to run the same tests in different environment (eg. simulated vs physical hardware)
With the system architecture in mind, we can organise tests based on:
TDD is successful because people started writing regression tests not because X% test coverage results in good code.
TDD is a discipline for programmers - Robert C Martin
Not a dogma.
Sometimes we don’t have the detailed architecture in front of us when writing code. So we create new units to see what works. Then ask ourselves if this unit is really worth having. This is an iterative design process we do on a daily basis. But how can we write unit tests if we don’t know what units are worth writing at all?
Test-first units leads to an overly complex web of intermediary objects and indirection in order to avoid doing anything that’s “slow”. - David Heinemeier Hansson
Some other benefits of TDD:
Against TDD:
Perform tests on different levels: TDD != Unit tests
- runs fast
- helps us to localise problems
It’s not a problem if the test covers more than one “unit”. But it needs to run fast.
It’s a unit when:
A test is not a unit test if:
- It talks to the database
- It communicates across the network
- It touches the file system
- It can’t run at the same time as any of your other unit tests
- You have to do special things to your environment (such as editing config files) to run it.
Yet it’s important to test integrations between units. And we can use unit test frameworks for this purpose, even though this is more than a unit test.
What is a test framework?
How to do unit testing in a constrained environment (embedded):
Want acceptance test driven development? Try using Robot. Robot is a DSL trying to be a scripting language. Decouple tests from the code as much as possible. Describe high level tests in a special language.
This is not your favourite scripting language:
Can we actually reuse keywords?
Or there will be few heavily-used keywords, and a really long tail of one-time keywords, which is a pain.
Does it with work with embedded?
Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don’t want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it’s own sub-directory. However, these tests no longer share a suite scope (because robot’s “suite scope” is actually a file scope” for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups. - elevation
Alternative if we don’t want another obscure DSL: pytest
Acceptance test
enable the user, customers or other authorized entity to determine whether to accept the system.
Acceptance test is normally black box testing. Developer test focuses on the code. We can combine these.
Component test
Once we have a definition for a component, we can call it a component test.
C/C++ Metrics: Line Counting metrics
https://lwn.net/Articles/637151/