Testing

Terminology

What are acceptance tests, integration tests and component tests? Before discussing test strategy, make sure to have common understanding of these phrases.

To discuss what is covered by eg. a component test, we need to define what a component is. So first specify system or component context and boundaries.

Then we can organise tests based on:

Test-driven development

TDD is successful because people started writing regression tests not because X% coverage results in good code.

TDD is a discipline for programmers - Robert C Martin

Not a dogma.

Sometimes we don't have the detailed architecture in front of us when writing code. So we create new units to see what works. Then ask ourselves if this unit is really worth having. This is an iterative design process we do on a daily basis. But how can we write unit tests if we don't know what units are worth writing at all?

Test-first units leads to an overly complex web of intermediary objects and indirection in order to avoid doing anything that's “slow”. - David Heinemeier Hansson

Some other benefits of TDD:

Testing is a spectrum: TDD != Unit TDD

What is a unit test?

  1. runs fast
  2. helps us to localise problems

Robert C. Martin: Working Effectively with Legace Code

It's not a problem if the test covers more than one “unit”. But it needs to run fast.

A test is not a unit test if:

Michael Feather

Yet it's important to test integrations between units. And we can use unit test frameworks for this purpose, even though this is more than a unit test.

Minimal test “frameworks”

What is a test framework?

Heavy frameworks get in the way.

It is challenging to do unit testing in a constrained environment.

Yet another integration test framework

Want acceptance test driven development? Try using Robot. Robot is a DSL trying to be a scripting language. Decouple tests from the code as much as possible. Describe high level tests in a special language.

This is not your favourite scripting language:

Can we actually reuse keywords?

Or there will be few heavily-used keywords, and a really long tail of one-time keywords, which is a pain.

Does it with work with embedded?

Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don't want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it's own sub-directory. However, these tests no longer share a suite scope (because robot's “suite scope” is actually a file scope” for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups. - elevation

Alternative if we don't want another obscure DSL: pytest

Dealing with different test levels

Acceptance test

enable the user, customers or other authorized entity to determine whether to accept the system.

Acceptance test is normally black box testing. Developer test focuses on the code. We can combine these.

Component test

Once we have a definition for a component, we can call it a component test.

Metrics

C/C++ Metrics: Line Counting metrics

Fuzzing

https://lwn.net/Articles/637151/

Clean code

Clean code and C++

Updated 05 February 2023