Summary
Create a test suite for the project to provide automated verification of firmware behaviour and protect the project against regressions during future development.
The test suite should become a permanent part of the project's development workflow and should be maintained alongside the firmware.
A major motivation for this work is that the project is expected to undergo significant architectural changes in the future, including:
- migration from Arduino IDE 1.8.19 to Arduino IDE 2.3.10
- porting from the current predominantly sequential Arduino architecture to an explicit FreeRTOS-based architecture while retaining the Arduino framework
The test suite should therefore establish a reliable baseline of expected behaviour preferably before these changes take place, allowing the existing and future implementations to be compared against the same behavioural expectations.
Goals
The test suite should:
- detect regressions introduced by future code changes
- establish a baseline of currently expected firmware behaviour
- verify important pieces of application logic automatically
- verify interactions between components where appropriate
- make future architectural changes safer
- provide confidence that the firmware continues to behave correctly after refactoring or porting
- integrate with the project's existing development and CI workflow
Test Strategy
The test suite should contain multiple levels of testing rather than relying exclusively on a single type of test.
Unit Tests
Create unit tests for individual pieces of logic that can be tested independently.
Unit tests should focus on deterministic behaviour and should verify:
- expected inputs and outputs
- boundary conditions
- invalid inputs
- error handling
- important edge cases
The tests should primarily verify behaviour and requirements, rather than implementation details.
Integration Tests
Create integration tests where the correctness of a feature depends on multiple components working together.
These tests should verify that the relevant components interact correctly and that the resulting behaviour matches the expected system behaviour.
Integration tests should be added where unit tests alone cannot provide sufficient confidence.
Hardware / System Tests
Identify functionality that cannot be meaningfully verified without actual ESP32 hardware.
Create an appropriate collection of hardware-level or system-level tests for such functionality.
These tests may initially be partially or entirely manual if reliable automated hardware testing is not yet available.
The important requirement is to establish a clear and repeatable procedure for verifying hardware-dependent behaviour.
Regression Testing
The test suite should serve as a regression test suite for the project.
Whenever the firmware is modified, the relevant tests should be run to verify that previously working behaviour has not been unintentionally broken.
Regression testing should be particularly important during:
- refactoring
- dependency updates
- toolchain changes
- Arduino IDE migration
- Arduino Core updates
- FreeRTOS architectural changes
- bug fixes
- feature development
Baseline Before Major Architectural Changes
Before beginning the planned FreeRTOS port, establish a baseline of the current firmware's behaviour.
The existing implementation should be tested and the resulting test suite should represent the expected behaviour of the current stable firmware.
This baseline will then be used throughout the architectural transition.
The objective is to be able to make changes to the internal implementation while continuously verifying that externally observable behaviour remains correct.
For example:
Current implementation
│
▼
Test suite
│
│ establishes expected behaviour
▼
FreeRTOS-based implementation
│
▼
Same test suite
│
▼
Verify that behaviour is preserved
Summary
Create a test suite for the project to provide automated verification of firmware behaviour and protect the project against regressions during future development.
The test suite should become a permanent part of the project's development workflow and should be maintained alongside the firmware.
A major motivation for this work is that the project is expected to undergo significant architectural changes in the future, including:
The test suite should therefore establish a reliable baseline of expected behaviour preferably before these changes take place, allowing the existing and future implementations to be compared against the same behavioural expectations.
Goals
The test suite should:
Test Strategy
The test suite should contain multiple levels of testing rather than relying exclusively on a single type of test.
Unit Tests
Create unit tests for individual pieces of logic that can be tested independently.
Unit tests should focus on deterministic behaviour and should verify:
The tests should primarily verify behaviour and requirements, rather than implementation details.
Integration Tests
Create integration tests where the correctness of a feature depends on multiple components working together.
These tests should verify that the relevant components interact correctly and that the resulting behaviour matches the expected system behaviour.
Integration tests should be added where unit tests alone cannot provide sufficient confidence.
Hardware / System Tests
Identify functionality that cannot be meaningfully verified without actual ESP32 hardware.
Create an appropriate collection of hardware-level or system-level tests for such functionality.
These tests may initially be partially or entirely manual if reliable automated hardware testing is not yet available.
The important requirement is to establish a clear and repeatable procedure for verifying hardware-dependent behaviour.
Regression Testing
The test suite should serve as a regression test suite for the project.
Whenever the firmware is modified, the relevant tests should be run to verify that previously working behaviour has not been unintentionally broken.
Regression testing should be particularly important during:
Baseline Before Major Architectural Changes
Before beginning the planned FreeRTOS port, establish a baseline of the current firmware's behaviour.
The existing implementation should be tested and the resulting test suite should represent the expected behaviour of the current stable firmware.
This baseline will then be used throughout the architectural transition.
The objective is to be able to make changes to the internal implementation while continuously verifying that externally observable behaviour remains correct.
For example: