Skip to content

[Feature] Create a comprehensive test suite for the project #49

Description

@RomanAlexandroff

Summary

Create a test suite for the project to provide automated verification of firmware behaviour and protect the project against regressions during future development.

The test suite should become a permanent part of the project's development workflow and should be maintained alongside the firmware.

A major motivation for this work is that the project is expected to undergo significant architectural changes in the future, including:

  • migration from Arduino IDE 1.8.19 to Arduino IDE 2.3.10
  • porting from the current predominantly sequential Arduino architecture to an explicit FreeRTOS-based architecture while retaining the Arduino framework

The test suite should therefore establish a reliable baseline of expected behaviour preferably before these changes take place, allowing the existing and future implementations to be compared against the same behavioural expectations.


Goals

The test suite should:

  • detect regressions introduced by future code changes
  • establish a baseline of currently expected firmware behaviour
  • verify important pieces of application logic automatically
  • verify interactions between components where appropriate
  • make future architectural changes safer
  • provide confidence that the firmware continues to behave correctly after refactoring or porting
  • integrate with the project's existing development and CI workflow

Test Strategy

The test suite should contain multiple levels of testing rather than relying exclusively on a single type of test.

Unit Tests

Create unit tests for individual pieces of logic that can be tested independently.

Unit tests should focus on deterministic behaviour and should verify:

  • expected inputs and outputs
  • boundary conditions
  • invalid inputs
  • error handling
  • important edge cases

The tests should primarily verify behaviour and requirements, rather than implementation details.

Integration Tests

Create integration tests where the correctness of a feature depends on multiple components working together.

These tests should verify that the relevant components interact correctly and that the resulting behaviour matches the expected system behaviour.

Integration tests should be added where unit tests alone cannot provide sufficient confidence.

Hardware / System Tests

Identify functionality that cannot be meaningfully verified without actual ESP32 hardware.

Create an appropriate collection of hardware-level or system-level tests for such functionality.

These tests may initially be partially or entirely manual if reliable automated hardware testing is not yet available.

The important requirement is to establish a clear and repeatable procedure for verifying hardware-dependent behaviour.


Regression Testing

The test suite should serve as a regression test suite for the project.

Whenever the firmware is modified, the relevant tests should be run to verify that previously working behaviour has not been unintentionally broken.

Regression testing should be particularly important during:

  • refactoring
  • dependency updates
  • toolchain changes
  • Arduino IDE migration
  • Arduino Core updates
  • FreeRTOS architectural changes
  • bug fixes
  • feature development

Baseline Before Major Architectural Changes

Before beginning the planned FreeRTOS port, establish a baseline of the current firmware's behaviour.

The existing implementation should be tested and the resulting test suite should represent the expected behaviour of the current stable firmware.

This baseline will then be used throughout the architectural transition.

The objective is to be able to make changes to the internal implementation while continuously verifying that externally observable behaviour remains correct.

For example:

Current implementation
        │
        ▼
   Test suite
        │
        │  establishes expected behaviour
        ▼
FreeRTOS-based implementation
        │
        ▼
   Same test suite
        │
        ▼
Verify that behaviour is preserved

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

advancedOnly for those who is highly familiar with the code basehelp wantedExtra attention is needed

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions