Skip to content

TESTING.md

Latest commit

 

History

History
145 lines (118 loc) · 9.33 KB

TESTING.md

File metadata and controls

145 lines (118 loc) · 9.33 KB

A brief overview of SpacetimeDB testing

From the perspective of a client SDK or module library developer

By pgoldman 2025-06-25, updated 2026-04-14

SpacetimeDB has good test coverage, but it is rather haphazardly spread across several suites. Some of the reasons for this are historical, and some have to do with our using multiple repositories. This document is an attempt to describe the test suites which would be useful to someone working on a new client SDK or module bindings library in a new language, or attempting to move an existing client SDK or module bindings library from an external repository in-tree.

The SDK tests

crates/testing/src/sdk.rs defines a test harness which was originally designed for testing client SDKs. The basic flow of a test using this harness is:

  • Build and freshly publish a module to construct a short-lived initially-empty database.
  • Use that module to run client codegen via spacetime generate into a client project.
  • Compile that client project with the newly-generated bindings.
  • Run the client project as a subprocess, passing the database name or Identity in an environment variable.
  • The client process connects to the database, runs whatever tests it likes, writes to stdout and/or stderr as it goes, then uses its exit code to report whether the test was successful or not.
  • If the subprocess's exit is non-zero, the test is treated as a failure, and the subprocess's stdout and stderr are reported.

This framework has since been used more generally for integration testing. In particular, we maintain equivalent Rust, C#, TypeScript, and C++ modules in the modules/sdk-test* family, and run the Rust SDK client project at sdks/rust/tests/test-client against them through sdks/rust/tests/test.rs. We similarly maintain modules/sdk-test-connect-disconnect* modules which run against sdks/rust/tests/connect_disconnect_client. There are also related SDK-harness-driven suites for event tables, procedures, and views, using modules such as modules/sdk-test-event-table, modules/sdk-test-procedure*, and modules/sdk-test-view*. The Unreal SDK also uses the same underlying harness through sdks/unreal/tests/sdk_unreal_harness.rs.

The harness is designed to support running multiple tests in parallel with the same client project, running client codegen exactly once per test suite run. This unfortunately still conflicts with our use of the suite to test that modules in different languages behave the same, as each test suite invocation will only run spacetime generate against one module language at a time, never all of them in the same run.

Testing a new module library

If you are developing a new module bindings library, and wish to add it to the SDK test suite so that the existing client test projects will run against it:

  1. Create modules/sdk-test-XX and modules/sdk-test-connect-disconnect-XX, where XX is some mnemonic for your language. Populate these with module code which defines all of the same tables and reducers as modules/sdk-test and modules/sdk-test-connect-disconnect respectively. Take care to use the same names, including casing, for tables, columns, indexes, reducers and other database objects.
  2. Modify sdks/rust/tests/test.rs to add an additional call to declare_tests_with_suffix! at the bottom, like declare_tests_with_suffix!(xxlang, "-XX"), if that driver is the right place for the new language. Some capabilities now live in separate suites in that file, such as procedures and views, so you may need to wire those up separately as well.
  3. Run the tests with cargo test -p spacetimedb-sdk --test test.

Testing a new client SDK

If you are developing a new client SDK, and wish to use the SDK test harness and existing modules so that it will run against modules/sdk-test and modules/sdk-test-connect-disconnect:

  1. Find somewhere sensible to define test projects test-client and connect_disconnect_client for your client SDK language. If your client SDK is in-tree, put these within its directory, following the existing layout under sdks/rust/tests/ or sdks/unreal/tests/.
  2. Use spacetime generate manually, or via the harness, to generate those projects' module_bindings.
  3. Populate those projects with client code matching sdks/rust/tests/test-client and sdks/rust/tests/connect_disconnect_client respectively.
    • Connect to SpacetimeDB running at http://localhost:3000.
    • Connect to the database whose name is in the environment variable SPACETIME_SDK_TEST_DB_NAME.
    • For test-client, take a test name as a command-line argument in argv[1], and dispatch to the appropriate test to run.
    • For connect_disconnect_client, there is only one test.
    • The Rust code jumps through some hoops to do assertions about asynchronous events with timeouts, using an abstraction called the TestCounter defined in sdks/rust/tests/test-counter. This is effectively a semaphore with a timeout. You may or may not need to replicate this behavior.
  4. Create integration tests in the SDK crate which construct spacetimedb_testing::sdk::Test objects, following sdks/rust/tests/test.rs or sdks/unreal/tests/test.rs as a template.
  5. Define #[test] tests for each test case you have implemented, which construct spacetimedb_testing::sdk::Test objects containing the various subcommand strings to run your client project, then call .run() on them.

Adding a new test case

If you want to add a new test case to the SDK test suite, to test some new or yet-untested functionality of either the module libraries or client SDKs:

  1. If necessary, add new tables and/or reducers to modules/sdk-test and friends which exercise the behavior you want to test.
  2. Add a new function, exec_foo, to the appropriate client project, such as sdks/rust/tests/test-client/src/lib.rs, which connects to the database, subscribes to tables and invokes reducers as appropriate, and performs assertions about the events it observes.
  3. Add a branch to that client's dispatch logic, such as the match in sdks/rust/tests/test-client/src/main.rs, which matches the test name foo and dispatches to call your exec_foo function.
  4. Add a #[test] test function to the relevant test driver, such as sdks/rust/tests/test.rs, which does make_test("foo").run(), where "foo" is the test name you chose in step 3.
  5. Repeat steps 2 through 4 for any other client projects which ought to cover the same behavior.
  6. Run the new test with the relevant cargo test command for that SDK.

Schema parity tests

crates/schema/tests/ensure_same_schema.rs is a separate but important companion to the SDK tests. It compares the extracted schemas of equivalent modules across languages, and is often the first place where casing, indexes, primary keys, or other schema details drift apart. As of writing, it covers the benchmarks, module-test, sdk-test, and sdk-test-connect-disconnect families.

If you add or update a cross-language module family, it is worth considering whether it should also be covered here.

The smoketests

crates/smoketests/ defines an integration and regression test suite using a Rust harness. These are useful primarily for testing the SpacetimeDB CLI, but can also be used to exercise publish flows, documentation, and other end-to-end behavior.

The smoketest harness is still primarily oriented around Rust modules, and it does not use the same client-project machinery as the SDK harness. It could be extended to do more in that direction, but that may not be worth the effort. As of writing, the smoketest suite includes dedicated coverage such as:

  • crates/smoketests/tests/smoketests/csharp_module.rs, which smoke-tests C# module compilation.
  • crates/smoketests/tests/smoketests/quickstart.rs, which replays the quickstart guide for Rust and C#.
  • crates/smoketests/DEVELOP.md, which documents how to run and write these tests.

One practical note is that the smoketests use prebuilt spacetimedb-cli and spacetimedb-standalone binaries, so if you modify those crates or their dependencies, you should rebuild before running the suite.

Standalone integration test

The spacetimedb-testing crate has an integration test file, crates/testing/tests/standalone_integration_test.rs. The tests in this file publish modules/module-test, modules/module-test-cs, modules/module-test-ts, and modules/module-test-cpp, then invoke reducers or procedures in them and inspect their logs to verify that the behavior is expected. These tests do not exercise the entire functionality of module-test, but by virtue of publishing it do assert that it is syntactically valid and that it compiles.

To add a new module library to the Standalone integration test suite:

  1. Create modules/module-test-XX, where XX is some mnemonic for your language.
  2. Populate this with module code which defines all of the same tables and reducers as the existing module-test family. If you notice any discrepancies between the existing languages, those parts may be compiled but not run, and so you are free to ignore them.
  3. Modify crates/testing/tests/standalone_integration_test.rs to define new #[test] #[serial] test functions which use your new module-test-XX module to do the same operations as the existing tests.
  4. Run the tests with cargo test -p spacetimedb-testing --test standalone_integration_test.