Multi-version Central compatibility#190
Open
cshmookler wants to merge 53 commits into
Open
Conversation
Implemented both factory functions that construct a CentralAdapter. Implemented a workaround for fetching Central's application and protocol versions.
The translation units containing the factory functions now link with the correct Windows libraries.
Remove the example for the CentralAdapter now that it's an implementation detail of ShmemSession. Add new compatibility version detection logic to ShmemSession::Impl. Split detectCompatProtocol into two separate methods: one for inspecting Central's binary for version information and another for inspecting the shared memory as the original function did. Compatability version detection for a STANDALONE + CENTRAL_COMPAT CereLink instance is (theoretically) supported, but this logic is dead because this configuration is never created by SdkSession::create.
…+ CENTRAL_COMPAT CereLink instances This configuration (STANDALONE + CENTRAL_COMPAT) appears to only be theoretical and is never instantiated by SdkSession.
Add central types for 4.2, 4.1, and 4.0. Move CentralConfigBuffer into config_buffer.h. Place version-specific constants and structs in namespaces and remove CENTRAL_ prefixes.
Macros in cbproto collide with names in central_types/ matching those in Central's source, so switch to variables to respect namespaces. Add annotations of version differences with 'VER:'
…yCFGBUFF. These structs ensure that each version of CentralLegacyCFGBUFF maps exactly to the corresponding version of Central's configuration buffer. Structure differences between versions are annotated with VER:
Each translator file can be diffed with another to see translation differences between versions. Translations are two-way between legacy versions and the current protocol version.
Diffable adapter files for quickly seeing differences between versions. Implement primitive get/set methods for all planned legacy versions (v3.11, v4.0, v4.1, v4.2). Partially implement integration with ShmemSession. Functional v4.2 and v4.1 compatiblity. Theoretically functional v4.0 and v3.11 compatibility, but untested.
Each Central adapter has access to all raw pointer buffers used by ShmemSession. These pointers are provided to the Adapter constructor. Buffer size methods are within the BootstrapAdapter so the buffer pointers are initialized before getting passed to the Adapter constructor. All translator methods must have the translation context provided so the signatures remains the same. The translator for cbCFGBUFF has a fromLegacy translator but not a toLegacy translator. This is due to cbCFGBUFF containing information for multiple instruments which is lost upon translation, thus making reversing the operation impossible without access to the original buffer. The translators for cbPcStatus also result in loss of multi-instrument information. This issue would be resolved by combining the translators with the central adapter. Translators for the spike buffer and cache are necessary for full compatibility.
Removed the CENTRAL_COMPAT layout and replaced it with CENTRAL. Spike buffer and cache operations with the CENTRAL layout are only performed for the most recent version of Central. In the future, most of the ShmemSession implementation should be moved into the Central adapter.
Each ShmemSession operates on a specific instrument. With the CENTRAL layout, the instrument filter is always active. Multi-instrument control is possible by creating multiple SdkSession instances (one per instrument).
Add translators for cbPKT_SPK, cbSPKBUFF, and cbSPKCACHE. Implement getSpikeCache and getRecentSpike for ShmemSession with layout == CENTRAL.
Rework how Blackrock .cmp channel-map files are applied to a device's
in-memory channel config.
Matching: rows are now matched to live channels by the device's own
(bank, term) read from chaninfo, instead of by an ordinal channel ID
(the old sort-and-assign start_chan+N scheme). start_chan shifts the
CMP's bank letters by start_chan/32 banks to target a headstage's banks
(129 -> +4 -> bank A maps to device bank E), which makes (bank, term) a
globally-unique join key. This is robust to incomplete CMPs.
position[]: the four cbPKT_CHANINFO.position slots now carry real
geometry {x, y, size, headstage_id} instead of {col, row, bank, elec}.
Columns: the //-comment header line immediately before the data (when
present) is the ground truth for column order, matched by name
(col/c, row/r, bank/b, elec/e, size/s, label/l); no header falls back
to the legacy "col row bank electrode [label]" order, so existing files
parse unchanged.
Units: with no size column and a unit-spaced index grid (every non-zero
col/row delta is exactly 1), values are electrode indices -> size 1 and
x/y/size scaled by the 400 um Utah-array pitch. With a size column, or
non-uniform spacing, values are taken at face value.
Labels: stored verbatim; the hs{N}- prefix is dropped since the stored
headstage id disambiguates reused labels.
pycbsdk/cmp.py is kept in sync with the C++ parser; session.py
docstrings corrected. C++ and Python unit tests updated; the
hardware-gated nplay integration tests join device readback to parser
output via ChanInfoField.BANK/TERM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng, because some arrays will have gaps in their grid and therefore have >1 spacing in some dimensions.
Rename the proposal to distinquish it from the feature overview (README.md). Fix shared_memory_architecture.md to reflect the changes to the shared memory implementation.
Each ShmemSession instance pertains to a particular instrument. The names of each individual shared memory buffer have been collapsed into a single parameter passed to the constructor. All methods of ShmemSession that previously took an InstrumentId as an argument no longer accept this value. In the future, it may make sense to move the version detection logic in ShmemSession::Impl outside of ShmemSession so it isn't rerun every time a new ShmemSession instance is constructed. A protocol version field could be added to the create() method so the protocol version can be provided.
ShmemSession is now per-instrument, so multi-instrument tests have been temporarily removed. Tests for enumerating active instruments or rejecting 'other' instruments have been removed. Several methods no longer accept an InstrumentId. These changes may be reverted in the future if the per-instrument ShmemSession model is reverted.
cbMAXPROCS and cbNUM_FE_CHANS differ between cbproto and cbhwlib. This difference only matters for Central 7.7.x and 7.8.x because these versions are where Central switched to cbproto types instead of locally defined packet types.
Version detection logic for Central has been moved to central_version.h and central_version.cpp. Two functions, detectCentralVersion and getProtocolVersion (maybe getCentralProtocol instead?), are used by ShmemSession::Impl to fetch the current version of Central and it's protocol version. In the future, these functions will be called outside of ShmemSession so they aren't called every time a new instrument is inspected.
Some transmit/receive buffer logic depends on the transmit/receive methods provided by the Central adapter even if Central is not in use. This fix always constructs the Central adapter for the most recent version so it's available.
…his point forward)
Common methods for all Adapter methods are tested. All copyArr and copyArr2D methods from base.h are tested. detectCentralVersion does not have unit tests and must be tested manually.
Collaborator
|
pycbsdk-windows-x64.zip |
Collaborator
|
I couldn't get this to work on my lab setup running a LEGACY_NSP with firmware 7.6.0 and Central 7.6.1. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow Cerelink to connect to older versions of Central.
Summary
Previously, CereLink could only interoperate with the single latest version of Central's shared
memory. This branch introduces a versioned compatibility layer that lets CereLink attach to
Central's shared memory across multiple protocol versions (7.0, 7.5, 7.6, 7.7, and 7.8/current),
automatically detecting the running Central version and translating its structs to and from
CereLink's native types.
The change also reworks ShmemSession into a per-instrument, layout-aware session and modernizes its
API across cbsdk and the examples/tests.
Motivation
Central exposes its configuration, status, and spike buffers through shared memory, but the binary
layout of those structs changes between protocol versions. Central's shared memory carries no usable
version field (the version field is a magic constant, and procinfo[].version's byte offset shifts
between versions), so CereLink previously assumed the newest layout and broke silently against any
other version. This work makes cross-version compatibility explicit and maintainable.
Approach
Adapter layer. Each supported protocol version has a BootstrapAdapter and Adapter. The
BootstrapAdapter reads the sizes of Central's shared-memory structs and instantiates an Adapter
holding raw pointers into each buffer. Each Adapter translates version-specific Central structs
to/from the native CereLink equivalents (cbproto + native_types.h), which are the common language
ShmemSession operates on.
across versions so they diff cleanly)
Splitting the previous monolithic central_types.h/adapters.h into per-version files makes
version-to-version differences reviewable via simple diffs, and adding a version is a documented
copy-diff-rectify procedure (see docs/multi_version_central_compat/README.md).
Version detection. New central_version.{h,cpp} detect Central's version on Windows by inspecting
Central.exe's ProductVersion (via the running-process list) and mapping the application version to a
protocol version. Detection was deliberately moved out of ShmemSession so the version is resolved
once and the correct adapter is selected at open. getCompatProtocolVersion() exposes the resolved
version to callers (returns CBPROTO_PROTOCOL_CURRENT for the NATIVE layout).
Per-instrument, layout-aware ShmemSession. The session now takes an explicit ShmemLayout (CENTRAL vs
NATIVE) and is created per instrument:
ShmemSession::create(Mode mode, ShmemLayout layout,
const std::string& name_qualifier, cbproto::InstrumentId id);
Segment names are synthesized internally from the layout and qualifier (Central's fixed well-known
names + instance suffix for CENTRAL; cbshm__ for NATIVE). Indexing always uses
packet.instrument so standalone-written packets land in the slot a later CLIENT would read.
Scope / Limitations
The compatibility layer covers Central's configuration, status, and spike buffers. The
receive/transmit buffers are still handled by version-specific logic spread across cbdev, cbproto,
and cbshm; folding these into an adapter is noted as future work in the docs.
Key Changes
central_types/v7_* headers, central_version.{h,cpp}, central_current.h
multi-instrument-safe cbPcStatus), large cbproto/types.h cleanup (macros → global constants; type
names matched to Central)
stack overflow when translating large structs; id/idx discrepancies; MSVC/Windows build fixes for
the adapters
runbook); updated shared-memory architecture doc and cbshm/cbsdk READMEs
new API and expanded
the new API
Compatibility Notes
name_qualifier, and instrument id). All in-repo callers, examples, and tests are updated.
Testing