Skip to content

Clean up procedure for ptf based#301

Closed
SkayKvit wants to merge 2 commits into
opencomputeproject:mainfrom
SkayKvit:clean_up_procedure_for_ptf_based
Closed

Clean up procedure for ptf based#301
SkayKvit wants to merge 2 commits into
opencomputeproject:mainfrom
SkayKvit:clean_up_procedure_for_ptf_based

Conversation

@SkayKvit

@SkayKvit SkayKvit commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

To minimize overhead for successful tests, the system implements a dedicated recovery mechanism that activates only after a test failure or class change. This prevents heavy NPU cleanup operations when the hardware state is valid, while resolving previous issues caused by misconfigured scope="class" fixtures, which frequently left "dirty" states for subsequent test runs.
The system relies on the following key components:

  1. **TopologyManager**: Manages the topology lifecycle and mocks critical npu methods with safe stubs, preventing framework crashes during hardware errors.
    
  2. **safe_execute**: Intercepts exceptions during NPU interactions, returning null objects instead of crashing the RPC queue.
    
  3. **verify_sai_clean_state**: Serves as a diagnostic helper that inspects the NPU's internal tables (such as FDB or VLAN members) to confirm the hardware is truly empty, acting as a final safeguard before proceeding with a new test suite.
    
  4. **sai_ptf_topology_manager**: A class-scoped fixture that caches the topology for all tests within a class, optimizing execution speed.
    
  5. **sai_ptf_topology**: A function-scoped fixture that performs "smart" recovery: it resets the NPU and flushes Redis queues only when a failure is detected or a class mismatch occurs, correcting issues with improper scope contexts.
    
  6. **setup_teardown**: Automates hardware configuration, synchronizing the NPU state before every test and ensuring a predictable, clean environment.
    

Oleksandr Nyzkoshapka added 2 commits July 9, 2026 14:53
Comment thread tests/conftest.py
Comment on lines +89 to +92
@pytest.fixture(scope="module", autouse=True)
def skip_all(testbed_instance):
testbed = testbed_instance
if testbed is not None and len(testbed.npu) != 1:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this fixture shouldn't be in conftest.py because it is only applicable for testbeds with npu. Testbed can have phy or dpu instead

Comment thread tests/conftest.py
except BaseException:
pass

if prev_test_failed and 'verify_sai_clean_state' in globals():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this?

Comment thread tests/conftest.py
Helper function to verify that the ASIC has no leftover user objects.
"""
try:
fdb_keys = npu.get_db_keys("ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY*")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For npu it doesn't matter which sai client it uses: Redis or Thrift. It can't use data directly from Redis db like "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY*" here

@zikkend

zikkend commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

We can't simply mock the npu create or remove methods because if they fail, there is usually a legitimate reason. It's typically either a bug in the SAI-Challenger itself during topology creation, the test case setup/teardown, or the SAI implementation

@SkayKvit SkayKvit closed this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants