Upgrade pyquarkchain from Python 3.8 to Python 3.13#969
Draft
Upgrade pyquarkchain from Python 3.8 to Python 3.13#969
Conversation
- adjust_difficulty.py: replace bare `import monitoring` with `from quarkchain.tools import monitoring` (breaks when run from outside the tools directory) - adjust_difficulty.py: replace jsonrpc_async.Server with AsyncJsonRpcClient and use .call() method - monitoring.py: replace jsonrpc_async.Server with AsyncJsonRpcClient and use .call() method; close session via .close() - jsonrpc_client.py: fix AsyncJsonRpcClient.call signature to use *params (variadic) to match JsonRpcClient.call, fixing callers that pass positional arguments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix send_request in test_jsonrpc.py to unpack list params correctly instead of double-wrapping them (e.g. [["0x..."]] → ["0x..."]) - Add call_with_dict_params to AsyncJsonRpcClient for named params - Implement JSONRPCWebsocketServer.shutdown() to actually close the server, fixing test isolation hangs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track and cancel all fire-and-forget asyncio tasks that were leaking across tests, causing resource exhaustion and timeout failures. - AbstractConnection: add _loop_task and _handler_tasks tracking; use try/finally in active_and_loop_forever to ensure cleanup on cancellation; cancel _loop_task in close() - master.py: track SlaveConnection loop task and __init_cluster task; cancel _init_task on shutdown - slave.py: track MasterConnection, SlaveConnection, PeerShardConnection loop tasks and __start_server task - shard.py: track PeerShardConnection loop task - miner.py: track and cancel mining task in disable() - simple_network.py: track Peer loop task and connect_seed task - test_utils.py: restructure shutdown_clusters with try/finally to guarantee task cleanup; await server.wait_closed() for slave servers - conftest.py: multi-round task cancellation; reset aborted_rpc_count
- Fix AttributeError: replace undefined self.external_port/self.internal_port/self.protocol with self.port - Fix _discover timeout: wrap async_search with asyncio.wait_for instead of waiting after it completes - Fix _run: guard _add_port_mapping with self._service check to avoid AttributeError - Fix _delete_port_mapping: delete both TCP and UDP mappings to match AddPortMapping - Remove dead code: _refresh_task and _running fields that were never used
- convert test_cluster.py and test_jsonrpc.py to unittest.IsolatedAsyncioTestCase - make create_test_clusters/shutdown_clusters async, ClusterContext async context manager - replace call_async() with await, assert_true_with_timeout with async_assert_true_with_timeout - replace _get_or_create_event_loop() with asyncio.get_running_loop() in master/slave - fix mock_pay_native_token_as_gas to support both sync and async wrapped functions - remove obsolete _get_or_create_event_loop, call_async, assert_true_with_timeout helpers - fix conftest to restore event loop after IsolatedAsyncioTestCase closes it
replace with Any type hint to avoid deprecation warning in websockets 14+
This reverts commit b4e190d.
Migrate test classes to IsolatedAsyncioTestCase so that asyncio.create_task calls in production code (shard_state, root_state) have a running event loop. - Replace asyncio.ensure_future with asyncio.create_task in production code - Replace _get_or_create_event_loop with asyncio.get_running_loop in master/slave - Convert ClusterContext to async context manager - Convert create_test_clusters / shutdown_clusters to async - Add fire_and_forget and async_assert_true_with_timeout utilities - Simplify conftest fixture to only restore event loop after IsolatedAsyncioTestCase teardown
…sues - Rename jsonrpcserver.py to jsonrpc_server.py for consistent naming - Replace armor with asyncio.shield for cancellation protection - Fix get_running_loop to get_event_loop for compatibility - Fix subscription.py import from old jsonrpcserver package - Fix indentation in test_jsonrpc.py and method name in stats.py
- Fix snake_case field names to camelCase for JSON-RPC responses (network_id->networkId, shard_size->chainSize, block_height->blockHeight, contract_address->contractAddress) - Fix resp.data.result access pattern to direct dict access - Add exception handling and cli.close() to prevent connection leaks - Add 0x prefix for address in balance_watcher query_balance
…umpy ethash_cy.pyx: typed C loop replacing the 256-iteration FNV parent mixing in calc_dataset_item. ethash.py auto-imports when built, falls back to pure Python otherwise. bench_hashimoto_compare.py extended with R3 column.
- ethash.py: rewrite with numpy uint32 arrays (R2); add ETHASH_LIB env var to select python/cython/auto at runtime - ethash_cy.pyx: add mix_parents (R3), cy_calc_dataset_item and cy_hashimoto_light with C keccak (R4) - keccak_tiny.c/h: portable C Keccak implementation for Cython R4 - ethpow.py: use ETHASH_LIB-aware hashimoto_light; simplify check_pow/mine - setup.py: build Cython extension with keccak_tiny.c - old_ethash.py: extract original hex-based implementation as reference baseline - bench_hashimoto_compare.py: merge bench_before_after.py; add R3/R4 sections; import old impl from old_ethash.py - test_ethash.py: use old_ethash as baseline for cython correctness test - remove bench_before_after.py
- Add ethereum/pow/ethash_rs: full Rust implementation of ethash (mkcache, hashimoto_light, mix_parents) using PyO3 0.22 + tiny-keccak - Integrate setuptools-rust into setup.py so build_ext --inplace places ethash_rs.so in the source tree alongside ethash_cy.so - Refactor ethash.py: ETHASH_LIB branches only import functions; _get_cache and hashimoto_light defined once with None-check fallback - Auto-detection order: ethash_rs -> ethash_cy -> pure Python - Update Dockerfile to install Rust toolchain and build both extensions - Update README and requirements.txt for Rust/maturin/setuptools-rust - Add test_rust_matches_python_fallback to verify Rust output
…andling - Override _cleanup() so stop() is called on service shutdown, preventing port mapping and aiohttp session leaks when the node exits - Close any existing session before creating a new one in discover() to avoid orphaned sessions on repeated calls - Handle SOAP error 718 (ConflictInMappingEntry) as success, matching the previous best-effort behavior from master - Roll back partial port mappings if AddPortMapping fails mid-loop - Import UpnpActionResponseError for typed SOAP error handling
- bench_hashimoto_compare.py: add R5 section covering rs_mkcache, rs_calc_dataset_item, and rs_hashimoto_light; skipped gracefully when the Rust extension is not available (same pattern as R3/R4) - ethash.py: fix auto-detect to check for the expected symbol (rs_hashimoto_light / cy_hashimoto_light) after import, preventing the ethash_rs/ Cargo source directory (a namespace package) from being mistaken for a built extension
Add _discover_lock so that concurrent callers cannot race on _session and _service state. The lock wraps the entire discover() body, ensuring each call fully completes (including session cleanup in the finally block) before the next begins.
… fix bool type, deduplicate - Guard on_response with early return if _service already found (SSDP sends one response per device sub-type, so the callback fires ~9x) - Extract location from CaseInsensitiveDict headers (newer async_upnp_client passes headers dict, not a response object) - Recursively traverse embedded_devices to find WANIPConnection service (root device only exposes Layer3Forwarding; WANIPConnection lives in the InternetGatewayDevice → WANDevice → WANConnectionDevice subtree) - Pass NewEnabled=True (bool) instead of 1; async_upnp_client validates the UPnP boolean type strictly
Use asyncio.Event + wait_for so _discover returns as soon as the first WANIPConn service is found instead of waiting the full 30-second timeout. Add post-await re-check to guard against concurrent on_response tasks that all pass the initial self._service guard before any of them completes.
Connect the UDP socket to the UPnP router's IP (parsed from the SSDP
location URL and stored in _router_host) instead of 8.8.8.8, so the OS
routing table selects the interface that actually reaches the router.
Matches the behaviour of the old netifaces-based
find_internal_ip_on_device_network() without the extra dependency.
Tests updated accordingly:
- test_get_internal_ip sets _router_host and asserts connect target
- Add test_get_internal_ip_no_router_host: returns None before discover()
- Add test_add_port_mapping_no_internal_ip: raises RuntimeError when
_router_host is None
- Fix fake_search to pass dict response so response.get('location') works
- Set fake_device.embedded_devices={} to prevent MagicMock recursion
- Move MOCK_ROUTER_HOST to top constants with clarifying comments
- Remove trivial guard tests (no_service, already_none, exits_on_cancel)
…(10) - Replace global np.seterr(over='ignore') with np.errstate(over='ignore') scoped to the two FNV multiply sites in calc_dataset_item and hashimoto, avoiding unintended suppression of overflow warnings elsewhere - Use dtype='<u4' (explicit little-endian) in ethash_sha3_512/256 instead of native-endian np.uint32, matching Ethereum spec on big-endian hosts - Restore lru_cache(10): 8 shards can span ~6-7 different epochs simultaneously, so 2-3 slots would cause frequent cache eviction - test_cython_matches_python_fallback and test_rust_matches_python_fallback now fail on ImportError instead of skipping
Install C build tools and Rust toolchain in CI (the test container purges them to reduce image size), then run setup.py build_ext --inplace before pytest so test_cython_matches_python_fallback and test_rust_matches_python_fallback do not fail on ImportError.
This reverts commit 9124e8b.
qkchash_llrb.h: remove <T> from move constructor name — template-id on constructors is ill-formed in C++20 (was a warning with -std=c++17) Dockerfile: remove post-build cleanup of gcc/Rust/build-essential so the test image retains the toolchain needed to rebuild extensions in CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dependencies (requirements.txt)
asyncio API migration
JSON-RPC library replacement
Python 3.12+ removed APIs
p2p module
Test stability
Code formatting
Test plan