Skip to content

[BUGFIX] Handle non-UTF-8 output from subprocess tools - #298

Merged
treee111 merged 1 commit into
treee111:developfrom
Mirochill:fix-271-subprocess-decode-errors
Aug 20, 2026
Merged

[BUGFIX] Handle non-UTF-8 output from subprocess tools#298
treee111 merged 1 commit into
treee111:developfrom
Mirochill:fix-271-subprocess-decode-errors

Conversation

@Mirochill

Copy link
Copy Markdown
Contributor

This PR...

  • Avoids UnicodeDecodeError when external map tooling writes non-UTF-8 bytes to stdout or stderr.

Considerations and implementations

run_subprocess_and_log_output() keeps decoding subprocess output as UTF-8, but now passes errors="backslashreplace" to subprocess.run(). This prevents bytes such as 0xa0 from raising while still making undecodable bytes visible in the captured log output.

Closes #271

How to test

  1. git diff --check
  2. git show --check --stat --oneline HEAD
  3. Not run locally

Pull Request Checklist

@treee111

Copy link
Copy Markdown
Owner

Local test result

Tested PR #298 in isolation on branch develop using the gdal-dev Anaconda environment.

Test run: python -m unittest tests.test_osm_maps -v

test_version_and_tags_of_country_config_file ... ok
test_encoding_open_sea_osm ... ok
test_folder_name_many_countries ... ok
test_input_country_malta ... ok
test_calc_border_countries_input_country ... FAIL  ← pre-existing, unrelated to this PR
test_calc_border_countries_input_xy_coordinates_1tile ... ok
test_calc_border_countries_input_xy_coordinates_2tiles ... ok
test_calc_without_border_countries_input_country ... ok
test_calc_without_border_countries_xy_coordinates_1tile ... ok
test_calc_without_border_countries_xy_coordinates_2tiles ... ok

Ran 10 tests in 78.591s

9/10 pass. The one failure (test_calc_border_countries_input_country) is pre-existing and unrelated to this PR — it fails on the unmodified baseline too.

The fix (errors="backslashreplace" on both subprocess.run call sites) is correct. It prevents UnicodeDecodeError when external tools like Osmosis emit non-UTF-8 bytes — instead of crashing, invalid bytes are escaped in the log output. No existing tests cover this code path directly, but the change is a safe no-op for well-behaved UTF-8 output and only activates on malformed bytes.

@treee111 treee111 changed the title [FIX] Handle non-UTF-8 output from subprocess tools [BUGFIX] Handle non-UTF-8 output from subprocess tools Aug 20, 2026
@treee111
treee111 marked this pull request as ready for review August 20, 2026 12:49
@treee111
treee111 force-pushed the fix-271-subprocess-decode-errors branch from bcdc2bd to c2fd4f1 Compare August 20, 2026 12:50

@treee111 treee111 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the PR @Mirochill !

@treee111
treee111 merged commit 830ecdf into treee111:develop Aug 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 17: invalid start byte

2 participants