Add pillar_mask_output config option and fix full-value pillar masking - #69812
Add pillar_mask_output config option and fix full-value pillar masking#69812Akm0d wants to merge 11 commits into
Conversation
VCOPS-98852: harden pillar output masking (VCOPS-77716, VCOPS-84671). - salt.utils.secret.serial() only redacted non-empty strings; truthy int/float/bool and non-empty bytes leaked through pillar.get() and related functions with their real value even though the repr path already redacted those types. Extracted a shared _is_redactable_scalar() predicate used by both _masked_repr() and serial() so the two can't drift apart again. - Added the pillar_mask_output master/minion config option (default True) as a global killswitch, seeded via salt.utils.secret.configure() from salt.pillar.get_pillar()/get_async_pillar(). When False, hide()/serial()/mask_output() no-op and pillar values are never wrapped or redacted. - Documented pillar_mask_output in doc/ref/configuration/master.rst. - Updated tests that encoded the old (buggy) passthrough behavior and added coverage for bytes redaction and the new config toggle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per review feedback on PR saltstack#69812: the module-level _ENABLED flag + configure() seeded from get_pillar() was a one-off pattern not used anywhere else in the codebase. pillar_merge_lists/pillar_safe_render_error are both read inline via self.opts.get(...)/__opts__.get(...) at each call site, with no caching. Replaced with an explicit enabled= parameter on hide()/serial()/ mask_output()/no_log_mask(), with every call site (salt/pillar/__init__.py, salt/modules/pillar.py, salt/client/ssh/wrapper/pillar.py, salt/state.py, salt/output/__init__.py) passing its own opts.get("pillar_mask_output", True) — matching the existing pillar boolean-option pattern exactly, no shared/global state left in salt.utils.secret. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per maintainer feedback on PR saltstack#69812: "Don't disable masking wholesale .. The config option should only change the default for pillar.items." Reverted the enabled= parameter and every call site outside pillar.items() (hide()/serial()/mask_output()/no_log_mask() in salt/utils/secret.py are back to their original signatures; salt/pillar/__init__.py, salt/client/ssh/wrapper/pillar.py, salt/state.py, salt/output/__init__.py are unchanged). pillar_mask_output now only affects the unmask-default computation inside salt.modules.pillar.items() — pillar.get/item/raw/ext, no_log state output, and the general CLI output safety net keep masking by default regardless of this option. Callers can still always override via pillar.items(unmask=True/False) explicitly. Updated config/doc/changelog wording and tests to match the narrower scope (added test_items_respects_pillar_mask_output_config_option and test_pillar_get_ignores_pillar_mask_output_config_option). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
You got some failing tests |
|
I submitted duplicate PR for same pillar mask global override, closed it now |
Real CI on PR saltstack#69812 surfaced that redact_state_ret_secrets()'s _collect_secret_literals() has no cycle-detection guard, unlike its siblings serial()/mask_output()/expose() in the same file. In the state.orchestrate/runner path, self.opts.get("pillar") is an OptsDict/ListProxy with a self-referential __iter__, so the recursive walker crashes with RecursionError - state.orchestrate fails outright in ~15 tests, not just degraded output. Separately, and even setting the crash aside, the literal-substring scan has no way to distinguish an actual secret from an ordinary pillar value used as a template parameter. Confirmed at real scale: the words "branch" and "master" - core Salt/git vocabulary - are present somewhere in the CI harness's minion pillar and got redacted out of unrelated assertions, breaking tests/integration/states/test_git.py (6 tests, every OS in the matrix) plus several salt-ssh suites. ~30 distinct upstream tests failed across ~9 platforms. Removes redact_state_ret_secrets()/redact_known_secrets()/ _collect_secret_literals() and the call site in salt/state.py. Reverts the three test files that were patched only to accommodate this mechanism's fallout (test_state_test.py, test_salt_call.py's test_42116_cli_pillar_override, test_file.py's test_state_skip_req) back to their original assertions. Keeps the unrelated unmask=True fixes (a real, independent pre-existing bug) and gap #2 (name masking under no_log), neither of which caused any CI failures.
Explicit unmask=False wins — the output stays masked (**********), regardless of pillar_mask_output: False in the minion config. Here's why, in salt/modules/pillar.py's items(): The pillar_mask_output config check only runs inside if unmask is None: — i.e., it only supplies a default when the caller doesn't pass unmask at all. Since your example passes unmask=False explicitly, that whole branch is skipped, and execution goes straight to else: return serial(ret) — redacted output. So the config option's only effect is on the default behavior of a bare pillar.items call with no unmask kwarg. An explicit unmask=False (or unmask=True) always overrides it in either direction — pillar_mask_output can't force masking back on if someone explicitly asks for unmask=True, and it can't force unmasking if someone explicitly asks for unmask=False. |
VCOPS-98852: harden pillar output masking (VCOPS-77716, VCOPS-84671).
What does this PR do?
What issues does this PR fix or reference?
Fixes
Previous Behavior
Remove this section if not relevant
New Behavior
Remove this section if not relevant
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes/No