Sobelow does not detect when secrets are in the test env

[Adzz]

In my runtime.exs file I have:

if config_env() == :test do
  config(:ex_aws, access_key_id: "dummy", secret_access_key: "dummy")
end

There is a check Sobelow.Config.Secrets which says:

  Sobelow detects missing hard-coded secrets by checking the prod
  configuration.

This check fails even though the env is :test. There is also no way to ignore it as far as I can tell? Putting this does not work:

if config_env() == :test do
  # sobelow_skip ["Config.Secrets"]
  config(:ex_aws, access_key_id: "dummy", secret_access_key: "dummy")
end

[houllette]

Hey @Adzz - sorry you're running into this issue. So that particular check only appears to exclude the config.exs and Sobelow in general has no concept of what mix environment the program is currently being run in (test vs. prod) since it is just statically reading and evaluating code. So as far as Sobelow is concerned, it is parsing the entire runtime.exs file (which is typically used in production) and detecting the fuzzy string search of "secret" and coming back with a finding.

The module documentation is definitely a little vague there by using the "prod" terminology and could probably stand to be improved.

Typically for # sobelow_skip to work, it has to be put on the outside of the offending function - truth be told I don't know off the top of my head how that changes for config files since they're a bit special, but its worth a shot putting the skip before the if statement while i dig into the codebase to see how config skips are handled.