Lazy config evaluation

[HamiltonRepoMigrationBot]

Issue by elijahbenizzy
Monday Feb 07, 2022 at 03:51 GMT
Originally opened as stitchfix/hamilton#64

---

[Short description explaining the high-level reason for the pull request]

Additions

Removals

Changes

Testing

Screenshots

If applicable

Notes

Todos

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows the standards laid out in the TODO link to standards
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist

Python

• python 3.6
• python 3.7

---

elijahbenizzy included the following code: https://github.com/stitchfix/hamilton/pull/64/commits

[HamiltonRepoMigrationBot]

Comment by elijahbenizzy
Monday Feb 07, 2022 at 03:55 GMT

---

Need to add tests/figure out what to test. Hopefully this demonstrates the issue. IMO chainmap is actually better here -- supporting a lazyconfig is reasonable. That said, I'd prefer the simplicity with a dictionary.

[HamiltonRepoMigrationBot]

Comment by elijahbenizzy
Monday Feb 07, 2022 at 03:58 GMT

---

Pros of this:

1. It allows lazily evaluated configs
2. Its idiomatic -- types are only as constricted as they need to be (E.G. why does config have to be a dict?)
3. It solves the problem a customer has

Cons of this:

1. We can do less with the config/inputs - E.G. if we want to store them somewhere or print them out we'll be running up against the same problem.
2. Added complexity for a specific use-case

[elijahbenizzy]

I don't think this is something we particularly want to handle now... Configs shouldn't be particularly complicated -- anything too complex should fit in the DAG itself.

If there's a complex process that determines how to shape it, we can figure that out. In the case o f the original motivation for this problem, we ended up just loading everything in parallel upfront.