Skip to content

generate/hcl: support module calls through source keyword#130

Open
tormath1 wants to merge 9 commits into
cycloidio:masterfrom
tormath1:mt-56-support-remote-sources
Open

generate/hcl: support module calls through source keyword#130
tormath1 wants to merge 9 commits into
cycloidio:masterfrom
tormath1:mt-56-support-remote-sources

Conversation

@tormath1

Copy link
Copy Markdown
Contributor

In this PR, we add the support for module calls. It supports local, remote and Terraform registry ones.

The function moduleInstall is called recursively in order to walk through the ModuleCalls of each module to feed a ManagedResource slice. It stores a trace of each installed element to avoid infinite loops and modules are cached into $XDG_CACHE directory.

Then it loops over the ManagedResource slice with the current behavior.

A few notes:

  1. I'd like to use in-memory FS rather than actual FS but go-getter needs to be updated consequently (Abstract the os access via interface hashicorp/go-getter#83: I started to work on it btw)
  2. I still need to find a way to properly testing this :)

Output example using this stack: https://github.com/cycloid-community-catalog/stack-gke

$ ./inframap generate --hcl generate/testdata/stack-gke
strict digraph G {
        "google_container_node_pool.pools"->"google_container_cluster.primary";
        "null_resource.delete_default_kube_dns_configmap"->"google_container_cluster.primary";
        "null_resource.delete_default_kube_dns_configmap"->"google_container_node_pool.pools";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"google_container_node_pool.pools";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"google_container_node_pool.pools";
        "kubernetes_config_map.ip-masq-agent"->"google_container_cluster.primary";
        "kubernetes_config_map.ip-masq-agent"->"google_container_node_pool.pools";
        "null_resource.wait_for_cluster"->"google_container_cluster.primary";
        "null_resource.wait_for_cluster"->"google_container_node_pool.pools";
        "google_service_account.cluster_service_account"->"random_string.cluster_service_account_suffix";
        "kubernetes_config_map.kube-dns"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns"->"google_container_node_pool.pools";
        "google_container_cluster.primary" [ shape=ellipse ];
        "google_container_node_pool.pools" [ shape=ellipse ];
        "google_service_account.cluster_service_account" [ shape=ellipse ];
        "kubernetes_config_map.ip-masq-agent" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns-upstream-namservers" [ shape=ellipse ];
        "null_resource.delete_default_kube_dns_configmap" [ shape=ellipse ];
        "null_resource.wait_for_cluster" [ shape=ellipse ];
        "random_string.cluster_service_account_suffix" [ shape=ellipse ];

}

Closes: #56, #54

@xescugc xescugc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you also still working on adding the afero.FS implemtation into the TF lib?

Comment thread generate/hcl.go
Comment on lines +63 to +66
managedResources := make(map[string]*configs.Resource)
for rk, rv := range mod.ManagedResources {
managedResources[rk] = rv
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing we are doing right now on the State is check if we have to prefix a resource with the module because the same resource could exist on another module aws_instance.front could be in multiple modules.

We only do that when we have more than 1 module we should try to do the same as we'll find the same issue.

Comment thread generate/hcl.go
Comment on lines +353 to +368
for _, vers := range meta.Versions {
v, err := version.NewVersion(vers.Version)
if err != nil {
return fmt.Errorf("unable to create version from string: %w", err)
}

if latest == nil || v.GreaterThan(latest) {
latest = v
}

if call.Version.Required.Check(v) {
if match == nil || v.GreaterThan(match) {
match = v
}
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment a bit this logic.

Comment thread generate/hcl.go
Comment thread generate/hcl.go Outdated
Comment on lines +399 to +408
dst := path.Join(cachePath, name)
// TODO: we should add a logic to invalidate
// the cache
if _, err := os.Stat(dst); os.IsNotExist(err) {
client := &getter.Client{
Src: src,
Dst: dst,
Pwd: pwd,
Mode: getter.ClientModeDir,
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we have versions, we can assume versions are immutable, so you could use a dst := path.Join(cachePath, fmt.Sprintf("%s-%s", name, version))

Comment thread generate/hcl.go
Comment on lines +421 to +426
// fill the final map of managed resources
// using the config freshly loaded
for rk, rv := range m.ManagedResources {
(*mRes)[rk] = rv
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially prefix it with them module if more than one or we'll have collision.

Comment thread cmd/root.go
Comment on lines +26 to +31
PersistentPreRun: func(cmd *cobra.Command, args []string) {
if debug {
logsOut = os.Stdout
}
log.Init(logsOut, debug)
},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer not to include TC for this, copy the part you need (activation of TF logs) and open an issue to add Debugging to IM (if we do not have one already)

@xescugc

xescugc commented Sep 2, 2021

Copy link
Copy Markdown
Member

Ping @Thomas-lhuillier hehe.

@Thomas-lhuillier

Copy link
Copy Markdown

What do I have to do? 😅

@tormath1

tormath1 commented Sep 2, 2021

Copy link
Copy Markdown
Contributor Author

@Thomas-lhuillier I guess @xescugc wanted to ping me instead - but feel free to work on this PR if you want to. 🙈

Anyways, thanks for the ping ! Besides the comments to address, I guess there is not so much effort to provide to get this PR land into a new release.

Are you also still working on adding the afero.FS implemtation into the TF lib?

Nope. It was quite a mess to bring afero.FS support to the lib IIRC.

@Thomas-lhuillier

Copy link
Copy Markdown

@Thomas-lhuillier I guess @xescugc wanted to ping me instead - but feel free to work on this PR if you want to. 🙈

Makes sense 👍 Well, this all seems out of reach to me, maybe after a couple months/years training :trollface:

@xescugc
xescugc force-pushed the master branch 2 times, most recently from c0bd35f to d9632dd Compare September 3, 2021 07:54
@xescugc

xescugc commented Sep 6, 2021

Copy link
Copy Markdown
Member

Yes I think it was an autocomplete error from GH and I did not check haha the ping is for @tormath1 (ok if i do @th the first one is Thomas haha)

@tormath1 tormath1 mentioned this pull request Oct 11, 2022
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hcl: support module sources

3 participants