Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Terraform 0.12 #51

Closed
ivankorn opened this issue Aug 16, 2019 · 15 comments · Fixed by #52
Closed

Add support for Terraform 0.12 #51

ivankorn opened this issue Aug 16, 2019 · 15 comments · Fixed by #52
Assignees
Labels
enhancement New feature or request

Comments

@ivankorn
Copy link
Contributor

ivankorn commented Aug 16, 2019

Please, note:
0.12 support for managed_instance_group example depends on this.
Module to be updated once this task is done.
Linked:

@ivankorn
Copy link
Contributor Author

Fixed by #49

@ivankorn
Copy link
Contributor Author

I'm moving discussion from closed PR #49 here

@itscaro, thank you for doing a good job on migrating examples!
Unfortunately your PR for it uses deprecated MIG template, so terraform plan discovers a lot of errors like this

$ terraform plan

Error: Invalid value for module argument

  on main.tf line 65, in module "gce-lb-http":
  65:   target_tags       = [module.mig1.target_tags, module.mig2.target_tags]

The given value is not suitable for child module variable "target_tags"
defined at ../../variables.tf:48,1-23: element 0: string required.


Error: Unsupported block type

  on mig.tf line 20, in data "template_file" "group-startup-script":
  20:   vars {

Blocks of type "vars" are not expected here. Did you mean to define argument
"vars"? If so, use the equals sign to assign it a value.

As @morgante advised here the older MIG module is deprecated and we should use https://github.com/terraform-google-modules/terraform-google-vm/tree/master/modules/mig instead.

@itscaro, unless you're willing to continue your work I'm going to pick up from where you left and proceed with following:

  • Switch MIG and other depdency-modules to TF 0.12 compatible
  • Add strict types for variables
  • Remove instances of unnecessary string interpolation from the code base (if any left)
  • Remove unnecessary "element" calls (if any left)
  • Update types to number/bool where necessary

Also I'd like to alight this module a bit with terraform-google-modules

By adding terraform, bash and white-space linters + doc_generator as a first step in this direction.

@itscaro
Copy link
Contributor

itscaro commented Aug 20, 2019

@ivankorn please go ahead, I'm going to be off till Sep. Thank you.

@ivankorn
Copy link
Contributor Author

ivankorn commented Aug 20, 2019

@ivankorn please go ahead, I'm going to be off till Sep. Thank you.
Okay, then :) have a good time @itscaro

@ivankorn ivankorn self-assigned this Aug 21, 2019
ivankorn added a commit to ivankorn/terraform-google-lb-http that referenced this issue Aug 22, 2019
- 'examples/basic' migrated and provisioned

```
$ terraform apply -auto-approve
module.mig1_template.data.google_compute_image.image_family: Refreshing state...
module.mig2.data.google_compute_zones.available: Refreshing state...
module.mig2_template.data.google_compute_image.image_family: Refreshing state...
module.mig1.data.google_compute_zones.available: Refreshing state...
module.mig2_template.data.google_compute_image.image: Refreshing state...
module.mig1_template.data.google_compute_image.image: Refreshing state...
data.template_file.group-startup-script: Refreshing state...
google_compute_network.default: Creating...
module.gce-lb-http.google_compute_http_health_check.default[0]: Creating...
module.gce-lb-http.google_compute_global_address.default: Creating...
module.gce-lb-http.google_compute_http_health_check.default[0]: Creation complete after 4s [id=mig-basic-lb-backend-0]
module.gce-lb-http.google_compute_global_address.default: Creation complete after 4s [id=mig-basic-lb-address]
google_compute_network.default: Still creating... [10s elapsed]
google_compute_network.default: Still creating... [20s elapsed]
google_compute_network.default: Creation complete after 28s [id=mig-basic]
google_compute_subnetwork.group1: Creating...
google_compute_subnetwork.group2: Creating...
google_compute_subnetwork.group1: Still creating... [10s elapsed]
google_compute_subnetwork.group2: Still creating... [10s elapsed]
google_compute_subnetwork.group2: Creation complete after 17s [id=us-east1/mig-basic-group2]
module.mig2_template.google_compute_instance_template.tpl: Creating...
google_compute_subnetwork.group1: Still creating... [20s elapsed]
module.mig2_template.google_compute_instance_template.tpl: Creation complete after 5s [id=mig-basic-group2-20190822225210380000000001]
module.mig2.google_compute_region_instance_group_manager.mig: Creating...
google_compute_subnetwork.group1: Creation complete after 28s [id=us-west1/mig-basic-group1]
google_compute_subnetwork.lb: Creating...
module.mig1_template.google_compute_instance_template.tpl: Creating...
module.mig2.google_compute_region_instance_group_manager.mig: Still creating... [10s elapsed]
module.mig1_template.google_compute_instance_template.tpl: Creation complete after 5s [id=mig-basic-group1-20190822225221497600000002]
module.gce-lb-http.google_compute_firewall.default-hc[0]: Creating...
module.mig1.google_compute_region_instance_group_manager.mig: Creating...
google_compute_subnetwork.lb: Still creating... [10s elapsed]
module.gce-lb-http.google_compute_firewall.default-hc[0]: Creation complete after 8s [id=mig-basic-lb-hc-0]
module.mig2.google_compute_region_instance_group_manager.mig: Still creating... [20s elapsed]
module.mig1.google_compute_region_instance_group_manager.mig: Still creating... [10s elapsed]
google_compute_subnetwork.lb: Still creating... [20s elapsed]
module.mig1.google_compute_region_instance_group_manager.mig: Creation complete after 18s [id=gl-ivankorniienko-seed/us-west1/mig-basic-group1-mig]
module.mig2.google_compute_region_instance_group_manager.mig: Creation complete after 29s [id=gl-ivankorniienko-seed/us-east1/mig-basic-group2-mig]
module.gce-lb-http.google_compute_backend_service.default[0]: Creating...
google_compute_subnetwork.lb: Creation complete after 29s [id=us-west1/mig-basic-lb]
module.gce-lb-http.google_compute_backend_service.default[0]: Creation complete after 9s [id=mig-basic-lb-backend-0]
module.gce-lb-http.google_compute_url_map.default[0]: Creating...
module.gce-lb-http.google_compute_url_map.default[0]: Creation complete after 4s [id=mig-basic-lb-url-map]
module.gce-lb-http.google_compute_target_http_proxy.default[0]: Creating...
module.gce-lb-http.google_compute_target_http_proxy.default[0]: Creation complete after 4s [id=mig-basic-lb-http-proxy]
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Creating...
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Still creating... [10s elapsed]
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Creation complete after 18s [id=mig-basic-lb]

Apply complete! Resources: 15 added, 0 changed, 0 destroyed.

Outputs:

load-balancer-ip = 35.244.166.214
```
@jtfogarty
Copy link

@ivankorn I've been following your work on this. I'm looking for this internal lb example
are you working on this? if not, are the items you mentioned above needed to convert this to TF 0.12? Thanks

@morgante
Copy link
Contributor

@jtfogarty We're hoping to update all the LB modules to v0.12 in the next week or two.

@ivankorn
Copy link
Contributor Author

@ivankorn I've been following your work on this. I'm looking for this internal lb example
are you working on this?

@jtfogarty, not at the moment. But given the update Morgante provided - there are chances for me to pick up some more of lb modules soon.

if not, are the items you mentioned above needed to convert this to TF 0.12? Thanks

More than that, I think in addition to those, we should take care of following:

  • Add linters and updated helper scripts
  • Standardize tests
  • Switch to new CI

Just to satisfy immediate functionality needs last two points can probably be addressed out of migration scope.
All the TF modules will likely follow the terraform-google-modules new approach

ivankorn added a commit to ivankorn/terraform-google-lb-http that referenced this issue Sep 2, 2019
….12 syntax, interpolation, dynamic blocks, etc):

- examples/basic -> examples/multi-mig-http-lb
- examples/http-nat-gateway -> examples/mig-nat-http-lb
- examples/https-content -> examples/multi-backend-multi-mig-bucket-https-lb
- examples/multiple-certs (similar to above, but with multiple certs)

ref: terraform-google-modules#51
@Dev25
Copy link
Contributor

Dev25 commented Sep 3, 2019

As part of this 0.12 move is it worth moving straight to using for_each (0.12.6+) over the count/index method for managing resources.

https://www.terraform.io/docs/configuration/resources.html#for_each-multiple-resource-instances-defined-by-a-map-or-set-of-strings

I know other modules are looking to move to using for_each now that is it released e.g. GKE with node-pools, in this module it can be used for backend_services/health_checks

@morgante
Copy link
Contributor

morgante commented Sep 4, 2019

@Dev25 Yes, I think that's a good idea. @ivankorn Please switch to using for_each.

ivankorn added a commit to ivankorn/terraform-google-lb-http that referenced this issue Sep 6, 2019
ref: terraform-google-modules#51

```
[18:54][user@host:~/workspace/google/terraform-google-lb-http/examples/mig-nat-http-lb]$ terraform apply -auto-approve
module.mig.data.google_compute_zones.available: Refreshing state...
module.mig_template.data.google_compute_image.image: Refreshing state...
module.mig_template.data.google_compute_image.image_family: Refreshing state...
data.template_file.group-startup-script: Refreshing state...
module.cloud-nat.random_string.name_suffix: Creating...
module.cloud-nat.random_string.name_suffix: Creation complete after 0s [id=jb4pjk]
google_compute_network.default: Creating...
module.gce-lb-http.google_compute_global_address.default: Creating...
module.gce-lb-http.google_compute_health_check.default[0]: Creating...
module.gce-lb-http.google_compute_health_check.default[0]: Creation complete after 5s [id=mig-http-lb-http-80-6666cd76]
module.gce-lb-http.google_compute_global_address.default: Creation complete after 5s [id=mig-http-lb-address]
google_compute_network.default: Still creating... [10s elapsed]
google_compute_network.default: Still creating... [20s elapsed]
google_compute_network.default: Creation complete after 28s [id=tf-lb-http-mig-nat]
google_compute_router.default: Creating...
google_compute_subnetwork.default: Creating...
module.gce-lb-http.google_compute_firewall.default["tf-lb-http-mig-nat"]: Creating...
google_compute_router.default: Creation complete after 5s [id=us-west1/lb-http-router]
module.cloud-nat.google_compute_router_nat.main: Creating...
google_compute_subnetwork.default: Still creating... [10s elapsed]
module.gce-lb-http.google_compute_firewall.default["tf-lb-http-mig-nat"]: Still creating... [10s elapsed]
module.cloud-nat.google_compute_router_nat.main: Still creating... [10s elapsed]
google_compute_subnetwork.default: Still creating... [20s elapsed]
module.gce-lb-http.google_compute_firewall.default["tf-lb-http-mig-nat"]: Still creating... [20s elapsed]
module.cloud-nat.google_compute_router_nat.main: Creation complete after 17s [id=us-west1/lb-http-router/cloud-nat-lb-http-router]
module.gce-lb-http.google_compute_firewall.default["tf-lb-http-mig-nat"]: Creation complete after 28s [id=mig-http-lb-tf-lb-http-mig-nat-hc]
google_compute_subnetwork.default: Still creating... [30s elapsed]
google_compute_subnetwork.default: Creation complete after 39s [id=us-west1/tf-lb-http-mig-nat]
module.mig_template.google_compute_instance_template.tpl: Creating...
module.mig_template.google_compute_instance_template.tpl: Creation complete after 4s [id=tf-lb-http-mig-nat-20190906155713028900000001]
module.mig.google_compute_region_instance_group_manager.mig: Creating...
module.mig.google_compute_region_instance_group_manager.mig: Still creating... [10s elapsed]
module.mig.google_compute_region_instance_group_manager.mig: Still creating... [20s elapsed]
module.mig.google_compute_region_instance_group_manager.mig: Creation complete after 29s [id=gl-ivankorniienko-seed-251912/us-west1/tf-lb-http-mig-nat-mig]
module.gce-lb-http.google_compute_backend_service.default["0"]: Creating...
module.gce-lb-http.google_compute_backend_service.default["0"]: Creation complete after 9s [id=mig-http-lb-backend-0]
module.gce-lb-http.google_compute_url_map.default[0]: Creating...
module.gce-lb-http.google_compute_url_map.default[0]: Creation complete after 4s [id=mig-http-lb-url-map]
module.gce-lb-http.google_compute_target_http_proxy.default[0]: Creating...
module.gce-lb-http.google_compute_target_http_proxy.default[0]: Creation complete after 4s [id=mig-http-lb-http-proxy]
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Creating...
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Creation complete after 10s [id=mig-http-lb]

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Outputs:

load-balancer-ip = 35.190.121.34
[18:58][user@host:~/workspace/google/terraform-google-lb-http/examples/mig-nat-http-lb]$ less plat.output.txt
[19:05][user@host:~/workspace/google/terraform-google-lb-http/examples/mig-nat-http-lb]$ ./test.sh
+ set -e
++ terraform output load-balancer-ip
+ URL=http://35.190.121.34
+ status=0
+ count=0
+ [[ 0 -lt 720 ]]
+ [[ 0 -ne 200 ]]
+ echo 'INFO: Waiting for load balancer...'
INFO: Waiting for load balancer...
++ curl -sf -m 5 -o /dev/null -w '%{http_code}' http://35.190.121.34
+ status=200
+ (( count=count+1 ))
+ sleep 5
+ [[ 1 -lt 720 ]]
+ [[ 200 -ne 200 ]]
+ [[ 1 -lt 720 ]]
+ echo 'INFO: PASS'
INFO: PASS
```

```
[18:54][user@host:~/workspace/google/terraform-google-lb-http/examples/multi-mig-http-lb]$ terraform apply -auto-approve
module.mig2.data.google_compute_zones.available: Refreshing state...
module.mig2_template.data.google_compute_image.image_family: Refreshing state...
module.mig1_template.data.google_compute_image.image_family: Refreshing state...
module.mig2_template.data.google_compute_image.image: Refreshing state...
module.mig1_template.data.google_compute_image.image: Refreshing state...
module.mig1.data.google_compute_zones.available: Refreshing state...
data.template_file.group-startup-script: Refreshing state...
module.cloud-nat-group1.random_string.name_suffix: Creating...
module.cloud-nat-group2.random_string.name_suffix: Creating...
module.cloud-nat-group1.random_string.name_suffix: Creation complete after 0s [id=mds1om]
module.cloud-nat-group2.random_string.name_suffix: Creation complete after 0s [id=971f2q]
google_compute_network.default: Creating...
module.gce-lb-http.google_compute_global_address.default: Creating...
module.gce-lb-http.google_compute_health_check.default[0]: Creating...
module.gce-lb-http.google_compute_global_address.default: Creation complete after 5s [id=multi-mig-lb-http-address]
module.gce-lb-http.google_compute_health_check.default[0]: Creation complete after 5s [id=multi-mig-lb-http-http-80-6666cd76]
google_compute_network.default: Still creating... [10s elapsed]
google_compute_network.default: Creation complete after 17s [id=multi-mig-lb-http]
google_compute_router.group1: Creating...
google_compute_subnetwork.group2: Creating...
google_compute_subnetwork.group1: Creating...
google_compute_router.group2: Creating...
google_compute_router.group2: Creation complete after 4s [id=us-east1/multi-mig-lb-http-gw-group2]
module.cloud-nat-group2.google_compute_router_nat.main: Creating...
google_compute_router.group1: Creation complete after 4s [id=us-west1/multi-mig-lb-http-gw-group1]
module.cloud-nat-group1.google_compute_router_nat.main: Creating...
module.gce-lb-http.google_compute_firewall.default["multi-mig-lb-http"]: Creating...
google_compute_subnetwork.group2: Still creating... [10s elapsed]
google_compute_subnetwork.group1: Still creating... [10s elapsed]
module.cloud-nat-group2.google_compute_router_nat.main: Still creating... [10s elapsed]
module.cloud-nat-group1.google_compute_router_nat.main: Still creating... [10s elapsed]
module.gce-lb-http.google_compute_firewall.default["multi-mig-lb-http"]: Still creating... [10s elapsed]
google_compute_subnetwork.group2: Still creating... [20s elapsed]
google_compute_subnetwork.group1: Still creating... [20s elapsed]
module.gce-lb-http.google_compute_firewall.default["multi-mig-lb-http"]: Creation complete after 17s [id=multi-mig-lb-http-multi-mig-lb-http-hc]
module.cloud-nat-group2.google_compute_router_nat.main: Creation complete after 17s [id=us-east1/multi-mig-lb-http-gw-group2/multi-mig-lb-http-cloud-nat-group2]
module.cloud-nat-group1.google_compute_router_nat.main: Creation complete after 18s [id=us-west1/multi-mig-lb-http-gw-group1/multi-mig-lb-http-cloud-nat-group1]
google_compute_subnetwork.group2: Creation complete after 27s [id=us-east1/multi-mig-lb-http-group2]
module.mig2_template.google_compute_instance_template.tpl: Creating...
google_compute_subnetwork.group1: Creation complete after 27s [id=us-west1/multi-mig-lb-http-group1]
module.mig1_template.google_compute_instance_template.tpl: Creating...
module.mig2_template.google_compute_instance_template.tpl: Creation complete after 4s [id=multi-mig-lb-http-group2-20190906155604990200000001]
module.mig2.google_compute_region_instance_group_manager.mig: Creating...
module.mig1_template.google_compute_instance_template.tpl: Creation complete after 5s [id=multi-mig-lb-http-group1-20190906155605140900000002]
module.mig1.google_compute_region_instance_group_manager.mig: Creating...
module.mig2.google_compute_region_instance_group_manager.mig: Still creating... [10s elapsed]
module.mig1.google_compute_region_instance_group_manager.mig: Still creating... [10s elapsed]
module.mig2.google_compute_region_instance_group_manager.mig: Creation complete after 19s [id=gl-ivankorniienko-seed-251912/us-east1/multi-mig-lb-http-group2-mig]
module.mig1.google_compute_region_instance_group_manager.mig: Creation complete after 19s [id=gl-ivankorniienko-seed-251912/us-west1/multi-mig-lb-http-group1-mig]
module.gce-lb-http.google_compute_backend_service.default["0"]: Creating...
module.gce-lb-http.google_compute_backend_service.default["0"]: Still creating... [10s elapsed]
module.gce-lb-http.google_compute_backend_service.default["0"]: Creation complete after 17s [id=multi-mig-lb-http-backend-0]
module.gce-lb-http.google_compute_url_map.default[0]: Creating...
module.gce-lb-http.google_compute_url_map.default[0]: Creation complete after 4s [id=multi-mig-lb-http-url-map]
module.gce-lb-http.google_compute_target_http_proxy.default[0]: Creating...
module.gce-lb-http.google_compute_target_http_proxy.default[0]: Creation complete after 4s [id=multi-mig-lb-http-http-proxy]
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Creating...
module.gce-lb-http.google_compute_global_forwarding_rule.http[0]: Creation complete after 9s [id=multi-mig-lb-http]

Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

Outputs:

load-balancer-ip = 35.190.47.152
[18:57][user@host:~/workspace/google/terraform-google-lb-http/examples/multi-mig-http-lb]$ ./test.sh
+ set -e
++ terraform output load-balancer-ip
+ URL=http://35.190.47.152
+ status=0
+ count=0
+ [[ 0 -lt 720 ]]
+ [[ 0 -ne 200 ]]
+ echo 'INFO: Waiting for load balancer...'
INFO: Waiting for load balancer...
++ curl -sf -m 5 -o /dev/null -w '%{http_code}' http://35.190.47.152
+ status=200
+ (( count=count+1 ))
+ sleep 5
+ [[ 1 -lt 720 ]]
+ [[ 200 -ne 200 ]]
+ [[ 1 -lt 720 ]]
+ echo 'INFO: PASS'
INFO: PASS
[19:14][user@host:~/workspace/google/terraform-google-lb-http/examples/multi-mig-http-lb]$
```
@ivankorn
Copy link
Contributor Author

ivankorn commented Sep 6, 2019

@Dev25 Yes, I think that's a good idea. @ivankorn Please switch to using for_each.

for_each switch is commited here

For now interface is kept everywhere but here. That can be easily reverted, but I don't see any sense in keeping at list that. I'm also going to come up with a proposal on more changes as discussed with @aaron-lane on the call today
Since the PR is still in draft I left some comments here and there on interface changes I'd like to make. Followed by migration plan in readme/changelog of course

@ivankorn
Copy link
Contributor Author

ivankorn commented Sep 9, 2019

@morgante @Dev25 fyi: for_each adds a bit to a good amount of entropy on the migration task.
We're migrating

  • from 0.11 to 0.12
  • from GoogleCloudPlatform/managed-instance-group to terraform-google-modules/vm/google//modules/instance_template and terraform-google-modules/vm/google//modules/mig (and adding a number of resources as deps for them)
  • from GoogleCloudPlatform/nat-gateway/google to terraform-google-modules/cloud-nat

As I reported currently for_each has a bug which brakes all multi-backend examples down

cc: @paulpalamarchuk @nick4fake @kopachevsky

@ivankorn
Copy link
Contributor Author

ivankorn commented Sep 11, 2019

Update: As @morgante advised I pushed for_each changes in a separate branch aside and reverted them back

TF issue ref: hashicorp/terraform#22735
cc: @kopachevsky @nick4fake @paulpalamarchuk @ingwarr @aaron-lane

@aaron-lane aaron-lane added the enhancement New feature or request label Sep 17, 2019
@sebastiansirch
Copy link

Hi @ivankorn ,
just want to let you know that I already tested and used your current working state in a project where we needed a Terraform 12 compatible version :-) Works fine.
Best,
Sebastian

@ivankorn
Copy link
Contributor Author

ivankorn commented Sep 19, 2019

Hi, @sebastiansirch
Thank you for your feedback! I'm very happy to hear that migration efforts were successful! :)
I'll notify you once PR get merged, it's under code review which shouldn't take much time.

Please note: there are #53 and #54 opened for interface changes, so you might need to adjust your side code as well after those get merged.
Thanks

@mercuriete
Copy link

@ivankorn

Thank you for this.
I already have a custom version of this module removing everything wasnt working.
I will switch to this official when possible.

Thank you very much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants