Feature: Read before Create

[kiwiscott]

When managing resources that have already been created in a stack such as an existing aws_security_group, or in our case a Vsphere cluster we’d like the option to read the current state before running an update and for terraform to explicitly manage changes to the fields that are explicitly set in config (this is standard at the moment).

Our idea is to add a new lifecycle option, read_before_create, that will reach out to the infrastructure and read the current state before trying to create new infrastructure.

We’d have to take into account the fields that would be the keys from the providers as well (in most cases it appears to be a name).

If we can change the core by adding a new lifecycle change then we shouldn’t need to make any changes to providers (which at this stage is probably an overwhelming task).

Making this change would allow many more uses cases around managing infrastructure that has a pre-existing state including both cloud and physical hosts.

One example where the existing approach is difficult for us is in the vsphere provider to manage a switch. The switch is automatically created at OS build time but then can’t be manipulated in TF without a manual import.

I haven’t gone through all the features of ‘terraform import’ but I’m trying that.

If there’s interest we could devote some time to looking into how this could work in the code base. (It’s theoretically an easy change).

PS. Sorry for the stupid typo in the title!!

[mildwonkey]

Hi @kiwiscott! I fixed your typo, no need to apologize :)

This is a great question. Importing previously created resources with terraform import is the expected workflow in your given situation, so I'm not sure what you are envisioning that's different. Could you elaborate? How would you like to see terraform identify resources that have already been created other than the command-line import?

[pdecat]

I too could see a need for this sort of automatic import if exists and not yet in state.

A concrete use case with the kubernetes provider would be changing the annotations of a pre-provisioned gp2 storage class that comes with AWS EKS to unmake the default:

resource "kubernetes_storage_class" "gp2" {
  lifecycle {
    auto_import = true # NOT IMPLEMENTED
  }

 metadata {
    name = "gp2"

    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "false"
    }
  }

...
}

resource "kubernetes_storage_class" "gp2_encrypted" {
 metadata {
    name = "gp2-encrypted"

    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }

...
}

Note: that came up in the kubernetes #terraform-providers channel today.

[yruss972]

I would also like to see an import_if_exists lifecycle.
This would be incredibly helpful in project migrations/upgrades especially in automated environments where users don't have direct access to the state backend.

It could either somehow flag the resource as already imported once to prevent import on the next apply, or just continue accepting whatever current state is read as the current base for the apply.

Another interesting option would be readonly_if_exists - effectively turning the resource into a data resource, without requiring code changes.

[paddycarver]

From a provider developer perspective, I think this would help with a lot of resources that are being "created" (don't exist in Terraform state) but exist in the API because they're global resources for the account that always exist.

If I had to make a proposal, I think the ability for providers to modify the current state part of a diff during PlanResourceChange in addition to the planned state would open the door for this. Alternatively, a separate RPC call that could let the providers add something to state (given the config and after Configure is called) before PlanResourceChange could also be a workable path forward, though seems like a larger lift.

[melinath]

This would be useful for the GCP provider as well. We have a large number of IAM policy resources (called google_*_iam_policy and google_*_iam_binding) that are considered "authoritative" – which is to say, they will clobber anything that is in GCP that might conflict.

This means that the first time a user runs a terraform apply, they are presented with an innocuous "creation" plan – but in fact, the policy ends up deleting policies in GCP and replacing them with what is in terraform with no warning.

This can lead to problems like users locking themselves out of their projects or losing critical and hard-to-recreate policies that were auto-generated internally by GCP as part of certain services or handwritten by the user.

This is particularly a noticeable issue because an IAM policy will always exist for a given resource in GCP, which means there's always something to take into account.

This can be worked around with a manual import, but that doesn't necessarily scale well (i.e. you'd have to do it for every new IAM policy; it wouldn't just be a one-off import) and it's an easy step to miss. This is a huge footgun for users. It would be great to have a feature to address this kind of use case.

[LDVSOFT]

I feel like this could be valuable for some resources that are identifiable from their inputs. This can be related to several auto-imports proposals (like #26364).

For example, aws_iam_group_membership is fully defined by user and membership groups: looks at import id, for example. This resource could be detected as existing before being created, and Terraform user can choose what should happen on creation: either to fail because resource already exists (user is already a group member) or actually accept it as being already existing, maybe requiring an update. In contrast, there can't be anything done with aws_instance because Terraform has no idea what EC2 Instance to read before running create.

There are other resources in AWS provder and other providers (sorry, but I've got most experience with Terrafrom using AWS) that are identified in that manner. Some of them are just global account settings. I guess all of them could use an option that reads state before creating, or on the other side, imports existing resource state and updates resource to desired state instead of creating from scratch.

[jpetrich]

I'd love to see this prioritized. I came here after running into the issue with the google cloud permissions issue mentioned above. It would be cool to at least have the output of plan be more intuitively correct in situations like google_project_iam_policy rather than technically correct but misleading.