Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(CDK Deploy): (Elastic IP tags are not created as part of creation process) #26878

Closed
bturner-cpacket opened this issue Aug 24, 2023 · 11 comments
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud bug This issue is a bug. needs-cfn This issue is waiting on changes to CloudFormation before it can be addressed. p2

Comments

@bturner-cpacket
Copy link

Describe the bug

We have an AWS Service Control Policy (SCP) applied from our organization that prevents the creation of new resources if they dont have an owner tag in the format of an email. This has not been an issue for creation resources using the AWS console, the AWS CLI, or Python3 with boto3, however, it appears to be an issue with the AWS CDK.

The issue does not happen with instances and volumes with the same restrictions, only Elastic IP's.

I am guessing the issue is that with the CDK, the Elastic IP is created, then tags are applied after the Elastic IP has been created but I am not sure.

Here is my sample code the reproduces the problem:

from aws_cdk import (
    aws_ec2 as ec2,
    core as cdk,
)


class MyEc2ProjectStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(self, "MyVpc",
                      max_azs=2,
                      )

        instance = ec2.Instance(self, "MyInstance",
                                instance_type=ec2.InstanceType("t2.micro"),
                                machine_image=ec2.MachineImage.latest_amazon_linux(),
                                propagate_tags_to_volume_on_creation=True,
                                vpc=vpc,
                                )

        cdk.Tags.of(instance).add("owner", "a@b.com")
        cdk.Tags.of(instance).add("Name", "a-test-cdk")
        cdk.Tags.of(vpc).add("owner", "a@b.com")

When I run cdk synth, I see the proper tags on the EIP's:

Resources:
  MyVpcF9F0CA6F:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/Resource
  MyVpcPublicSubnet1SubnetF6608456:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: MyVpcF9F0CA6F
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      CidrBlock: 10.0.0.0/18
      MapPublicIpOnLaunch: true
      Tags:
        - Key: aws-cdk:subnet-name
          Value: Public
        - Key: aws-cdk:subnet-type
          Value: Public
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/Subnet
  MyVpcPublicSubnet1RouteTableC46AB2F4:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: MyVpcF9F0CA6F
      Tags:
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/RouteTable
  MyVpcPublicSubnet1RouteTableAssociation2ECEE1CB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: MyVpcPublicSubnet1RouteTableC46AB2F4
      SubnetId:
        Ref: MyVpcPublicSubnet1SubnetF6608456
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/RouteTableAssociation
  MyVpcPublicSubnet1DefaultRoute95FDF9EB:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId:
        Ref: MyVpcPublicSubnet1RouteTableC46AB2F4
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: MyVpcIGW5C4A4F63
    DependsOn:
      - MyVpcVPCGW488ACE0D
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/DefaultRoute
  MyVpcPublicSubnet1EIP096967CB:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/EIP

However when I run cdk deploy I get errors :

✨  Synthesis time: 4.22s

This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes
┌───┬────────────────────────────────┬────────┬────────────────┬───────────────────────────────┬───────────┐
│   │ Resource                       │ Effect │ Action         │ Principal                     │ Condition │
├───┼────────────────────────────────┼────────┼────────────────┼───────────────────────────────┼───────────┤
│ + │ ${MyInstance/InstanceRole.Arn} │ Allow  │ sts:AssumeRole │ Service:ec2.${AWS::URLSuffix} │           │
└───┴────────────────────────────────┴────────┴────────────────┴───────────────────────────────┴───────────┘
Security Group Changes
┌───┬─────────────────────────────────────────────┬─────┬────────────┬─────────────────┐
│   │ Group                                       │ Dir │ Protocol   │ Peer            │
├───┼─────────────────────────────────────────────┼─────┼────────────┼─────────────────┤
│ + │ ${MyInstance/InstanceSecurityGroup.GroupId} │ Out │ Everything │ Everyone (IPv4) │
└───┴─────────────────────────────────────────────┴─────┴────────────┴─────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Do you wish to deploy these changes (y/n)? y
MyEc2ProjectStack: deploying... [1/1]
MyEc2ProjectStack: creating CloudFormation changeset...
[████████··················································] (4/29)

10:14:19 PM | CREATE_FAILED        | AWS::EC2::EIP                         | MyVpc/PublicSubnet2/EIP
Resource handler returned message: "You are not authorized to perform this operation. Encoded authorization failure message: <REDACTED>
(Service: Ec2, Status Code: 403, Request ID: ec6461a0-7f7d-4865-802d-4544a6efb207)" (RequestToken: 35b1d1ff-fa03-2d39-8263-be53a5fcae69, HandlerErrorCode: AccessDenied)
10:14:19 PM | CREATE_FAILED        | AWS::EC2::EIP                         | MyVpc/PublicSubnet1/EIP
Resource handler returned message: "You are not authorized to perform this operation. Encoded authorization failure message: <REDACTED> (Service: Ec2, Status Code: 403, Request ID: bc8936a3-e911-4060-a124-62e4b1a71966)" (RequestToken: d4145145-00d3-e4a2-4a3d-7d339550cb3a, HandlerErrorCode: AccessDenied)
10:14:20 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack            | MyEc2ProjectStack
The following resource(s) failed to create: [MyVpcIGW5C4A4F63, MyVpcPublicSubnet1EIP096967CB, MyVpcF9F0CA6F, MyInstanceInstanceRole1C4D4747, MyVpcPublicSubnet2EIP8CCBA239, CDKMetadata]. Rollback requested b
y user.
10:14:20 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack            | MyEc2ProjectStack
The following resource(s) failed to create: [MyVpcIGW5C4A4F63, MyVpcPublicSubnet1EIP096967CB, MyVpcF9F0CA6F, MyInstanceInstanceRole1C4D4747, MyVpcPublicSubnet2EIP8CCBA239, CDKMetadata]. Rollback requested b
y user.

Decoding the error message shows me the issue is the SCP for Elastic IP's not being properly tagged.

I have tried variations of using ec2.CfnVPC but none of those worked.

vpc = ec2.CfnVPC(self, "MyVpc",  
    cidr_block="10.0.0.0/16",  # You need to specify a CIDR block  
    tags=[  
        cdk.CfnTag(key="owner", value="a@b.com"),  
    ],  
)

If I remove the SCP section for elastic IP owner tag and keep the same policy for instances and volumes I am able to successfully deploy my stack.

Expected Behavior

Successful deployment of my CDK stack

Current Behavior

Failure to deploy due to the Elastic IP not being tagged at creation time.

Reproduction Steps

Here is my sample code the reproduces the problem:

from aws_cdk import (
    aws_ec2 as ec2,
    core as cdk,
)


class MyEc2ProjectStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(self, "MyVpc",
                      max_azs=2,
                      )

        instance = ec2.Instance(self, "MyInstance",
                                instance_type=ec2.InstanceType("t2.micro"),
                                machine_image=ec2.MachineImage.latest_amazon_linux(),
                                propagate_tags_to_volume_on_creation=True,
                                vpc=vpc,
                                )

        cdk.Tags.of(instance).add("owner", "a@b.com")
        cdk.Tags.of(instance).add("Name", "a-test-cdk")
        cdk.Tags.of(vpc).add("owner", "a@b.com")

When I run cdk synth, I see the proper tags on the EIP's:

Resources:
  MyVpcF9F0CA6F:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/Resource
  MyVpcPublicSubnet1SubnetF6608456:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: MyVpcF9F0CA6F
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      CidrBlock: 10.0.0.0/18
      MapPublicIpOnLaunch: true
      Tags:
        - Key: aws-cdk:subnet-name
          Value: Public
        - Key: aws-cdk:subnet-type
          Value: Public
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/Subnet
  MyVpcPublicSubnet1RouteTableC46AB2F4:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: MyVpcF9F0CA6F
      Tags:
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/RouteTable
  MyVpcPublicSubnet1RouteTableAssociation2ECEE1CB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: MyVpcPublicSubnet1RouteTableC46AB2F4
      SubnetId:
        Ref: MyVpcPublicSubnet1SubnetF6608456
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/RouteTableAssociation
  MyVpcPublicSubnet1DefaultRoute95FDF9EB:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId:
        Ref: MyVpcPublicSubnet1RouteTableC46AB2F4
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: MyVpcIGW5C4A4F63
    DependsOn:
      - MyVpcVPCGW488ACE0D
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/DefaultRoute
  MyVpcPublicSubnet1EIP096967CB:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
        - Key: owner
          Value: a@b.com
    Metadata:
      aws:cdk:path: MyEc2ProjectStack/MyVpc/PublicSubnet1/EIP

However when I run cdk deploy I get errors :

✨  Synthesis time: 4.22s

This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes
┌───┬────────────────────────────────┬────────┬────────────────┬───────────────────────────────┬───────────┐
│   │ Resource                       │ Effect │ Action         │ Principal                     │ Condition │
├───┼────────────────────────────────┼────────┼────────────────┼───────────────────────────────┼───────────┤
│ + │ ${MyInstance/InstanceRole.Arn} │ Allow  │ sts:AssumeRole │ Service:ec2.${AWS::URLSuffix} │           │
└───┴────────────────────────────────┴────────┴────────────────┴───────────────────────────────┴───────────┘
Security Group Changes
┌───┬─────────────────────────────────────────────┬─────┬────────────┬─────────────────┐
│   │ Group                                       │ Dir │ Protocol   │ Peer            │
├───┼─────────────────────────────────────────────┼─────┼────────────┼─────────────────┤
│ + │ ${MyInstance/InstanceSecurityGroup.GroupId} │ Out │ Everything │ Everyone (IPv4) │
└───┴─────────────────────────────────────────────┴─────┴────────────┴─────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Do you wish to deploy these changes (y/n)? y
MyEc2ProjectStack: deploying... [1/1]
MyEc2ProjectStack: creating CloudFormation changeset...
[████████··················································] (4/29)

10:14:19 PM | CREATE_FAILED        | AWS::EC2::EIP                         | MyVpc/PublicSubnet2/EIP
Resource handler returned message: "You are not authorized to perform this operation. Encoded authorization failure message: <REDACTED>
(Service: Ec2, Status Code: 403, Request ID: ec6461a0-7f7d-4865-802d-4544a6efb207)" (RequestToken: 35b1d1ff-fa03-2d39-8263-be53a5fcae69, HandlerErrorCode: AccessDenied)
10:14:19 PM | CREATE_FAILED        | AWS::EC2::EIP                         | MyVpc/PublicSubnet1/EIP
Resource handler returned message: "You are not authorized to perform this operation. Encoded authorization failure message: <REDACTED> (Service: Ec2, Status Code: 403, Request ID: bc8936a3-e911-4060-a124-62e4b1a71966)" (RequestToken: d4145145-00d3-e4a2-4a3d-7d339550cb3a, HandlerErrorCode: AccessDenied)
10:14:20 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack            | MyEc2ProjectStack
The following resource(s) failed to create: [MyVpcIGW5C4A4F63, MyVpcPublicSubnet1EIP096967CB, MyVpcF9F0CA6F, MyInstanceInstanceRole1C4D4747, MyVpcPublicSubnet2EIP8CCBA239, CDKMetadata]. Rollback requested b
y user.
10:14:20 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack            | MyEc2ProjectStack
The following resource(s) failed to create: [MyVpcIGW5C4A4F63, MyVpcPublicSubnet1EIP096967CB, MyVpcF9F0CA6F, MyInstanceInstanceRole1C4D4747, MyVpcPublicSubnet2EIP8CCBA239, CDKMetadata]. Rollback requested b
y user.

Decoding the error message shows me the issue is the SCP for Elastic IP's not being properly tagged.

I have tried variations of using ec2.CfnVPC but none of those worked.

vpc = ec2.CfnVPC(self, "MyVpc",  
    cidr_block="10.0.0.0/16",  # You need to specify a CIDR block  
    tags=[  
        cdk.CfnTag(key="owner", value="a@b.com"),  
    ],  
)

If I remove the SCP section for elastic IP owner tag and keep the same policy for instances and volumes I am able to successfully deploy my stack.

Possible Solution

Similar to how the CDK has a propagate_tags_to_volume_on_creation=True flag it would be good if we could do this for Elastic IP's?

Additional Information/Context

No response

CDK CLI Version

2.86.0 (build 1130fab)

Framework Version

No response

Node.js Version

v16.20.1

OS

AWS Linux - 4.14.255-314-253.539.amzn2.x86_64

Language

Python

Language Version

Python 3.7.16

Other information

chatGPT gave me these helpful nuggets of information :)

Currently, the CDK does not have a direct way to add tags during EIP creation when using the low-level CfnEIP construct. This makes it tricky when SCPs require tags upon resource creation, as is your case.

Feedback to AWS: If you believe this is an issue with how CDK is handling the Elastic IP creation and tagging process, you might want to provide feedback to AWS or raise it in the AWS CDK GitHub repository.

@bturner-cpacket bturner-cpacket added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 24, 2023
@github-actions github-actions bot added the @aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud label Aug 24, 2023
@indrora indrora added p2 and removed needs-triage This issue or PR still needs to be triaged. labels Aug 24, 2023
@indrora
Copy link
Contributor

indrora commented Aug 24, 2023

The way I would propose that you solve this is to use a CDK Aspect on the stack.

CDK Aspects are run before the whole tree is synthesized into what will become CloudFormation templates, and allow you to modify the tree of items just before they are reified into CloudFormation templates. They implement a Visitor pattern.

This will allow you to ensure that the tags of a construct down to a CloudFormation item in a deeper construct are tagged appropriately.

For more information, see: https://docs.aws.amazon.com/cdk/v2/guide/aspects.html

@peterwoodworth
Copy link
Contributor

Tags are implemented with an aspect in the first place. Also, if the template has the tags in the first place, then the CDK is doing its job and this could be a service error, or an issue with the CloudFormation implementation of the resource

@peterwoodworth peterwoodworth added the needs-cfn This issue is waiting on changes to CloudFormation before it can be addressed. label Aug 25, 2023
@bturner-cpacket
Copy link
Author

Hi @peterwoodworth , I see that tag needs-cfn was applied... is there something more I can do to pin down if this is indeed an issue with the CloudFormation implementation of Elastic IP tagging?

@peterwoodworth
Copy link
Contributor

If the template is synthesizing with tags (which it is), and you've properly tracked down that this is an issue with the tags on the EIP resource, then this isn't a CDK issue because the only role we play in this is synthesizing the template.

@peterwoodworth
Copy link
Contributor

Also - I wasn't able to reproduce the issue with the CDK snippet provided. It appears the Cfn template you provided might be different than the one the CDK code you provided would synthesize though

@peterwoodworth peterwoodworth added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. needs-reproduction This issue needs reproduction. labels Aug 25, 2023
@bturner-cpacket
Copy link
Author

bturner-cpacket commented Aug 25, 2023

Hi @peterwoodworth - Thanks for looking. I don't think you can reproduce the issue unless your AWS account your testing in has a service control policy that blocks the creation of elastic IP addresses without specific tags created with the resource?

That is the root issue, we don't allow resources to be created unless they have an owner tag at creation time, and for all other methods (Python, AWS CLI, AWS Console) you can attach a tag at creation time. I think the issue here is that you can not with the CDK?

For example, our SCP as it relates to Elastic IP tagging looks something like this:

    {
      "Sid": "DenyElasticIPCreationNoOwnerTag",
      "Effect": "Deny",
      "Action": [
        "ec2:AllocateAddress"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:elastic-ip/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/owner": "true"
        }
      }
    },
    {
      "Sid": "DenyElasticIPCreationInvalidOwnerTag",
      "Effect": "Deny",
      "Action": [
        "ec2:AllocateAddress"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:elastic-ip/*"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:RequestTag/owner": [
            "*@*.*"
          ]
        }
      }
    },

If I remove these sections from our SCP, the CDK succeeds in creating the Elastic IP addresses.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Aug 25, 2023
@peterwoodworth
Copy link
Contributor

Right sorry - However, I am seeing that the tags are properly deploying on my end. If I were to make a guess, I'd guess that the CloudFormation implementation of this resource likely creates the resource, and then adds tags with a subsequent API call. Which would break an SCP like this. If you have premium support I suggest reaching out to them, else I can create a ticket internally

@peterwoodworth peterwoodworth removed the needs-reproduction This issue needs reproduction. label Aug 25, 2023
@bturner-cpacket
Copy link
Author

I did submit a ticket in AWS and got a response that they were able to see the same behavior on their end. This can be replicated when using just Cloudformation as well. They tried deploying the below Template:

MyVpcPublicSubnet1EIP096967CB:
Type: AWS::EC2::EIP
Properties:
Domain: vpc
Tags:
- Key: Name
Value: MyEc2ProjectStack/MyVpc/PublicSubnet1
- Key: owner
Value: a@b.com

Creation of AWS::EC2::EIP Resource, currently triggers a "AllocateAddress" API followed with a separate “CreateTags” API Call which tags the resource. Due to this the creation of AWS::EC2::EIP fails because the SCP on expects the Tags to be passed in the same AllocateAddress API.

It looks like the only thing to do is to wait for the CloudFormation team to address this on their side. My understanding is that this is not a CDK issue and so having this remained closed is appropriate.

@peterwoodworth
Copy link
Contributor

Yeah makes sense, thanks for getting back to me

@github-actions
Copy link

github-actions bot commented Sep 5, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@bturner-cpacket
Copy link
Author

Just in case anyone else stumbles upon this thread...

The AWS CloudFormation team pushed a fix for this on 2023-10-21 and I was able to validate that with my workflow above I no longer see the issue. Adding a tag to an elastic IP at creation time is now supported in CloudFormation, and by extension through the CDK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud bug This issue is a bug. needs-cfn This issue is waiting on changes to CloudFormation before it can be addressed. p2
Projects
None yet
Development

No branches or pull requests

3 participants