cdk deploy: v8 internal error thrown with expired credentials and broken bootstrap stack

[Gamecock]

Describe the bug

When following python workshop cdk deploy failed with opaque stack trace.
This occurred after credentials had expired.
Then updated credentials and CDK diff ran successfully
CDK deploy still did not run.
CDK synth also ran successfully

Expected Behavior

Updateed files to deploy and/or error message indicating issue.

Current Behavior

Stack Trace:
2259252@USSTP00044562 MINGW64 /c/Projects/AWS/CDK-python/cdk_workshop (master)
$ cdk deploy

Fatal error in , line 0

Check failed: AddGrowableFunctionTable( &record->dynamic_table, record->runtime_function, record->runtime_function_count, record->runtime_function_count, reinterpret_cast(start), reinterpret_cast(reinterpret_cast<uint8_t*>(start) + size_in_bytes)).

#FailureMessage Object: 0000009D8055E9E0
1: 00007FF6EE2B79CF v8::internal::CodeObjectRegistry::~CodeObjectRegistry+114207
2: 00007FF6EE1D3E9F std::basic_ostream<char,std::char_traits >::operator<<+65103
3: 00007FF6EEEB26C2 V8_Fatal+162
4: 00007FF6EE4F146E v8::internal::AssemblerBase::RecordComment+926
5: 00007FF6EEA66C7C v8::internal::Isolate::Init+3644
6: 00007FF6EE7174B9 v8::internal::Deserializerv8::internal::LocalIsolate::GetChecksum+585
7: 00007FF6EEB6FA44 v8::Isolate::Initialize+340
8: 00007FF6EE1FE2C1 v8::internal::Isolate::stack_guard+52417
9: 00007FF6EE27AA7C node::Start+204
10: 00007FF6EE09897C RC4_options+348284
11: 00007FF6EF0FFDB8 v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+14472
12: 00007FF8930D4034 BaseThreadInitThunk+20
13: 00007FF8934C3691 RtlUserThreadStart+33

Reproduction Steps

Followed this tutorial, previous page worked as expected:
https://cdkworkshop.com/30-python/40-hit-counter/600-permissions.html

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.28.1 (build d035432)

Framework Version

No response

Node.js Version

v16.15.1

OS

Windows2019

Language

Python

Language Version

Python 3.8.0

Other information

No response

[Gamecock]

I was able to resolve by updating cdk version based on the recommendation from cdk synth.

---

*** Newer version of CDK is available [2.29.0] ***
*** Upgrade recommended (npm install -g aws-cdk) ***

---

However allowing the credentials to expire generated a similar stack trace.

$ cdk deploy b'\r\n' b'\r\n' b'#\r\n' b'# Fatal error in , line 0\r\n' b'# Check failed: AddGrowableFunctionTable( &record->dynamic_table, record->runtime_function, record->runtime_function_count, record->runtime_function_count, reinterpret_cast<DWORD64>(start), reinterpret_cast<DWORD64>(reinterpret_cast<uint8_t*>(start) + size_in_bytes)).\r\n' b'#\r\n' b'#\r\n' b'#\r\n' b'#FailureMessage Object: 00000014148FEA50\r\n' b' 1: 00007FF72B6D79CF v8::internal::CodeObjectRegistry::~CodeObjectRegistry+114207\r\n' b' 2: 00007FF72B5F3E9F std::basic_ostream<char,std::char_traits<char> >::operator<<+65103\r\n' b' 3: 00007FF72C2D26C2 V8_Fatal+162\r\n' b' 4: 00007FF72B91146E v8::internal::AssemblerBase::RecordComment+926\r\n' b' 5: 00007FF72BE86C7C v8::internal::Isolate::Init+3644\r\n' b' 6: 00007FF72BB374B9 v8::internal::Deserializer<v8::internal::LocalIsolate>::GetChecksum+585\r\n' b' 7: 00007FF72BF8FA44 v8::Isolate::Initialize+340\r\n' b' 8: 00007FF72B61E2C1 v8::internal::Isolate::stack_guard+52417\r\n' b' 9: 00007FF72B69AA7C node::Start+204\r\n' b'10: 00007FF72B4B897C RC4_options+348284\r\n' b'11: 00007FF72C51FDB8 v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+14472\r\n' b'12: 00007FF8930D4034 BaseThreadInitThunk+20\r\n' b'13: 00007FF8934C3691 RtlUserThreadStart+33\r\n' Traceback (most recent call last): File "app.py", line 3, in <module> import aws_cdk as cdk File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\aws_cdk\__init__.py", line 1051, in <module> from ._jsii import * File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\aws_cdk\_jsii\__init__.py", line 11, in <module> import constructs._jsii File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\constructs\__init__.py", line 41, in <module> from ._jsii import * File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\constructs\_jsii\__init__.py", line 11, in <module> __jsii_assembly__ = jsii.JSIIAssembly.load( File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_runtime.py", line 43, in load _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path)) File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\__init__.py", line 269, in load self.provider.load(LoadRequest(name=name, version=version, tarball=tarball)) File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 338, in load return self._process.send(request, LoadResponse) File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_utils.py", line 24, in wrapped stored.append(fgetter(self)) File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 333, in _process process.start() File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 275, in start self.handshake() File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 299, in handshake self._next_message(), _HelloResponse File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 242, in _next_message return json.loads(self._process.stdout.readline(), object_hook=ohook) File "C:\Users\2259252\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 370, in loads return cls(**kw).decode(s) File "C:\Users\2259252\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\2259252\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Error in atexit._run_exitfuncs: Traceback (most recent call last): File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 284, in stop self._process.stdin.close() OSError: [Errno 22] Invalid argument

This time updating the ENVs for credentials resolved the issue.

[rix0rrr]

You're running into an internal V8 error. I have no idea how that gets caused by expired AWS credentials.

[peterwoodworth]

If you can find a consistent way to reproduce this, let us know. Otherwise, I'm not sure there's much we can do to help here. Following the Python workshop works for me and I've never come across this particular error when using expired credentials

[Gamecock]

I updated to CDK version 2.31.0 (build b67950d).
I no longer get this error with expired credential. I now get:
The security token included in the request is expired

I did figure out a way to replicate the stack trace. If the CDK bootstrap stack either does not exist or is corrupted in the region I can generate the below stack trace. I was having some trouble with the CDK originally, so that may have been the issue all along.

b'\r\n'
b'\r\n'
b'#\r\n'
b'# Fatal error in , line 0\r\n'
b'# Check failed: AddGrowableFunctionTable( &record->dynamic_table, record->runtime_function, record->runtime_function_count, record->runtime_function_count, reinterpret_cast<DWORD64>(start), reinterpret_cast<DWORD64>(reinterpret_cast<uint8_t*>(start) + size_in_bytes)).\r\n'
b'#\r\n'
b'#\r\n'
b'#\r\n'
b'#FailureMessage Object: 0000008D874FEEF0\r\n'
b' 1: 00007FF7226379CF v8::internal::CodeObjectRegistry::~CodeObjectRegistry+114207\r\n'
b' 2: 00007FF722553E9F std::basic_ostream<char,std::char_traits<char> >::operator<<+65103\r\n'
b' 3: 00007FF7232326C2 V8_Fatal+162\r\n'
b' 4: 00007FF72287146E v8::internal::AssemblerBase::RecordComment+926\r\n'
b' 5: 00007FF722DE6C7C v8::internal::Isolate::Init+3644\r\n'
b' 6: 00007FF722A974B9 v8::internal::Deserializer<v8::internal::LocalIsolate>::GetChecksum+585\r\n'
b' 7: 00007FF722EEFA44 v8::Isolate::Initialize+340\r\n'
b' 8: 00007FF72257E2C1 v8::internal::Isolate::stack_guard+52417\r\n'
b' 9: 00007FF7225FAA7C node::Start+204\r\n'
b'10: 00007FF72241897C RC4_options+348284\r\n'
b'11: 00007FF72347FDB8 v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+14472\r\n'
b'12: 00007FF89A734034 BaseThreadInitThunk+20\r\n'
b'13: 00007FF89CAB3691 RtlUserThreadStart+33\r\n'
Traceback (most recent call last):
  File "app.py", line 3, in <module>
    import aws_cdk as cdk
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\aws_cdk\__init__.py", line 1051, in <module>
    from ._jsii import *
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\aws_cdk\_jsii\__init__.py", line 11, in <module>
    import constructs._jsii
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\constructs\__init__.py", line 41, in <module>
    from ._jsii import *
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\constructs\_jsii\__init__.py", line 11, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load(
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_runtime.py", line 43, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\__init__.py", line 269, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 338, in load
    return self._process.send(request, LoadResponse)
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_utils.py", line 24, in
wrapped
    stored.append(fgetter(self))
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 333, in _process
    process.start()
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 275, in start
    self.handshake()
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 299, in handshake
    self._next_message(), _HelloResponse
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 242, in _next_message
    return json.loads(self._process.stdout.readline(), object_hook=ohook)
  File "C:\Users\2259252\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 370, in loads
    return cls(**kw).decode(s)
  File "C:\Users\2259252\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\2259252\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Projects\AWS\CDK-python\cdk_workshop\.venv\lib\site-packages\jsii\_kernel\providers\process.py", line 284, in stop
    self._process.stdin.close()
OSError: [Errno 22] Invalid argument

Subprocess exited with error 1```

[Gamecock]

A little more experimentation, and I think it's expired creds and broken bootstrap.

[Gamecock]

I aslo tested with typescript cdk, slightly different behavior. I get the stacktrace regardless of credentials if bootstrap stack is broken.

[peterwoodworth]

I believe this PR will fix issues related to expired credentials throwing unhelpful errors #21052

[rix0rrr]

Is your Python CDK app performing AWS calls? We generally recommend you don't do that.

[rix0rrr]

I cannot reproduce this behavior, and I also don't see how it can be caused by expired credentials (as none of the code in the call stack performs AWS calls).

It's also not clear what broken bootstrap means.

I'm sorry, but this must be something specific to your computer, and as such not really fixable by us. Closing.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.