iOS chiptool crash - _cppCommissioner->Shutdown();

[kean-apple]

Problem

iOS CHIPtool crashed after 2nd pairing attempt

SHA: 2e10854

1. Reset M5 board using Setup->Reset to Factory
2. Launch iOS chiptool app
3. Scan QR code to pair M5 board using iOS chiptool (it failed)
4. Reset M5 board again
5. Scan QR code to pair using iOS chiptool..seem to crash at this point

[Pairing-M5-failed-CHipTool.txt](https://github.com/project-chip/connectedhomeip/files/9305099/Pairing-M5-failed-CHipTool.txt) [CHIP-Tool-iOS-crash.txt](https://github.com/project-chip/connectedhomeip/files/9305100/CHIP-Tool-iOS-crash.txt)

[kean-apple]

CHIP-Tool-iOS-crash.txt

[kean-apple]

Pairing-M5-failed-CHipTool.txt

[bzbarsky-apple]

Have been trying to reproduce (by failing first commissioning due to failure to discover the device, then failing it due to having the wrong passcode) but haven't managed yet...

[bzbarsky-apple]

Managed to reproduce this by hacking the server to never respond to reads (just no-opped reporting::Engine::Run), then scanning the QR code, then scanning the QR code again while waiting for a response to the read the first commissioning attempt kicked off.

Stack:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
    frame #0: 0x0000000113a43624 Matter`chip::SessionManager::SystemLayer(this=0x0000000000000000) at SessionManager.h:369:44
    frame #1: 0x0000000113a6ec48 Matter`chip::Messaging::ExchangeContext::CancelResponseTimer(this=0x00000002823f1f80) at ExchangeContext.cpp:441:71
    frame #2: 0x0000000113a6f820 Matter`chip::Messaging::ExchangeContext::OnSessionReleased(this=0x00000002823f1f80) at ExchangeContext.cpp:405:9
    frame #3: 0x0000000113a3b034 Matter`chip::SessionHolderWithDelegate::SessionReleased(this=0x00000002823f1fc0) at Session.h:138:19
    frame #4: 0x0000000113a95aec Matter`chip::Transport::Session::NotifySessionReleased(this=0x00000002832c12c0) at Session.h:248:31
    frame #5: 0x0000000113a8ae34 Matter`chip::Transport::SecureSession::MarkForEviction(this=0x00000002832c12c0) at SecureSession.cpp:149:9
    frame #6: 0x0000000113a95e84 Matter`auto chip::SessionManager::Shutdown(this=0x000000016b535eff, session=0x00000002832c12c0)::$_0::operator()<chip::Transport::SecureSession*>(chip::Transport::SecureSession*) const at SessionManager.cpp:119:18
    frame #7: 0x0000000113a95e10 Matter`chip::internal::LambdaProxy<chip::Transport::SecureSession, chip::SessionManager::Shutdown()::$_0>::Call(context=0x000000016b535eff, target=0x00000002832c12c0) at Pool.h:126:16
    frame #8: 0x0000000113a6be30 Matter`chip::internal::HeapObjectList::ForEachNode(this=0x00000002832d4618, context=0x000000016b535eff, lambda=(Matter`chip::internal::LambdaProxy<chip::Transport::SecureSession, chip::SessionManager::Shutdown()::$_0>::Call(void*, void*) at Pool.h:125))(void*, void*)) at Pool.cpp:126:17
    frame #9: 0x0000000113a95d98 Matter`chip::Loop chip::HeapObjectPool<chip::Transport::SecureSession>::ForEachActiveObject<chip::SessionManager::Shutdown(this=0x00000002832d4608, function=0x000000016b535f67)::$_0>(chip::SessionManager::Shutdown()::$_0&&) at Pool.h:401:25
    frame #10: 0x0000000113a8f9f0 Matter`chip::Loop chip::Transport::SecureSessionTable::ForEachSession<chip::SessionManager::Shutdown()::$_0>(this=0x00000002832d4600, function=0x000000016b535f67)::$_0&&) at SecureSessionTable.h:84:25
    frame #11: 0x0000000113a8f650 Matter`chip::SessionManager::Shutdown(this=0x00000002832d45a0) at SessionManager.cpp:118:21
    frame #12: 0x0000000113a3f6f0 Matter`chip::Controller::DeviceControllerSystemState::Shutdown(this=0x0000000283cdd4a0) at CHIPDeviceControllerFactory.cpp:441:22
    frame #13: 0x0000000113a29530 Matter`chip::Controller::DeviceControllerSystemState::Release(this=0x0000000283cdd4a0) at CHIPDeviceControllerSystemState.h:160:13
    frame #14: 0x0000000113a29474 Matter`chip::Controller::DeviceController::Shutdown(this=0x0000000109814c00) at CHIPDeviceController.cpp:349:19
  * frame #15: 0x0000000113a2a6e8 Matter`chip::Controller::DeviceCommissioner::Shutdown(this=0x0000000109814c00) at CHIPDeviceController.cpp:530:23
    frame #16: 0x000000011238683c Matter`-[MTRDeviceController shutDownCppController](self=0x000000010b019a00, _cmd="shutDownCppController") at MTRDeviceController.mm:140:27
    frame #17: 0x00000001122746b0 Matter`-[MTRControllerFactory(self=0x00000002826d3330, _cmd="controllerShuttingDown:", controller=0x000000010b019a00) controllerShuttingDown:] at MTRControllerFactory.mm:572:9
    frame #18: 0x00000001123867ec Matter`-[MTRDeviceController cleanupAfterStartup](self=0x000000010b019a00, _cmd="cleanupAfterStartup") at MTRDeviceController.mm:131:5
    frame #19: 0x00000001123867b4 Matter`-[MTRDeviceController shutdown](self=0x000000010b019a00, _cmd="shutdown") at MTRDeviceController.mm:125:5
    frame #20: 0x00000001048ddce0 CHIPTool`MTRRestartController(controller=0x000000010b019a00) at DefaultsUtils.m:113:5
    frame #21: 0x00000001048d6c84 CHIPTool`-[QRCodeViewController _restartMatterStack](self=0x0000000109019a00, _cmd="_restartMatterStack") at QRCodeViewController.m:801:27
    frame #22: 0x00000001048d6d70 CHIPTool`-[QRCodeViewController handleRendezVousDefault:](self=0x0000000109019a00, _cmd="handleRendezVousDefault:", payload=@"MT:-24J0YXE00KA0648G00") at QRCodeViewController.m:812:5
    frame #23: 0x00000001048d6a7c CHIPTool`-[QRCodeViewController handleRendezVous:rawPayload:](self=0x0000000109019a00, _cmd="handleRendezVous:rawPayload:", payload=0x0000000283fc83f0, rawPayload=@"MT:-24J0YXE00KA0648G00") at QRCodeViewController.m:783:9
    frame #24: 0x00000001048d53fc CHIPTool`-[QRCodeViewController showPayload:rawPayload:isManualCode:](self=0x0000000109019a00, _cmd="showPayload:rawPayload:isManualCode:", payload=0x0000000283fc83f0, rawPayload=@"MT:-24J0YXE00KA0648G00", isManualCode=NO) at QRCodeViewController.m:603:5
    frame #25: 0x00000001048d73e8 CHIPTool`-[QRCodeViewController displayQRCodeInSetupPayloadView:rawPayload:error:](self=0x0000000109019a00, _cmd="displayQRCodeInSetupPayloadView:rawPayload:error:", payload=0x0000000283fc83f0, rawPayload=@"MT:-24J0YXE00KA0648G00", error=0x0000000000000000) at QRCodeViewController.m:872:9
    frame #26: 0x00000001048d7864 CHIPTool`__38-[QRCodeViewController scannedQRCode:]_block_invoke_3(.block_descriptor=0x000000028185d7c0) at QRCodeViewController.m:889:13
    frame #27: 0x0000000104cb27bc libdispatch.dylib`_dispatch_client_callout + 20
    frame #28: 0x0000000104cb5970 libdispatch.dylib`_dispatch_continuation_pop + 788
    frame #29: 0x0000000104ccc2cc libdispatch.dylib`_dispatch_source_invoke + 1676
    frame #30: 0x0000000104cc2ac8 libdispatch.dylib`_dispatch_main_queue_drain + 788
    frame #31: 0x0000000104cc27a4 libdispatch.dylib`_dispatch_main_queue_callback_4CF + 44
    frame #32: 0x00000001db65e2f0 CoreFoundation`__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
    frame #33: 0x00000001db6181f4 CoreFoundation`__CFRunLoopRun + 2532
    frame #34: 0x00000001db62b6b8 CoreFoundation`CFRunLoopRunSpecific + 600
    frame #35: 0x00000001f76c5374 GraphicsServices`GSEventRunModal + 164
    frame #36: 0x00000001ddf90e88 UIKitCore`-[UIApplication _run] + 1100
    frame #37: 0x00000001ddd125ec UIKitCore`UIApplicationMain + 364
    frame #38: 0x00000001048e9220 CHIPTool`main(argc=3, argv=0x000000016b537850) at main.m:27:12
    frame #39: 0x000000010499dce4 dyld`start + 520

[bzbarsky-apple]

We are doing DeviceControllerSystemState::Shutdown while there is a live exchange around that's waiting for a response. We shut down the session manager, which evicts the session, which calls ExchangeContext::OnSessionReleased which tries to cancel the response timer, but the exchange manager's session manager reference is gone by that point, so our attempt to get the system layer as:

mExchangeMgr->GetSessionManager()->SystemLayer()

ends up with a null-deref, and we crash. That last bit is similar to #20085 but for a different timer.

[bzbarsky-apple]

Fundamentally, device controller shutdown needs to cancel its pending commissioning work, but the APIs it's using (the InteractionModel convenience APIs) don't really allow that...

@mrjerryjohns I guess we could try using the lower-level APIs here, but it would be nice if the convenience APIs (optionally?) handed out some sort of token that could be used to cancel the async operation. I might try to do something like that when I get back.

[mrjerryjohns]

I think fixing #20880 is the right solution here. Shutting down the ExchangeMgr should shut-down all active exchanges, which should correctly trickle up and shut-down all active IM objects as well no?

[woody-apple]

This is no longer showing up in crashes.

[bzbarsky-apple]

For what it's worth, I think this is going to be effectively fixed by #22282: we will be ensuring that shutdown evicts sessions earlier, while more of the stack is up.