DUT server: random crashes in MarkAsDefunct

[bluebin14]

Problem

DUT firmware [Nordic] randomly crashes sometimes in MarkAsDefunct. Does not look platform specific and unfortunately cannot debug it at the moment. Two crashes attached.

Proposed Solution

Check what could cause the random crash.

[bluebin14]

@Damian-Nordic

[bluebin14]

TCCADMIN15UART.txt
anothercrash.txt

[bzbarsky-apple]

@bluebin14 Is this crashing inside MarkAsDefunct, or after MarkAsDefunct returns and we are doing the OnSessionHang notification? It's not clear to me from the logged stacks which function we are actually crashing in....

That said, we are clearly not getting to the MoveToState call inside MarkAsDefunct, since that would log things that are not appearing in the log. So we're in one of the other states, and it would be really good to figure out which one we're in... @bluebin14 do you have a complete log for one of those crashes so we can look at previous state transition logs for the session involved, since we're not logging the state?

[bluebin14]

I do not have complete logs but it may have resulted from the controller exiting due to --timeout setting (i.e. without cleaning up) leaving the device in an undefined state then continuing with Sigma 1 with next chip-tool call, that caused the crash.

[mrjerryjohns]

Any updates here @bluebin14 ? If you could try to repro it, that would be really helpful.

[bluebin14]

I did 2 things that currently seem to not produce the crash: reduce sleep interval to 300 ms and increase command timeout. After SVE2 I can debug it further.

[bzbarsky-apple]

Is it possible that this was basically the same issue as #23093 and hence got fixed by #23097 ?

@bluebin14

[bluebin14]

@bzbarsky-apple it is possibly related so closing it for now since it is not reproducible with current code.