Deadlock on ssh agent forwarding session failure

[jcaamano]

When ssh session open for agent forwarding fails as part of ssh connection, no further connection attempts are made. Looking at the code, this is due to a deadlock on calling ssh Connect recursively.

Fix attempt: #23661

Terraform Version

0.12.16

Terraform Configuration Files

Issue can happen with any ssh connection block that enables agent use either explicitly or implicitly by setting agent environment variables.

resource "null_resource" "master_wait_cloudinit" {
  depends_on = [libvirt_domain.master]
  count      = var.masters

  connection {
    host = element(
      libvirt_domain.master.*.network_interface.0.addresses.0,
      count.index,
    )
    user     = var.username
    password = var.password
    type     = "ssh"
  }

  provisioner "remote-exec" {
    inline = [
      "cloud-init status --wait > /dev/null",
    ]
  }
}

Debug Output

https://gist.github.com/jcaamano/2910aff27fd0d5cad0101c0334b04e18

Crash Output

Expected Behavior

When opening the ssh session for agent forwarding fails as part of the ssh connection, connection should be reattempted. Relevant log section:

remote-exec-provisioner (internal) 2019/12/11 14:43:16 [DEBUG] Telling SSH config to forward to agent
remote-exec-provisioner (internal) 2019/12/11 14:43:16 [DEBUG] Setting up a session to request agent forwarding
remote-exec-provisioner (internal) 2019/12/11 14:43:16 [DEBUG] opening new ssh session
remote-exec-provisioner (internal) 2019/12/11 14:43:16 [WARN] ssh session open error: 'ssh: unexpected packet in response to channel open: <nil>', attempting reconnect

Actual Behavior

No further connection attempts are made.

Steps to Reproduce

Unknown. On my scenario I suspect it is failing due to cloud init setting up ssh on the remote simultaneously, but have not been able to pinpoint a precise reason.

Additional Context

References

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.