503 failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4005:803::200a%5D:443: connect: No route to host (65)

[crizCraig]

Environment details

• Mac Sonoma 14.4
• Python version: Python 3.12.7
• pip version: 24.2
• google-cloud-firestore version: 2.20.1

Steps to reproduce

1. Make several reads and writes with Firestore in my local environment (non debug mode) with my app
2. Everything works fine for a bit despite these warnings

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740615594.970244 5064574 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers
I0000 00:00:1740615594.977429 5064574 fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers

3. Then I get stacktraces like this for the same reads and writes that worked previously. Restarting the process fixes it again.

Traceback (most recent call last):
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 77, in wait_for_connection
    await self._call.wait_for_connection()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/grpc/aio/_call.py", line 659, in wait_for_connection
    await self._raise_for_status()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/grpc/aio/_call.py", line 272, in _raise_for_status
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4005:803::200a%5D:443: connect: No route to host (65)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-26T16:20:12.910755-08:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4005:803::200a%5D:443: connect: No route to host (65)"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/api_core/retry/retry_unary_async.py", line 155, in retry_target
    return await target()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 178, in error_remapped_callable
    await call.wait_for_connection()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 79, in wait_for_connection
    raise exceptions.from_grpc_error(rpc_error) from rpc_error
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2607:f8b0:4005:803::200a%5D:443: connect: No route to host (65)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/cloud/firestore_v1/async_document.py", line 367, in get
    response_iter = await self._client._firestore_api.batch_get_documents(
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/api_core/retry/retry_unary_async.py", line 230, in retry_wrapped_func
    return await retry_target(
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/google/api_core/retry/retry_unary_async.py", line 171, in retry_target
    await asyncio.sleep(sleep)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/tasks.py", line 665, in sleep
    return await future
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/base_events.py", line 1986, in _run_once
    handle._run()
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/craigquiter/src/polychat/polychat/gcp.py", line 121, in log_to_bigquery
    await update_total_usage(user_email, user_id)
  File "/Users/craigquiter/src/polychat/polychat/utils.py", line 31, in wrapped_fn_async
    ret = await fn(*args, **kwargs)
  File "/Users/craigquiter/src/polychat/polychat/gcp.py", line 137, in update_total_usage
    prev = await get_db_usage_summary(user_email, user_id)
  File "/Users/craigquiter/src/polychat/polychat/utils.py", line 31, in wrapped_fn_async
    ret = await fn(*args, **kwargs)
> File "/Users/craigquiter/src/polychat/polychat/gcp.py", line 489, in get_db_usage_summary
    doc = await asyncio.wait_for(doc_ref.get(), timeout=15)  # 15 second timeout
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/tasks.py", line 519, in wait_for
    async with timeouts.timeout(timeout):
  File "/Users/craigquiter/miniconda3/envs/polychat/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

Thanks in advance for any help. This doesn't appear to be happening on Cloud Run where it's hosted, but I am getting very long response times from Firebase there despite Firebase monitoring reporting good response times. While investigating that issue, the above came up.