watchmedo not working on M1 Mac

[diogohudson]

Hey Guys,

We have two mac Big Sur, one intel and the other M1. watchmedo is not working when called from the M1.

test_app| Inotify._raise_error() test_app | File "/usr/local/lib/python3.8/dist-packages/watchdog/observers/inotify_c.py", line 405, in _raise_error test_app | raise OSError(err, os.strerror(err)) **test_app | OSError: [Errno 38] Function not implemented** test_app_valid exited with code 1 test_app exited with code 1

watchmedo is being called inside a docker image.

Our compose.yml command:
command: watchmedo auto-restart -d test_app/test_app_valid/ -p '*.py' -- python3 -m debugpy --listen 0.0.0.0:20000 -m test_app. test_app_valid

[BoboTiG]

It seems the docker image does not know inotify stuff. What is the base OS of the docker image?

[aantn]

I'm having a similar issue with the watchdog library (not watchmedo):

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/app/robusta/utils/directory_watcher.py", line 47, in watch
    observer.start()
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/api.py", line 256, in start
    emitter.start()
  File "/usr/local/lib/python3.8/site-packages/watchdog/utils/__init__.py", line 93, in start
    self.on_thread_start()
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify.py", line 118, in on_thread_start
    self._inotify = InotifyBuffer(path, self.watch.is_recursive)
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify_buffer.py", line 35, in __init__
    self._inotify = Inotify(path, recursive)
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify_c.py", line 155, in __init__
    Inotify._raise_error()
  File "/usr/local/lib/python3.8/site-packages/watchdog/observers/inotify_c.py", line 405, in _raise_error
    raise OSError(err, os.strerror(err))
OSError: [Errno 38] Function not implemented

The base image is python:3.9-slim.

[BoboTiG]

@aantn, could you provide minimal repoduction steps?

[aantn]

I tried, but the minimal example I created doesn't reproduce the issue. Some differences between the original error and my failed reproduction of it:

1. The original error occured in a pod running on a KIND cluster on an M1 Macbook. The reproduction was done with Docker on an M1 Macbook
2. The original error occured in a full application so it's possible something else in the application is impacting this.

If I have time I will dig deeper on this, but I can't promise. I have a lot of other stuff going on right now and can't prioritize this.

[BoboTiG]

Even without being able to reproduce, can you tell me what to do to try reproducing?

[aantn]

I've got a working reproduction now.

Just run docker run -it us-central1-docker.pkg.dev/genuine-flight-317411/devel/watchdog_error:v1 on an M1 Macbook.
Alternatively (if you don't want to run a random docker image I tell you to :) then build https://github.com/aantn/watchdog_error on an x64 machine and run that x64 image on an M1 Macbook.

$ docker run -it us-central1-docker.pkg.dev/genuine-flight-317411/devel/watchdog_error:v1

Unable to find image 'us-central1-docker.pkg.dev/genuine-flight-317411/devel/watchdog_error:v1' locally
v1: Pulling from genuine-flight-317411/devel/watchdog_error
f8416d8bac72: Pull complete
76cee30eba29: Pull complete
25db5d180001: Pull complete
567b57d5964a: Pull complete
b489ab31cb6f: Pull complete
e38e2753d4ca: Pull complete
60a2361060f6: Pull complete
Digest: sha256:9b5f134c4d32d3df2cdff4b638218b68c39547f0281000170eef1c3c461d0a88
Status: Downloaded newer image for us-central1-docker.pkg.dev/genuine-flight-317411/devel/watchdog_error:v1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
starting
Traceback (most recent call last):
  File "/test.py", line 8, in <module>
    observer.start()
  File "/usr/local/lib/python3.9/site-packages/watchdog/observers/api.py", line 256, in start
    emitter.start()
  File "/usr/local/lib/python3.9/site-packages/watchdog/utils/__init__.py", line 93, in start
    self.on_thread_start()
  File "/usr/local/lib/python3.9/site-packages/watchdog/observers/inotify.py", line 118, in on_thread_start
    self._inotify = InotifyBuffer(path, self.watch.is_recursive)
  File "/usr/local/lib/python3.9/site-packages/watchdog/observers/inotify_buffer.py", line 35, in __init__
    self._inotify = Inotify(path, recursive)
  File "/usr/local/lib/python3.9/site-packages/watchdog/observers/inotify_c.py", line 155, in __init__
    Inotify._raise_error()
  File "/usr/local/lib/python3.9/site-packages/watchdog/observers/inotify_c.py", line 405, in _raise_error
    raise OSError(err, os.strerror(err))
OSError: [Errno 38] Function not implemented

[haydenth]

We've spent a lot of time working on this and the problem is that emulation on M1 does not support inotify .. This is documented here:

https://docs.docker.com/desktop/mac/apple-silicon/

In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

We've had to start removing watchdog from our python setups to support our M1 devs. Fortunately, some libraries (like flask) have alternative options to auto-reloading outside of using inotify but not all do.

[chrisdail]

If you have the flexibility of using an arm64 based container, this is also another solution to this problem. The docker issue is with inotify under the emulation mode only. If you have the option to use an arm based container, watchdog should work normally.

[BoboTiG]

The only solution is to switch to arm64-capable, eventually multi-arch, images.

[boomshadow]

We've spent a lot of time working on this and the problem is that emulation on M1 does not support inotify .. This is documented here:

https://docs.docker.com/desktop/mac/apple-silicon/

In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

We've had to start removing watchdog from our python setups to support our M1 devs. Fortunately, some libraries (like flask) have alternative options to auto-reloading outside of using inotify but not all do.

Because the problem is entirely related to qemu, if you switch from Docker desktop to OrbStack, the problem goes away! You can have watchmedo on Apple Silicon once again! Tested and using for myself for a while now.

[BoboTiG]

Thanks a lot for the information @boomshadow 🍾