Python 3.10

[nebhead]

FYI: @weberbox I had to switch to an older laptop recently for my daily driver, and it's got PopOS 22.04 installed. It came with Python 3.10 by default, which is great for general performance, but it seems to have broken something with Flask-SocketIO in the Python collections. Might require filing a bug or something. I won't have time to debug this week, as I'll be away from the computer, but thought I would give a heads up.

ben@pop-os:~/syncthing/Tech-Projects/PiFire$ python app.py 
Traceback (most recent call last):
  File "/home/ben/syncthing/Tech-Projects/PiFire/app.py", line 17, in <module>
    from flask_socketio import SocketIO
  File "/usr/local/lib/python3.10/dist-packages/flask_socketio/__init__.py", line 9, in <module>
    from socketio import socketio_manage  # noqa: F401
  File "/usr/local/lib/python3.10/dist-packages/socketio/__init__.py", line 9, in <module>
    from .zmq_manager import ZmqManager
  File "/usr/local/lib/python3.10/dist-packages/socketio/zmq_manager.py", line 5, in <module>
    import eventlet.green.zmq as zmq
  File "/usr/local/lib/python3.10/dist-packages/eventlet/__init__.py", line 17, in <module>
    from eventlet import convenience
  File "/usr/local/lib/python3.10/dist-packages/eventlet/convenience.py", line 7, in <module>
    from eventlet.green import socket
  File "/usr/local/lib/python3.10/dist-packages/eventlet/green/socket.py", line 21, in <module>
    from eventlet.support import greendns
  File "/usr/local/lib/python3.10/dist-packages/eventlet/support/greendns.py", line 66, in <module>
    setattr(dns, pkg, import_patched('dns.' + pkg))
  File "/usr/local/lib/python3.10/dist-packages/eventlet/support/greendns.py", line 61, in import_patched
    return patcher.import_patched(module_name, **modules)
  File "/usr/local/lib/python3.10/dist-packages/eventlet/patcher.py", line 132, in import_patched
    return inject(
  File "/usr/local/lib/python3.10/dist-packages/eventlet/patcher.py", line 109, in inject
    module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
  File "/usr/local/lib/python3.10/dist-packages/dns/namedict.py", line 35, in <module>
    class NameDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

[weberbox]

@nebhead I think I know what the issue is.

In python 3.10 the module MutableMapping has moved from:
collections.MutableMapping to
collections.abc.MutableMapping

I have not fully checked yet but I am pretty sure that I used mutable mapping for checking and inputting values in settings.json.

Edit:
Scratch my first though it seems something in the SocketIO package itself or maybe eventlet is using MutableMapping from collections not collections.abc

This might be an issue as I think I am using a specific version of eventlet for a reason and I suspect that it would need to be updated to fix this but newer versions break something that I cant seem recall right now.

I am busy this weekend but I will take a look if I get some free time if not then during the week

[weberbox]

Here is what worked for me to get the newer version of eventlet (0.33.1) to run. I still have not tried it with python 3.10 but it should work since the entire issue was due to the older version of eventlet 0.30.2 and the change to MutableMapping

1. Remove gunicorn3 from the apt packages by remove or purge
2. Install the specific version of gunicorn via pip (not apt)

sudo pip3 install https://github.com/benoitc/gunicorn/archive/refs/heads/master.zip#egg=gunicorn==20.1.0

3. Update supervisor webapp.conf file to use the new gunicorn and not use /usr/bin/gunicorn3
   [program:webapp]
   command=gunicorn -k eventlet -b 0.0.0.0:8000 -w 1 app:app
   directory=/usr/local/bin/pifire
   autostart=true
   autorestart=true
   startretries=3
   stderr_logfile=/usr/local/bin/pifire/logs/webapp.err.log
   stdout_logfile=/usr/local/bin/pifire/logs/webapp.out.log
   user=root

I now start app.py manually with
sudo gunicorn -k eventlet -b 0.0.0.0:8000 -w 1 app:app

If you try this let me know if it works out. I have been meaning to upgrade to python 3.10 and give a try but have been working on other stuff.

The whole issue is a major api change in eventlet that breaks it working with gunicorn. Normally eventlet 0.30.2 works because it was before the api change but now that python 3.10 is here eventlet 0.30.2 breaks because python moved MutableMapping from collections to collections.abc

So if we upgrade to eventlet 0.33.1 the app will not start due to a conflict between eventlet and gunicorn. If we keep eventlet 0.30.2 then the app will not start due to the python MutableMapping change.

The only workaround is pinning gunicorn to an older version that does not seem to break eventlet or wait until eventlet truly fixes the issue with a newer version. It was supposed to be fixed in 0.33.1 but it was not.

[nebhead]

All makes sense, but unfortunately, I'm still getting the same error. Maybe it's an issue with my configuration. I wouldn't recommend moving to Python 3.10 right now, as I'm worried you might get backed into the same corner.

ben@pop-os:~/syncthing/Tech-Projects/PiFire$ sudo gunicorn -k eventlet -b 0.0.0.0:8000 -w 1 app:app

Error: class uri 'eventlet' invalid or not found: 

[Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gunicorn/util.py", line 99, in load_class
    mod = importlib.import_module('.'.join(components))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/gunicorn/workers/geventlet.py", line 10, in <module>
    import eventlet
  File "/usr/local/lib/python3.10/dist-packages/eventlet/__init__.py", line 17, in <module>
    from eventlet import convenience
  File "/usr/local/lib/python3.10/dist-packages/eventlet/convenience.py", line 7, in <module>
    from eventlet.green import socket
  File "/usr/local/lib/python3.10/dist-packages/eventlet/green/socket.py", line 21, in <module>
    from eventlet.support import greendns
  File "/usr/local/lib/python3.10/dist-packages/eventlet/support/greendns.py", line 66, in <module>
    setattr(dns, pkg, import_patched('dns.' + pkg))
  File "/usr/local/lib/python3.10/dist-packages/eventlet/support/greendns.py", line 61, in import_patched
    return patcher.import_patched(module_name, **modules)
  File "/usr/local/lib/python3.10/dist-packages/eventlet/patcher.py", line 132, in import_patched
    return inject(
  File "/usr/local/lib/python3.10/dist-packages/eventlet/patcher.py", line 109, in inject
    module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
  File "/usr/local/lib/python3.10/dist-packages/dns/namedict.py", line 35, in <module>
    class NameDict(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
]

Previously, I was just running Python (without Gunicorn) because I wasn't setting up all of the reverse proxy stuff, etc. on my PC, was just wanting to do basic testing. In the past, it had worked OK without Gunicorn launching it, however, I wasn't testing the Android SocketIO stuff in this configuration.

[weberbox]

No worries on upgrading. I run multiple VM instances of DietPi in Hyper-V for all my pi/pifire testing. So if I mess something up too bad I just nuke it and spin up a new VM.

I wonder if your dns package is causing the issue.

File "/usr/local/lib/python3.10/dist-packages/dns/namedict.py", line 35, in <module> class NameDict(collections.MutableMapping):

What version of dnspython do you have installed in pip?

[nebhead]

No worries on upgrading. I run multiple VM instances of DietPi in Hyper-V for all my pi/pifire testing. So if I mess something up too bad I just nuke it and spin up a new VM.

Oh nice! And here I've been testing on a spare Pi Zero!

I wonder if your dns package is causing the issue.

File "/usr/local/lib/python3.10/dist-packages/dns/namedict.py", line 35, in <module> class NameDict(collections.MutableMapping):

What version of dnspython do you have installed in pip?

Looks like I have version 1.16.0 installed for dnspython.

Honestly, I've been having some other strange problems with PopOS. I'm thinking about just wiping this laptop and going with good old Ubuntu (which is what I'm running on my desktop box).

[nebhead]

Well holy cow, that did it. Just ran pip install --upgrade dnspython and boom, we are back in business! Thanks for your help on this one!