Deprecation Warnings

[adamjstewart]

Describe the bug

tensorboardX is using deprecated features of protobuf. This requires every project that indirectly depends on protobuf to add the following hack to their pyproject.toml: #654.

Minimal runnable code to reproduce the behavior

$ python -Werror -c 'import tensorboardX'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/__init__.py", line 5, in <module>
    from .torchvis import TorchVis
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/torchvis.py", line 10, in <module>
    from .writer import SummaryWriter
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/writer.py", line 16, in <module>
    from .comet_utils import CometLogger
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/comet_utils.py", line 7, in <module>
    from .summary import _clean_tag
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/summary.py", line 12, in <module>
    from .proto.summary_pb2 import Summary
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/summary_pb2.py", line 16, in <module>
    from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/tensor_pb2.py", line 16, in <module>
    from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 18, in <module>
    DESCRIPTOR = _descriptor.FileDescriptor(
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 1034, in __init__
    _Deprecated('FileDescriptor')
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 97, in _Deprecated
    warnings.warn(
DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.

Expected behavior

I would not expect tensorboardX to use deprecated features.

Screenshots

N/A

Environment

$ pip list|grep -E "torch|proto|tensor"
efficientnet-pytorch          0.7.1
protobuf                      3.20.1
pytorch-lightning             1.8.5
pytorch-sphinx-theme          0.0.24
segmentation-models-pytorch   0.3.1
tensorboardX                  2.5.1
torch                         1.13.1
torchmetrics                  0.11.0
torchvision                   0.14.1

Python environment

Python 3.10.8, Spack Environment

Additional context

This issue was first reported in #653 and ignored in #654. The correct solution is to not use deprecated features.

[lanpa]

Hi @adamjstewart, I was unable to reproduce the warning with your code. Here's my freshly created environment (with docker run -it --rm python:3.10 bash followed by some pip install):

Would you show your full environment?

[adamjstewart]

Here's the full environment (collapsed so as not to take up the entire screen).

Click to expand

$ pip list
Package                       Version
----------------------------- -----------
affine                        2.1.0
aiohttp                       3.8.1
aiosignal                     1.2.0
alabaster                     0.7.12
antlr4-python3-runtime        4.8
anyio                         3.6.1
appdirs                       1.4.4
appnope                       0.1.0
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
asttokens                     2.0.8
astunparse                    1.6.3
async-timeout                 4.0.2
attrs                         22.1.0
Babel                         2.10.3
backcall                      0.2.0
beautifulsoup4                4.11.1
beniget                       0.4.1
black                         22.10.0
bleach                        5.0.1
Bottleneck                    1.3.5
build                         0.7.0
cachetools                    5.2.0
Cartopy                       0.21.1
certifi                       2022.9.14
cffi                          1.15.1
cftime                        1.0.3.4
charset-normalizer            2.0.12
click                         8.1.3
click-plugins                 1.1.1
cligj                         0.5.0
cmocean                       2.0
colorama                      0.4.5
commonmark                    0.9.1
contourpy                     1.0.5
coverage                      6.4.4
cycler                        0.11.0
debugpy                       1.6.3
decorator                     5.1.1
defusedxml                    0.7.1
docutils                      0.19
efficientnet-pytorch          0.7.1
einops                        0.6.0
entrypoints                   0.4
et-xmlfile                    1.0.1
executing                     1.1.0
fastjsonschema                2.16.2
Fiona                         1.8.22
flake8                        5.0.4
fonttools                     4.37.3
frozenlist                    1.3.1
fsspec                        2022.11.0
future                        0.18.2
gast                          0.5.3
GDAL                          3.6.1
geocube                       0.3.2
geopandas                     0.11.1
gevent                        1.5.0
greenlet                      1.1.3
h5py                          3.7.0
idna                          3.4
imagesize                     1.4.1
importlib-metadata            4.12.0
iniconfig                     0.0.0
ipykernel                     6.16.0
ipython                       8.5.0
ipython-genutils              0.2.0
ipywidgets                    8.0.2
isort                         5.10.1
jaraco.classes                3.2.2
jdcal                         1.3
jedi                          0.18.1
Jinja2                        3.1.2
joblib                        1.2.0
json5                         0.9.10
jsonschema                    4.16.0
jupyter_client                7.3.5
jupyter_core                  5.1.0
jupyter-server                1.21.0
jupyterlab                    3.4.8
jupyterlab-pygments           0.2.2
jupyterlab-server             2.10.3
jupyterlab-widgets            3.0.3
keyring                       23.9.1
kiwisolver                    1.3.2
kornia                        0.6.9
laspy                         2.2.0
lightning-utilities           0.4.1
lxml                          4.9.1
MarkupSafe                    2.1.1
matplotlib                    3.6.2
matplotlib-inline             0.1.6
mccabe                        0.7.0
mercurial                     5.8
mistune                       2.0.4
more-itertools                8.14.0
multidict                     6.0.2
munch                         2.2.0
mypy                          0.991
mypy-extensions               0.4.3
nbclassic                     0.4.8
nbclient                      0.6.7
nbconvert                     7.0.0
nbformat                      5.7.0
nbmake                        1.3.3
nbsphinx                      0.8.8
nest-asyncio                  1.5.5
netCDF4                       1.5.8
notebook                      6.4.12
notebook_shim                 0.2.2
numexpr                       2.8.3
numpy                         1.24.0
odc-geo                       0.1.2
omegaconf                     2.1.0
openpyxl                      3.0.3
packaging                     21.3
pandas                        1.5.2
pandocfilters                 1.5.0
parso                         0.8.3
pathspec                      0.10.1
pep517                        0.12.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow-SIMD                   9.0.0.post1
pip                           22.2.2
pkginfo                       1.8.3
platformdirs                  2.5.2
pluggy                        1.0.0
ply                           3.11
pretrainedmodels              0.7.4
prometheus-client             0.14.1
prompt-toolkit                3.0.31
protobuf                      3.20.1
psutil                        5.9.2
ptyprocess                    0.7.0
pure-eval                     0.2.2
py                            1.11.0
pybind11                      2.10.0
pycocotools                   2.0.2
pycodestyle                   2.9.1
pycparser                     2.21
pydantic                      1.10.2
pydocstyle                    6.1.1
pyflakes                      2.5.0
pygeos                        0.10
Pygments                      2.13.0
pyparsing                     3.0.9
pyproj                        3.3.1
pyrsistent                    0.18.1
pyshp                         2.1.0
pystac                        1.4.0
pytest                        7.1.3
pytest-cov                    3.0.0
python-dateutil               2.8.2
pythran                       0.11.0
pytorch-lightning             1.8.5
pytorch-sphinx-theme          0.0.24
pytz                          2022.2.1
pyupgrade                     2.31.1
PyYAML                        6.0
pyzmq                         24.0.1
radiant-mlhub                 0.4.1
rarfile                       4.0
rasterio                      1.3.4
readme-renderer               37.3
requests                      2.28.1
requests-toolbelt             0.9.1
rfc3986                       1.4.0
rich                          12.5.1
rioxarray                     0.4.1.post0
Rtree                         1.0.1
scikit-learn                  1.2.0
scipy                         1.9.3
SCons                         4.3.0
segmentation-models-pytorch   0.3.1
Send2Trash                    1.8.0
setuptools                    59.4.0
Shapely                       1.8.5
six                           1.16.0
sniffio                       1.3.0
snowballstemmer               2.2.0
snuggs                        1.4.1
soupsieve                     2.3.2.post1
Sphinx                        5.3.0
sphinx-rtd-theme              0.5.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-programoutput   0.15
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.5.0
tensorboardX                  2.5.1
terminado                     0.15.0
threadpoolctl                 3.1.0
timm                          0.4.12
tinycss2                      1.1.1
tokenize-rt                   4.2.1
toml                          0.10.2
tomli                         2.0.1
torch                         1.13.1
torchmetrics                  0.11.0
torchvision                   0.14.1
tornado                       6.2
tqdm                          4.64.1
traitlets                     5.7.1
twine                         4.0.1
typing_extensions             4.3.0
urllib3                       1.26.12
vermin                        1.5.1
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              1.4.1
wheel                         0.37.1
widgetsnbextension            4.0.3
xarray                        2022.3.0
yarl                          1.8.1
zipfile-deflate64             0.2.0
zipp                          3.8.1

[adamjstewart]

If you revert #654 you should also be able to reproduce the issue in CI.

[lanpa]

Thanks, I have found that I can reproduce with: PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python -Werror -c 'import tensorboardX'. This additional environment variable is also used in the test script so the deprecation message can be reproduced in CI.
Would you check if you have PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION set in your environment? By default, PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION is set to cpp, which should be faster.

[adamjstewart]

I do not have that environment variable set, but both values error for me:

$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python -Werror -c 'import tensorboardX'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/__init__.py", line 5, in <module>
    from .torchvis import TorchVis
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/torchvis.py", line 10, in <module>
    from .writer import SummaryWriter
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/writer.py", line 16, in <module>
    from .comet_utils import CometLogger
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/comet_utils.py", line 7, in <module>
    from .summary import _clean_tag
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/summary.py", line 12, in <module>
    from .proto.summary_pb2 import Summary
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/summary_pb2.py", line 16, in <module>
    from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/tensor_pb2.py", line 16, in <module>
    from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 18, in <module>
    DESCRIPTOR = _descriptor.FileDescriptor(
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 1034, in __init__
    _Deprecated('FileDescriptor')
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 97, in _Deprecated
    warnings.warn(
DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python -Werror -c 'import tensorboardX'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/__init__.py", line 5, in <module>
    from .torchvis import TorchVis
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/torchvis.py", line 10, in <module>
    from .writer import SummaryWriter
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/writer.py", line 16, in <module>
    from .comet_utils import CometLogger
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/tensorboardX/comet_utils.py", line 5, in <module>
    from google.protobuf.json_format import MessageToJson
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/json_format.py", line 54, in <module>
    from google.protobuf.internal import type_checkers
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/type_checkers.py", line 51, in <module>
    from google.protobuf.internal import decoder
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/decoder.py", line 87, in <module>
    from google.protobuf.internal import encoder
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/encoder.py", line 71, in <module>
    from google.protobuf.internal import wire_format
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/internal/wire_format.py", line 36, in <module>
    from google.protobuf import descriptor
  File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/pyext/__init__.py)

[lanpa]

What about PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python -Werror -c "import google.protobuf"?

[adamjstewart]

No errors there (tried google, google.protobuf, and google.protobuf.pyext).

[lanpa]

Hi, I think the root cause is related to the protobuf in your spack env. Can you try PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python -Werror -c "from google.protobuf import __version__; print(__version__);from google.protobuf.pyext import _message" ?

According to File "/Users/Adam/.spack/.spack-env/view/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module> from google.protobuf.pyext import _message This should trigger an error and print the actual protobuf used.

[adamjstewart]

Yes, __version__ is 3.20.1, and the latter import shows the error.

[adamjstewart]

Note that Spack installs separate packages for protobuf (cpp) and protobuf (python). My cpp implementation is version 3.21.7, while my python implementation is 3.20.1. Should those always match?

[lanpa]

Thank you for telling me that your Spack env have different versions of protobuf installed. By convention, the major should be matched. However, as indicated in
https://developers.google.com/protocol-buffers/docs/news/2022-05-06
https://github.com/protocolbuffers/protobuf/releases/tag/v21.0-rc1
I think 3.21.7 (cpp) corresponds to the interface of 4.21.7 (python)
so the version mismatch of the python API feels like a major one. (3.20.1 -> 4.21.7)

[adamjstewart]

Well there isn't a 4.X.Y (cpp) so I don't think it's possible to match major versions there. Do you mean minor versions?

[lanpa]

Is it possible to uninstall both and just pip install protobuf==3.20.1?

[adamjstewart]

I don't use pip. I can install different versions of cpp/python if necessary, just let me know which versions you want me to try.

I'm worried we're getting stuck down a rabbit hole here. If the issue also happens in CI, then it's not an issue with my installation, it's an issue with tensorboardX, right? Can we use the newer API instead?

[lanpa]

Please make sure that protobuf 3.20.1 (python) is installed and it's using the 3.20.1 (cpp) backend.
In CI, the deprecation warning were shown because I forced the script to use the python backend. By default, the cpp backend should be used. As your experiment indicates, the error is more related to protobuf rather than tensorboardX. If you are interested in the new API, you can change the version and compile with

tensorboardX/compile.sh

Line 6 in 8734a21

DESIRED_PROTO_VERSION="3.8.0"