500 Error Status on all GET Endpoints

[akramekhairi]

Any request to the API produces: Error: response status is 500 and Internal Server Error

[greysonmrx]

Any request to the API produces: Error: response status is 500 and Internal Server Error

Same here

[kryptografisch]

run with docker image of khalilosx/transfermarkt-api:latest

INFO:    0.0.0.0.1:39740 - "GET /competitions/L1/clubs HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
...
AttributeError: 'NoneType' object has no attribute 'xpath'

full log

docker run -it -p 8000:8000 khalilosx/transfermarkt-api:latest
INFO: Will watch for changes in these directories: ['/app']
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: Started reloader process [7] using WatchFiles
INFO: Started server process [9]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 172.17.0.1:39740 - "GET /competitions/L1/clubs HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application

• Exception Group Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
  | yield
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 189, in call
  | response_sent.set()
  | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 815, in aexit
  | raise BaseExceptionGroup(
  | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
  | result = await app( # type: ignore[func-returns-value]
  | File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
  | return await self.app(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in call
  | await super().call(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 113, in call
  | await self.middleware_stack(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 187, in call
  | raise exc
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 165, in call
  | await self.app(scope, receive, _send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 189, in call
  | response_sent.set()
  | File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
  | self.gen.throw(typ, value, traceback)
  | File "/usr/local/lib/python3.9/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
  | raise exc
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 187, in call
  | response = await self.dispatch_func(request, call_next)
  | File "/usr/local/lib/python3.9/site-packages/slowapi/middleware.py", line 124, in dispatch
  | return await call_next(request)
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 163, in call_next
  | raise app_exc
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 149, in coro
  | await self.app(scope, receive_or_disconnect, send_no_error)
  | File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in call
  | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  | raise exc
  | File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  | await app(scope, receive, sender)
  | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 715, in call
  | await self.middleware_stack(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 735, in app
  | await route.handle(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 288, in handle
  | await self.app(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 76, in app
  | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  | File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  | raise exc
  | File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  | await app(scope, receive, sender)
  | File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 73, in app
  | response = await f(request)
  | File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 301, in app
  | raw_response = await run_endpoint_function(
  | File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
  | return await run_in_threadpool(dependant.call, **values)
  | File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
  | return await anyio.to_thread.run_sync(func, *args)
  | File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync
  | return await get_async_backend().run_sync_in_worker_thread(
  | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
  | return await future
  | File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
  | result = context.run(func, *args)
  | File "/app/app/api/endpoints/competitions.py", line 21, in get_competition_clubs
  | tfmkt = TransfermarktCompetitionClubs(competition_id=competition_id, season_id=season_id)
  | File "", line 8, in init
  | File "/app/app/services/competitions/clubs.py", line 27, in post_init
  | self.raise_exception_if_not_found(xpath=Competitions.Profile.NAME)
  | File "/app/app/services/base.py", line 129, in raise_exception_if_not_found
  | if not self.get_text_by_xpath(xpath):
  | File "/app/app/services/base.py", line 180, in get_text_by_xpath
  | element = self.page.xpath(xpath)
  | AttributeError: 'NoneType' object has no attribute 'xpath'
  +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 189, in call
response_sent.set()
File "/usr/local/lib/python3.9/contextlib.py", line 137, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 187, in call
response = await self.dispatch_func(request, call_next)
File "/usr/local/lib/python3.9/site-packages/slowapi/middleware.py", line 124, in dispatch
return await call_next(request)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
result = context.run(func, *args)
File "/app/app/api/endpoints/competitions.py", line 21, in get_competition_clubs
tfmkt = TransfermarktCompetitionClubs(competition_id=competition_id, season_id=season_id)
File "", line 8, in init
File "/app/app/services/competitions/clubs.py", line 27, in post_init
self.raise_exception_if_not_found(xpath=Competitions.Profile.NAME)
File "/app/app/services/base.py", line 129, in raise_exception_if_not_found
if not self.get_text_by_xpath(xpath):
File "/app/app/services/base.py", line 180, in get_text_by_xpath
element = self.page.xpath(xpath)
AttributeError: 'NoneType' object has no attribute 'xpath'

[kryptografisch]

With some help of Claudi:
Short answer: this isn't your bug — it's Transfermarkt blocking/rate-limiting the scraper's request, and the library doesn't handle that gracefully.

My Workaround: Connect from a different IP that is not blocked, If you use a VPN try to change locations.