fix: replace urlretrieve in getRemoteFile

[afuetterer]

Failing unit tests

• see test: remove failing unit test #447
• FAILED tika/tests/test_tika.py::CreateTest::test_remote_jpg - urllib.error.HTTPError: HTTP Error 403: Forbidden
• FAILED tika/tests/test_tika.py::CreateTest::test_remote_pdf - urllib.error.HTTPError: HTTP Error 502: Bad Gateway

Tasks

• fix broken url to pdf file
• decide how to deal with urlretrieve
• rebase
• remove unit test 1 commit
• remove deps commit

[chrismattmann]

As per your comment in #442:

1. Let's remove the test_ssl_link.py test. Not sure what it's testing either and it has become outdated regardless.
2. Header issue. Please update.
3. Any old PDF file should do....

2. The header issue is not in the test, but in the library:
   

   tika-python/tika/tika.py

   Line 781 in beeaaaa

   urlretrieve(urlOrPath, destPath)

3. Same here I think. The broken pdf url is just step 1, I think if there is a valid url, it will still return 403

Got it, so we have to fix urlRetrieve then? Perhaps we should add a unit test for it too.

[afuetterer]

As I said above, replacing the url to a remote pdf, does help a bit.
Now we got from
FAILED tika/tests/test_tika.py::CreateTest::test_remote_pdf - urllib.error.HTTPError: HTTP Error 502: Bad Gateway
to
FAILED tika/tests/test_tika.py::CreateTest::test_remote_pdf - urllib.error.HTTPError: HTTP Error 403: Forbidden

The error is in getRemoteFile.

It is a common issue: https://www.google.com/search?q=python+urlretrieve+403+forbidden

Maybe use custom headers here or switch to using requests.

[afuetterer]

I suggest splitting the issues discovered during the work on this PR into smaller problems, that can be reviewed individually and merged faster.

The "main issue" discovered in this PR is the broken getRemoteFile. The problem is in tika-python's library, not the unit tests. Agreed?

How to proceed from here?

Write and use a new helper function? Something like _urlretrieve() It should use requests internally and send a user-agent of "tika-python"? Streaming response, due to potentially large files?

What do you think?

[chrismattmann]

I suggest splitting the issues discovered during the work on this PR into smaller problems, that can be reviewed individually and merged faster.

The "main issue" discovered in this PR is the broken getRemoteFile. The problem is in tika-python's library, not the unit tests. Agreed?

How to proceed from here?

Write and use a new helper function? Something like _urlretrieve() It should use requests internally and send a user-agent of "tika-python"? Streaming response, due to potentially large files?

What do you think?

Yes, perfect let's write a new _urlRetrieve that:

1. uses requests internally
2. uses tika-python as the User-Agent
3. allows for streaming, etc.

[afuetterer]

Please have a look at #446 and #447 in the meantime. These changes have been extracted from this PR.

[chrismattmann]

Merged #446

[chrismattmann]

Merged #447

[afuetterer]

Please check out the proposed helper function. Tests are passing.

[afuetterer]

I just realized, that this can be removed. I had a progress bar in mind. But without that, no need for keeping track of the downloaded bytes.

[chrismattmann]

LGTM!!