fix: percent-encode storage resource paths on the wire (#124) - #125
fix: percent-encode storage resource paths on the wire (#124)#125andrii-novikov wants to merge 3 commits into
Conversation
DIAL Core parses every resource path (in the URL and in JSON bodies such as
sourceUrl/destinationUrl) as a percent-encoded URL. A raw reserved character
(space, #, ?, [, ...) either got truncated by urlparse or reached Core
unencoded, making 'new URI(...)' throw and Core answer 404/500.
Add percent_encode_resource_url() plus DialStorageResourceMixin
.get_encoded_api_path(), which decode-then-encode each segment so decoded
('my file.txt') and already-encoded ('my%20file.txt', as returned by the API)
inputs converge without double-encoding. Apply it across files (upload,
download, delete, move_to, copy_to, get_metadata), the shared metadata.get
(also covers conversations and direct low-level calls), prompts, and
resource_permissions.grant.
Drop the _prepare_file_download wrapper; encode the url and decode the returned filename directly in the shared (files-only) _prepare_download_request.
d83b4f5 to
c6b6848
Compare
| StorageResourceType = Literal["files", "conversations", "prompts"] | ||
|
|
||
|
|
||
| def percent_encode_resource_url(url: str) -> str: |
There was a problem hiding this comment.
Let's rename it to _percent_encode_relative_url since there is nothing in the implementation of the function that's specific to DIAL resources, and I don't see why we need to make it public.
| Encodes before parsing so reserved characters (notably ``#`` and ``?``, | ||
| which ``urlparse`` would otherwise drop as fragment/query) survive. | ||
| """ | ||
| return self.get_api_path(percent_encode_resource_url(url)) |
There was a problem hiding this comment.
Let's instead move the encoding to the very place where url is converted into the resource: safe_parse_storage_resource.
Remove get_encoded_api_path, since get_api_path will do the encoding for you.
| "resourcePermissions": [ | ||
| {"url": url, "permissions": permissions} | ||
| { | ||
| "url": percent_encode_resource_url(url), |
There was a problem hiding this comment.
Let's extract this into a grant_body helper:
{
"resourcePermissions": [
{
"url": percent_encode_resource_url(url),
"permissions": permissions,
}
for url in resources
],
"receiver": receiver,
}
Reuse in sync and async versions. And use get_encoded_api_path just like we did in _move_copy_body.
| """ | ||
| return self.get_storage_resource(url).api_path | ||
|
|
||
| def get_encoded_api_path(self, url: str) -> str: |
There was a problem hiding this comment.
Let's extend the type of the argument to url: str | PurePosixPath; this will eliminate a lot of str(...) conversions downstream.
Applicable issues
Description of changes
DIAL Core parses every storage resource path — both in the request URL (
upload/download/delete/get_metadata) and in JSON body fields (sourceUrl/destinationUrlfor move/copy,urlfor permission grants) — as a percent-encoded path (each segment is URL-decoded viaUrlUtil.decodePath, which runsnew URI(segment)).A raw reserved character in a filename therefore broke things:
#/?were truncated by the client'surlparseinget_api_pathbefore the request was even built → wrong path → 404.[/]/ space and friends reached Core unencoded →new URI(...)throwsURISyntaxException, which Core wraps in a bareRuntimeException→ 500.The URL-slot ops only worked by accident (httpx encodes the URL path on the wire); the body-slot ops (
move_to/copy_to) had nothing encoding them at all.Fix:
percent_encode_resource_url()(stateless helper) andDialStorageResourceMixin.get_encoded_api_path()inhelpers/storage_resource.py.my file.txt) and an already-encoded one (my%20file.txt, as returned by the API) converge to the same wire form without double-encoding. Absolute URLs (always encoded by the API) pass through untouched.files.py— upload, download, delete, move_to, copy_to, get_metadatametadata.py— the sharedget(covers conversations and direct low-levelclient.metadata.get(...)calls)prompts.py— save, get, delete, get_metadataresource_permissions.py—grantbody urlsVerification:
files.get_metadataand low-levelmetadata.get.nox -s formatclean.Notes:
application/deployments/model/toolset) were left unchanged — they interpolate deployment/model names, not storage paths, and conventionally contain no reserved characters.%sequence can't be expressed from a decoded path (extremely rare).Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.