Skip to content

Request reusability and some upgrades - #16

Open
brosenfi wants to merge 1 commit into
3xMike:mainfrom
brosenfi:feature/reusable_infer_request
Open

Request reusability and some upgrades#16
brosenfi wants to merge 1 commit into
3xMike:mainfrom
brosenfi:feature/reusable_infer_request

Conversation

@brosenfi

Copy link
Copy Markdown

This PR is primarily about request reusability which was not previously available since the request was entirely consumed outside of the input buffers. In this update, there is a request release that is returned independently from the response future that is used to retrieve the request after release - the input buffers can now be retrieved from the released request. A request inner struct had to be created and a server argument added to infer_async in order to allow the existing request consumption pattern (now consuming just the outer shell of the request) while satisfying the borrow checker and still maintaining the lifespan being tied to the server instance. An example of the re-use has been added to README.md:

// Run inference.
let (response_fut, request_release) = request.infer_async(&server)?;

// Obtain results.
let response = response_fut.await?;

// Reuse request
let image = image::open("/data/dogs.jpg")?;
let image = image.as_flat_samples_u8();
let mut request = request_release.await?;
let _first_input = request.remove_all_inputs()?;
request
    .add_default_allocator()
    .add_input("IMAGE", Buffer::from(image))?;
let (response_fut, _) = request.infer_async(&server)?;
let response = response_fut.await?;

Note in the documented simple example of triton C-API use, it appears to me the request release futures are only awaited after the output futures are awaited - not sure this implies what the order is however - the futures in this PR are returned independently which I believe matches the previous pattern of getting the response future then immediately taking the input release. Lastly on the request reusability - the allocator is documented as a single use construct in this crate - therefore I continued this pattern and it's consumed (also with the trace - both must be reset on the request to use) - but the simple C-API example shows it as being reusable as well (would need an upgrade to do that in this crate).

I also made a couple of smaller changes:

  • Upgraded to more recent packages to address security vulnerabilities (CVE-2020-25573 in linked-hash-map pointed at by serde-hjson) and for general uplift.
  • Fixed a couple of misspellings.

This is the PR I mentioned I would open in this issue .

…VE-2020-25573 in linked-hash-map pointed at by serde-hjson) and for general uplift.  Also, made the request reusable while still maintaining the consumption pattern of infer_async through use of a new request inner.  Finally, fixed a couple of misspellings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants