API client sample that respects rate limits

[adeutscher]

With a recent update to the API template, I recommended backend processing as being the specialty of this JobWorker template. It's far from the first time I've mentioned another template, but this time it was a BIG neon sign that said "hey, a JobWorker implementation should absolutely be accessing an API implementation to perform long-running tasks".

Action items:

• Adapt OAuth-aware Bar connector project from API Template
  • Like the API template, have a Core and an Implementation subproject.
  • Local Compose stack should absolutely have all the credential settings for connecting to the "Bar" API, including credentials in SSM and Key Vault, and the wiremock-bar service/config files.
    • Copy the OAuth rotation scripts, but this time place them in scripts/wiremock-bar/ subdirectory of test/local/
    • Implement a generic BarException, and adopt our CouldBeTransient property pattern from elsewhere in the repo.
    • Update BarUnauthorizedException and BarUnavailableException to inherit BarException and have hardcoded values for CouldBeTransient.
    • Rename BarUnavailableException to BarTemporarilyUnavailableException.
    • BarTemporarilyUnavailableException should inherit from BarException by way of the abstract BarReasonToWaitException. BarReasonToWaitException should have a nullable public property called RetryAfter.
    • Add BarRateLimitedException, inheriting from BarReasonToWaitException. Document in comments that a BarRateLimitedException suggests a rate limit response from the underlying service.
    • Update test/local/ README documentation to include setting/unsetting of Key Vault-friendly secret paths when prepping environment variables.
  • The main enhancement to this version of the Bar connector is that this version should respect the BarReasonToWaitException indefinitely. If the client receives a reason to wait from an attempt, then the client should respect the value of the RetryAfter property and delay using ISleepService, with a configurable fallback should RetryAfter be null that defaults to a constant value of 15 seconds should that configurable fallback be null.
    • Should continue retrying/respecting-wait-exceptions/waiting indefinitely. Note in comments that cancellation in this template is the Core job worker project configuration's problem. It's the JobWorker's duty to keep trying respectfully if rate limited.
    • For consistency, use Polly v8 pipeline for retries. Use a wrapper method that reuses the ResiliencePipeline within the BarConnector service implementation to recycle ResiliencePipeline.
    • Mention in comments that the Bar connector is a stand-in for an API template client and to refer to docs/bar-connector.md for last-mile instructions.
  • Update JobLogicRunner implementation to invoke Bar service after sleeping, just to make it feel included.
• Create a reference document at docs/bar-connector.md. In it, mention that:
  • This is meant as a placeholder to an OAuth API, such as our API Template.
    • Mention that this sample connector assumes OAuth authentication. For adopting static keys, consider adjusting plan using the request handler outlined in the Foo connector of the API Template.
  • Client will respect BarReasonToWaitException exceptions indefinitely as described above.
  • Authentication in this JobWorker template was made with OAuth in mind due to personal preference for projects and because it support the authorization that the API Template is set up for. If you are planning to use this for an API that instead uses static keys then the request handler implementation should be pivoted to be more like the FooConnector in the API Template.
  • Last-mile instructions that this general template cannot cover:
    • If you are using the API Template, then:
      • Adjust your API implementation to publish an interop package to a NuGet repository such as Azure DevOps, GitHub, or Nexus (include links to all 3).
      • Rename Bar.Core and Bar.Implementation projects as is appropriate for your target API.
      • Reference the interop NuGet package into your implementation's renamed Bar.Implementation project.
      • Write wrapper clients for the relevant clients from the interop NuGet that can translate thrown instances of SwaggerException to a generic BarException.
        • BarException translation should judge the status code in the SwaggerException to set a value to the CouldBeTransient. Recommend to abstract this decision into an exception arbiter service with a method called CouldSwaggerExceptionBeTransient.
          • The exception to this translation is a 429 status code, which should specifically be a BarRateLimitedException with the value of the Retry-After header.
      • Adjust your client factory to return the previously-discussed wrapper client.
    • If you are not using the API Template, then:
      • Rename Bar.Core and Bar.Implementation projects as is appropriate for your target API.
      • Adjust clients for HTTP requests from the subject API to translate non-successful status codes to a generic BarException.
        • BarException translation should judge the status code to set a value to the CouldBeTransient. Recommend to abstract this decision into an exception arbiter service with a method called CouldSwaggerExceptionBeTransient.
          • The exception to this translation is a 429 status code, which should specifically be a BarRateLimitedException with the value of the Retry-After header.
    • Rename classes whose names begin with Bar as is appropriate for your target API.

[adeutscher]

Went more than a little ham on the detail on this one. Like with some similarly-detailed issues, my hope is that the time spent writing a super-long issue will be made up for by feeding it into a prompt to break ground on this.

[adeutscher]

Kicked off the prompt already because I didn't see a reason why not to, then spent a bit of time reviewing the output.

• Happy with the stuff that I've reviewed so far.
• Local testing is a next-session task.
• Still need to review documentation.
• Still need to make unit tests.
• At this point I should really make solution sub-folders, but will probably leave that for a follow-up branch for silly blast-radius reasons.

[adeutscher]

Testing looking positive-ish so far, but still some points for improvement.

• The Bar connector is... certainly doing its job in retrying forever if it receives a reason to wait. However, the problem is that it's currently always retrying forever. That needs fixing.
• Updated local-testing README to address some drift that's happened over time.

[adeutscher]

• Bar connector grief ended up being a Wiremock setup issue. I jumped the gun when I described it as running forever, it was the inner retry loop.
• Updated job logic runner to have a special waiting carve-out. A requested sleep time of 404 or 429 will only sleep for 1 second before making a Bar callout.
• All of this is grand and functional, but I'm also running into the same issue that drove me to improve the monitor threads: Making HTTP callouts with Verbose-level logging is very noisy. Adjusting my approach so that accessing the Bar service in this example is explicitly configurable. Sending a job without enabling it will make the logic runner behave as it did before these changes.

[adeutscher]

New testing is looking good:

• Configurable Bar connector use works fine.
• Getting a 429 successfully waits for (functionally) forever.

[adeutscher]

Doing a solution-wide cleanup sweep, then we should be clear to set up a PR for this.