Cached service addresses are not invalidated and re-resolved when the address changes following a node/app restart or service upgrade

[dungimon]

Repro Steps:

1. Start the ServiceFabricGateway app and verify in the Service Fabric explorer.
2. Start another fabric hosted service and verify in the Service Fabric explorer.
3. Verify requests are correctly forwarded to this service via the ServiceFabricGateway app.
4. Stop and re-start this service and verify different ports have been assigned in the Service Fabric explorer.
5. Send requests to the service via the ServiceFabricGateway app.

It's during the final step you'll see the ServiceFabricGateway return a 500 response code with the exception message "No connection could be made because the target machine actively refused it [::1]:32001". The exception message should prove that the ServiceFabricGateway has cached the first resolved service address and is still forwarding requests to the old IP and port.

The ServiceFabricGateway app must be changed so that when a request is forwarded to an old cached address and fails to connect it must retry with the following steps:

• Invalidate the cache
• Resolve the address via the Service Fabric Naming Service and cache the address
• Forward the request

This will ensure requests are forwarded correctly in the event of application and node restarts and application upgrades.

The following links provide suggestions on how this can be achieved with the IExceptionHandler interface:

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-communication
http://bartwullems.blogspot.co.uk/2016/09/azure-service-fabric-service-endpoint.html
https://dzimchuk.net/implementing-a-rest-client-for-internal-communication-in-service-fabric