Load / Perf testing of search

[donaldgray]

Below text initially copied from #14

Performance testing — load test Search API with large Text objects; validate cache behaviour and LOH pressure under realistic traffic (see instructions/cache-usage.md)

We need to determine the performance of search, particularly memory usage for larger text objects as we will be running in a memory constrained environment. Use previous search-implementations resources as baseline for tests.

Approach

• Potential issue will be searches across multiple Manifests because each Manifest has it's own search-index that needs to be loaded into memory.
  • Tests should include both search and autocomplete as these are separate objects loaded into memory (autocomplete search indexes are generally orders of magnitude smaller than for search).
• Create text-service jobs for multiple different Manifests.
• How many requests can we handle before failure/performance degradation?
• How many concurrent requests can search before encountering issues?

[christinemacdonald]

Search Service Load Testing - SEARCH ENDPOINT - 5 minute run time

Number of indexed Manifests	Number of concurrent user threads	Number of Requests Made	Average Response Time	Number of Failures	Notes
10	5	449	111	0
12	5	342	878	0
14	5	255	1970	0
15	5	245	2009	0
15	6	232	3369	0
15	7	247	3987	0
15	8	348	2618	0
15	9	348	3468	0
15	10	291	5040	2	With 15 manifests, we can get up to 10 concurrent users before we see failure
16	3	156	1853	0
16	4	170	2666	0
16	5	216	2812	3	With 16 manifests, we consistently fail with just 5/6 concurrent users
16	5	181	3300	2
16	5	136	4573	0
16	6	130	6985	3

Search Service Load Testing - AUTOCOMPLETE ENDPOINT - 5 minute run time

Number of indexed Manifests	Number of concurrent user threads	Number of Requests Made	Average Response Time	Number of Failures	Notes
16	5	378	116	0
16	7	519	109	0
16	11	826	113	0
16	15	1103	113	0
16	25	1661	131	0
16	50	3234	163	0
16	100	5177	236	0
16	200	7063	434	0
16	500	18840	852	0	ramp up 3 per second or we never reach the 500 users

[donaldgray]

These figures look okay for expected usage. All of the Manifests tested were large samples. We can monitor usage and apply auto-scaling if required. Given the above I suspect memory usage would be best metric to scale on, rather than just no. of requests.

Made a small tweak to text-service to allow more granular control of how long items are cached for should we need it. (#61)