Develop test result script

[huitema]

Develop a simple script that presents the results of many tests.

The tests are grouped in categories, as coded in the script. This will probably evolve over time, as we add more tests.

In each category, we present a set of tables tracking either the execution time for "batch" tests, or the measure latency for "media" tests. In fact, we will present two tables:

• for batch tests, a table displaying the average time and another displaying the 90th percentile time.
• for media test, a table displaying the average frame latency and another displaying the 90th percentile of the max latency as found in 100 tests.

Each line will present three columns: the time or latency measured for C4, then for the same test unsing BBR and Cubic.

This presentation answers a critic that we heard during IETF 125:

• we should present the results of the 100 tests
• we should compare the C4 results against the results of BBR and Cubic.

[huitema]

Here are the first results, as measured with the current version of C4:

Statistics

Here is a collection of statistics on all test cases.

simple

Here the statistics for the simple test cases.

average time for simple tests

average time for simple tests	c4	bbr	cubic
alone	4647943	4688512	4476998
low_and_up	7926859	7506316	8049855
drop_and_back	7892651	7627307	7628463
blackhole	5628255	5811376	5695208
satellite	7660773	7472446	6704244

top 90% time for simple tests

top 90% time for simple tests	c4	bbr	cubic
alone	4806735	4700741	4521935
low_and_up	7929953	7511559	8071886
drop_and_back	7896874	7632814	7632383
blackhole	5629067	5815444	5699325
satellite	7661570	7472352	6704246

wifi

Here the statistics for the wifi test cases.

average time for wifi tests

average time for wifi tests	c4	bbr	cubic
wifi_bad	4095173	5316761	4060841
wifi_fade	5154116	5406372	5367521
wifi_suspension	4562814	4615882	4600515

top 90% time for wifi tests

top 90% time for wifi tests	c4	bbr	cubic
wifi_bad	4663533	7321342	4319893
wifi_fade	5479860	5572588	5581560
wifi_suspension	4563522	4616912	4600827

media

Here the statistics for the media test cases.

average av_latency for media tests

average av_latency for media tests	c4	bbr	cubic
media	33511	33427	33512
media10	45214	45008	47760
media_short_long	101035	133963	100771
media_wb	79290	85020	78309
media_wf	83584	85481	81348
media_ws	22844	21645	22492

top 90% max_latency for media tests

top 90% max_latency for media tests	c4	bbr	cubic
media	43453	43453	43453
media10	71128	71128	92163
media_short_long	117983	334491	110858
media_wb	270360	315884	265915
media_wf	289548	365087	304576
media_ws	197821	195521	197821

compete

Here the statistics for the compete test cases.

average time for compete tests

average time for compete tests	c4	bbr	cubic
vs_bbr	2945713	4502793	2852396
vs_c4	4446590	6314435	6198056
vs_cubic	3648071	3637191	5266135
wifi_bad_bbr	7487152	7909440	8354502
wifi_bad_c4	9903181	10548742	9243776
wifi_bad_cubic	8550470	8828130	9734756

top 90% time for compete tests

top 90% time for compete tests	c4	bbr	cubic
vs_bbr	2963475	4569995	2873067
vs_c4	4955209	6748970	6606557
vs_cubic	3971537	3966392	5578272
wifi_bad_bbr	11906585	12292138	13517663
wifi_bad_c4	12316529	12599665	12291643
wifi_bad_cubic	11211242	12016981	13410701

others

Here the statistics for the other test cases.

average time	c4	bbr	cubic
after_c4	5963997
alone_200	1174457
before_c4	3100256
ecn	4490759
ecn_bbr	13280026
ecn_c4	11591693
ecn_cubic	8615196
media_600fr	20057739
short_long	17442217
vs_bbr_lg	17203354
vs_bbr_lg2	17484790
vs_c4_lg	21166353
vs_c4_lg2	21164105
vs_cubic_lg	17638810
vs_cubic_lg2	17381870