[JS] Separate outputting logic from benchmarking

[GerkinDev]

Hi, I'm willing to use the package to run benchmarks but its API is a bit unusual.

It would be convenient IMO to separate outputting logic from benchmarking with something like this:

import { bench, group, run } from 'mitata';
import { summary, lineplot, exportReport } from 'mitata/reporting'
import { setTimeout } from 'node:timers/promises';

group(() => {
  bench('Sample sleep $duration', async (state) => {
    const duration = state.get('duration');
    for (const _sample of state) {
      await setTimeout(duration);
    }
  }).range('duration', 1, 16, 2);
});

const results = await run({});
if(process.env.CI !== 'true'){
  summary(results)
  lineplot(results)
}
exportReport(results, 'benchmark.json')

Having such setup would make it easy to generate reports from a dedicated environment, import it locally (with a fn like importReport) and do various visualization based on it.

I've seen #33 but the huge majority of data contained in the output are samples, which may or may not be useful to inspect afterhand.

Would you be OK with that kind of change ? Cheers !

[evanwashere]

I have been moving towards exposing more and more of terminal output through semi-public mitata.$ but charts currently are still driven by some stuff that's private

for exporting you can just call JSON.stringify() on await run({ format: 'mitata' | 'quiet' }) result to get same json as run({ format: 'json' })

    print(JSON.stringify(results,
      (_, v) => {
        if (!(v instanceof Error)) return v;
        return { message: String(v.message), stack: v.stack };
      }, 0));

[GerkinDev]

For the JSON output, I know, I've linked the issue where you said the same thing. But that's not answering this:

but the huge majority of data contained in the output are samples, which may or may not be useful to inspect afterhand.

I've ended up iterating the whole object to delete samples manually, dividing the output size by ~100 for a simple test benchmark

[cometkim]

Making printing and reporting logic configurable will definitely help multiple cases.

In my case, I'm running mitata in a worker, but had to monkey-patch console.log to preserve the mitata format and to pipe it to the host.

https://github.com/cometkim/unicode-segmenter/blob/4f479a4/benchmark/grapheme/perf.js#L14-L37

Otherwise, it can only wait long until all benchmarks are finished, and cannot give proper feedback on whether the benchmarks are working correctly.

[evanwashere]

there are plans to implement an option to replace the whole formatting logic for the upcoming history plugin (which will also allow rehydrating back into built-in formats and 3rd-party formatting plugins), but like i mentioned there are still some changes left for me to do before i can expose public render() function.

Making printing and reporting logic configurable will definitely help multiple cases.

In my case, I'm running mitata in a worker, but had to monkey-patch console.log to preserve the mitata format and to pipe it to the host.

https://github.com/cometkim/unicode-segmenter/blob/4f479a4/benchmark/grapheme/perf.js#L14-L37

i added an option to provide your own print function directly, and you can pass color through options too instead of patching the process await run({ colors: false, print(s) {} }) (v1.0.34 or newer)

For the JSON output, I know, I've linked the issue where you said the same thing. But that's not answering this:

but the huge majority of data contained in the output are samples, which may or may not be useful to inspect afterhand.

I've ended up iterating the whole object to delete samples manually, dividing the output size by ~100 for a simple test benchmark

there is new option to configure samples output in json await run({ json: { samples: false, debug: false } })