Feature request: Add a count of Apex tests failed in test summary

[jongpie]

Currently, when running Apex tests, the output has a summary that includes "tests ran", "pass rate", and "fail rate". But there doesn't seem to be anything that shows the number of test failures (only the percentage). For large orgs/projects with a lot of tests, the percentages aren't quite enough to have a sense of the state of a failed test run.

For example, here's a copy of a test run from I did today. It ran 1,126 test methods, and 1 of the methods failed.

• The summary shows 0% for Fail Rate - with rounding, that's correct (1 / 1126 --> 0.088% --> round down to 0%), but it's still conceptually a little strange to see a failure with Fail Rate 0% / Pass Rate 100%

=== Test Summary
NAME                 VALUE
───────────────────  ─────────────────────────────
Outcome              Failed
Tests Ran            1126
Pass Rate            100%
Fail Rate            0%
Skip Rate            0%
Test Run Id          707DL00003QoKaK
Test Setup Time      3514 ms
Test Execution Time  221752 ms
Test Total Time      225266 ms
Org Id               00DDL00000Cs7Ka2AJ
Username             test-fsh8qae2ofvh@example.com

In this example, 5 test methods failed. The output is essentially the same as the output above with 1 test failure - due to rounding, this is also reported as Fail Rate 0% (5 / 1126 --> 0.444% --> round down to 0%).

=== Test Summary
NAME                 VALUE
───────────────────  ─────────────────────────────
Outcome              Failed
Tests Ran            1126
Pass Rate            100%
Fail Rate            0%
Skip Rate            0%
Test Run Id          707DL00003QoKkj
Test Setup Time      3959 ms
Test Execution Time  200419 ms
Test Total Time      204378 ms
Org Id               00DDL00000Cs7Ka2AJ
Username             test-fsh8qae2ofvh@example.com

Could something like a Tests Failed line be added in the test results summary? Example below - this would be really handy for getting a quick sense of "how badly did I break things"

=== Test Summary
NAME                 VALUE
───────────────────  ─────────────────────────────
Outcome              Failed
Tests Ran            1126
Tests Failed         5
Pass Rate            100%
Fail Rate            0%
Skip Rate            0%
Test Run Id          707DL00003QoKaK
Test Setup Time      3514 ms
Test Execution Time  221752 ms
Test Total Time      225266 ms
Org Id               00DDL00000Cs7Ka2AJ
Username             test-fsh8qae2ofvh@example.com

[WillieRuemmele]

Hey @jongpie - we've been looking at redoing our test output, and will add this to our list. Are you seeing these when running sf apex * or as part of a sf project deploy start --test-level deployment?

[jongpie]

@WillieRuemmele sounds great! The output examples in my original message were from running sf apex run test --test-level RunLocalTests.

For comparison, I just ran sf project deploy validate --source-dir ./force-app/ --test-level RunLocalTests, and the output does indeed seem to include counts for Passing and Failing (shown below). And interestingly, it doesn't include the percentages at all - only the counts. Personally, I prefer the counts, but having both counts + percentages could be nice.

Test Results Summary
Passing: 1115
Failing: 1
 »    least 1% test coverage is required
Error (FailedValidationError): Failed to validate the deployment (0AfDL00002yvKsd0AE). Due To:
SomeTrigger - Test coverage of selected Apex Trigger is 0%, at least 1% test coverage is required

SomeClassTest.someTestMethod- System.NullPointerException: Attempt to de-reference a null object