diff --git a/manual/modules/lib/pages/testing.adoc b/manual/modules/lib/pages/testing.adoc index a0a56be..6a563b3 100644 --- a/manual/modules/lib/pages/testing.adoc +++ b/manual/modules/lib/pages/testing.adoc @@ -1993,6 +1993,7 @@ test-impossible-zmachine: impossible.zblorb impossible.regtest $(REGTEST) --interpreter dfrotz --game $^ ---- +[#make-test] === Testing With `make` Using `make` to automate your testing is fairly straightforward. First @@ -2103,6 +2104,7 @@ clean: .PHONY: all test clean ---- +[#releases] === Building Releases Finally, the main point of using `make` is to build your game so that @@ -2175,6 +2177,131 @@ you're using lots of extensions, need to customize your build, or want a different project layout than `dgt` requires, you might need to stick with `make`. See the `dgt` documentation for details. +[#beta] +== Exploratory Testing + +Thus far, we have covered _regression testing_: testing that verifies +that our game is working the way that we intended, and that we didn't +break anything with our most recent changes. Unit tests and +integration tests are both types of regression tests. + +There's another kind of testing that's equally important in +interactive fiction: _exploratory testing_. When we write unit tests +or `regtest.py` cases, we're testing things that we've thought +about in advance, and which have expected answers that we already know +and can test for. But interactive fiction is about creating a simulated +world in which the player is given the illusion of a lot of freedom to +navigate, and to try lots of different, possibly unlikely, things. + +As anyone who has ever been the game master for a tabletop role-playing +game will tell you, players will inevitably find ways to attempt wild +ideas that stray far from anything you thought up. If a player tries +something that makes sense, and the game doesn't respond accordingly, +it's going to look like a bug. Conversely, players may fail to find +solutions that we intended for them to find, because they phrase what +they want to do in a synonym of what we programmed, which we never +added to the parser's grammar. To take an example from this chapter, +release 2 of _Cloak of Darkness_ failing to understand `READ MESSAGE` +at the end of the game is a perfect example of both of these phenomena. + +The traditional way to see into these blind spots, and to anticipate +the things that players -- especially stuck players -- might try, is +to recruit some friendly players to play your game, and to try to +break it. When those players are our friends or family, that's called +_alpha testing_, and when they're complete strangers from over the +Internet, it's called _beta testing_. "Alpha" and "beta" are terms +that date back to before the rise of modern software engineering +methods with automated tests, when manual testing, done after +completion of all of the code, was the only way that software was +tested. Alpha testing happened "in house", performed by the QA +department of the company writing the software; beta testing took +place externally, with potential customers being given a free copy of +nearly-ready software in exchange for submitting bug reports when +something broke. The process was as rickety as it sounds, and was a +big part of the reason why so much technical literature from the 1990s +refers to a "software crisis." + +If you've been using the facilities described in this chapter to build +an effective suite of unit and integration and skein tests, your code +should be of much better quality than commercial IF games back in +their heyday were when they went out for beta testing. But it's still +worthwhile to engage beta testers, specifically to find things that +you didn't think of. + +Once you've recruited your testers, expect them to need some time to +play through your game, solve it, and spend some time banging on it to +try to elicit unusual behaviours. For a "competition-length" game +that's aimed at being solvable, or at least ready for review, after +two hours, four to six weeks is a reasonable time frame; a "full +length" game comparable to commercial IF will take longer. This means +that you need to be otherwise done with your game at least a month +before the deadline, if you're writing to a deadline such as a +competition submission window. Literature about software engineering +from back in the manual testing days refers to the testing and +debugging phase of a project taking _five times_ as long as writing +the code did. Writing good automated tests, and fixing the problems +that they reveal, will do much of that work much faster than it would +have taken in past decades, but it can still take as long or longer to +get through your beta test as it did to write your game. Beta testers +are volunteer hobbyists, not full-time professionals, and they need +enough time to get through all of your game, to try unlikely things in +all phases of it, and to retest new versions that incorporate changes +that the beta testing turns up. Don't try to rush your beta testing -- +the longer it goes, the better your game will be. + +Above all, make sure that you track all of the feedback that you get +from your testers, and prioritize it for action. Your exploratory +testing program is only as good as the improvements to your code that +come out of it. If you host your code base on a site like Github +(which hosts public repositories for free), it has a built-in issue +tracker that you can use, and even point your testers at to file +issues. But something as simple as a spreadsheet or even a text file +can help you keep track of what needs to be done. + +Remember, also, that some interpreters support transcripts. +Transcripts are a relic from the days when _Colossal Cave_ and _Zork_ +were played on mainframe line printers, but they're handy for tracking +exploratory testing. Encouraging your testers to play with +transcripting on, and to send you the transcript files, can reveal a +lot about how players who don't think like you will approach your game. + +Not everything that your testers file is necessarily a bug, or even +undesired behaviour. If you have enough testers, you might find that +they disagree with each other about the right thing in a number of +cases. Some issues will be actual bugs, or critically missing features +that should be added, and for those, you'll want to also write +automated tests to ensure that other changes that you make later don't +bring back the problems that you just fixed. + +At some point, the rate at which your testers file issues against your +code will drop off, you'll have fixed all of the issues that you've +deemed to need fixes (and which are worth fixing), and your game will +be ready for release. Congratulations! You've made a fully tested +game, which you can submit to a competition or release to the general +public. Getting good games into the hands of players is the whole +reason why Dialog exists. + +[#finding-testers] +=== Finding Beta Testers + +Where do you find beta testers? One common place is on +link:https://intfiction.org/[the Interactive Fiction Community Forum], +which has a whole category for +link:https://intfiction.org/c/playing/beta-testing/24[beta testing +requests]. Once you've implemented all of your game, and it passes all +of your tests, you can put out a call there, and elsewhere, for +players to beta test your game. If you look at some of the other +discussion threads in that forum, you'll see the kind of detail that's +typical to attract players to want to be the first to try your game. + +Do be sure to treat your testers well! At a minimum, + +- give them enough time to do a good job of testing, and for testing + subsequent versions, +- remember that they're volunteers who are helping you, and avoid + getting angry or impatient with them, and +- be sure to publicly thank them in your game's credits. + [#summary] == Summary @@ -2195,7 +2322,8 @@ Here's a summary of the advice from this chapter: your automated tests with exploratory (beta) testing. - Automate your testing process with `make` or `dgt`. - Test on all platforms for which you'll be releasing: Z-machine, web, - and vintage computing hardware. + and vintage computing hardware (the Commodore 64, and any other old + computers that later versions of Dialog may support). - Listen to your tests. If one fails, it's telling you something important. - Don't release a game for which any of the tests have failed.