diff options
author | Lars Wirzenius <liw@liw.fi> | 2013-06-19 20:42:58 +0100 |
---|---|---|
committer | Lars Wirzenius <liw@liw.fi> | 2013-06-19 20:42:58 +0100 |
commit | 2c288169fd38032575feb4414c039e9c1e9e8764 (patch) | |
tree | 365b3467e08cac322943ce00830dfb140fcfb77a | |
parent | 54e347e461327c09746b45d19b1d24631652da8e (diff) | |
download | cmdtest-2c288169fd38032575feb4414c039e9c1e9e8764.tar.gz |
Switch to "scenario testing"
-rw-r--r-- | NEWS | 6 | ||||
-rw-r--r-- | README | 4 | ||||
-rw-r--r-- | README.yarn | 60 | ||||
-rw-r--r-- | debian/control | 2 | ||||
-rw-r--r-- | simple.scenario (renamed from simple.story) | 12 | ||||
-rwxr-xr-x | yarn | 78 | ||||
-rw-r--r-- | yarn.1.in | 20 | ||||
-rwxr-xr-x | yarn.tests/simple.script | 2 | ||||
-rw-r--r-- | yarnlib/__init__.py | 2 | ||||
-rw-r--r-- | yarnlib/block_parser.py | 28 | ||||
-rw-r--r-- | yarnlib/block_parser_tests.py | 48 | ||||
-rw-r--r-- | yarnlib/elements.py | 8 | ||||
-rw-r--r-- | yarnlib/mdparser.py | 6 |
13 files changed, 142 insertions, 134 deletions
@@ -3,6 +3,12 @@ NEWS for cmdtest This file summarizes changes between releases of cmdtest. +Version 0.8, released UNRELEASED +-------------------------------- + +* Switch terminology to "scenario testing" from "story testing". Thanks + to Rob Kendrick for the suggestion. + Version 0.7, released 2013-06-15 -------------------------------- @@ -21,8 +21,8 @@ yarn `yarn` also black box tests Unix command line tools, but takes a different approach, where the emphasis is on verifying that the -tools works correctly in a scenario, or sequence of operations, or -what we call a "test story". `yarn` is inspired [BDD][BDD], +tools works correctly in a sequence of operations, or +what we call a "test scenario". `yarn` is inspired [BDD][BDD], behavior-driven development, and some of the implementations made by the Ruby community. diff --git a/README.yarn b/README.yarn index a117866..ae857e0 100644 --- a/README.yarn +++ b/README.yarn @@ -1,19 +1,19 @@ -README for story testing -======================== +README for scenario testing +=========================== Introduction ------------ -`yarn` is a story testing tool: you write a story describing how a +`yarn` is a scenario testing tool: you write a scenario describing how a user uses your software and what should happen, and express, using -very lightweight syntax, the story in such a way that it can be tested -automatically. The story has a simple, but strict structure: +very lightweight syntax, the scenario in such a way that it can be tested +automatically. The scenario has a simple, but strict structure: GIVEN some setup for the test WHEN thing that is to be tested happens THEN the post-conditions must be true -As an example, consider a very short test story for verifying that +As an example, consider a very short test scenario for verifying that a backup program works, at least for one simple case. GIVEN some live data in a directory @@ -24,10 +24,10 @@ a backup program works, at least for one simple case. (Note the addition of AND: you can have multiple GIVEN, WHEN, and THEN statements. The AND keyword makes the text be more readable.) -Stories are meant to be written in somewhat human readable language. +Scenarios are meant to be written in somewhat human readable language. However, they are not free form text. In addition to the GIVEN/WHEN/THEN structure, the text for each of the steps needs a computer-executable -implementation. This is done by using IMPLEMENTS. The backup story +implementation. This is done by using IMPLEMENTS. The backup scenario from above might be implemented as follows: IMPLEMENTS GIVEN some live data in a directory @@ -74,40 +74,40 @@ A test document is written in [Markdown][markdown], with block quoted code blocks being interpreted specially. Each block must follow the syntax defined here. -* Every step in a story is one line, and starts with a keyword. +* Every step in a scenario is one line, and starts with a keyword. * Each implementation (IMPLEMENTS) starts as a new block, and continues until there is a block that starts with another keyword. The following keywords are defined. -* **STORY** starts a new story. The rest of the line is the name of - the story. The name is used for documentation and reporting - purposes only and has no semantic meaning. STORY MUST be the - first keyword in a story, with the exception of IMPLEMENTS. +* **SCENARIO** starts a new scenario. The rest of the line is the name of + the scenario. The name is used for documentation and reporting + purposes only and has no semantic meaning. SCENARIO MUST be the + first keyword in a scenario, with the exception of IMPLEMENTS. The set of documents passed in a test run may define any number of - stories between them, but there must be at least one or it is a + scenarios between them, but there must be at least one or it is a test failure. The IMPLEMENTS sections are shared between the - documents and stories. + documents and scenarios. -* **ASSUMING** defines a condition for the story. The rest of the +* **ASSUMING** defines a condition for the scenario. The rest of the line is "matched text", which gets implemented by an IMPLEMENTS section. If the code executed by the implementation - fails, the story is skipped. + fails, the scenario is skipped. * **GIVEN** prepares the world for the test to run. If - the implementation fails, the story fails. + the implementation fails, the scenario fails. * **WHEN** makes the change to the world that is to be tested. - If the code fails, the story fails. + If the code fails, the scenario fails. * **THEN** verifies that the changes made by the GIVEN steps - did the right thing. If the code fails, the story fails. + did the right thing. If the code fails, the scenario fails. -* **FINALLY** specifies how to clean up after a story. If the - code fails, the story fails. All FINALLY blocks get run either when - encountered in the story flow, or at the end of the story, regardless - of whether the story is failing or not. +* **FINALLY** specifies how to clean up after a scenario. If the + code fails, the scenario fails. All FINALLY blocks get run either when + encountered in the scenario flow, or at the end of the scenario, regardless + of whether the scenario is failing or not. * **AND** acts as ASSUMING, GIVEN, WHEN, THEN, or FINALLY: whichever was used last. It must not be used unless the previous step was @@ -119,7 +119,7 @@ The following keywords are defined. an empty line (no characters, not even whitespace) between two block quoted code blocks starts a new one or not, so we resolve the ambiguity by specifiying that a code block directly following a code - block is a continuation unless it starts with one of the story + block is a continuation unless it starts with one of the scenario testing keywords. The shell commands get parenthesised parts of the match of the @@ -144,7 +144,7 @@ The following keywords are defined. Semantics: -* The name of each story (given with STORY) must be unique. +* The name of each scenario (given with SCENARIO) must be unique. * All names will be normalised before use (whitespace collapse, leading and trailing whitespace * Every ASSUMING, GIVEN, WHEN, THEN, FINALLY must be matched by @@ -152,7 +152,7 @@ Semantics: any code. * Every IMPLEMENTS may match any number of ASSUMING, GIVEN, WHEN, THEN, or FINALLY. The test runner can warn if an IMPLEMENTS is unused. -* If ASSUMING fails, that story is skipped, and any FINALLY steps +* If ASSUMING fails, that scenario is skipped, and any FINALLY steps are not run. See also @@ -169,14 +169,14 @@ TODO ---- * Add DEFINING, PRODUCING, if they turn out to be useful. -* Need something like ASSUMING, except fail the story if the +* Need something like ASSUMING, except fail the scenario if the pre-condition is not true. Useful for testing that you can ssh to localhost when flinging, for example. **DJAS**: We think this might be 'REQUIRING' and it still does not run the FINALLY group. * Consider the ordering some more. IMPLEMENTS can come anywhere - but otherwise stories are defined as: - * STORY + but otherwise scenarios are defined as: + * SCENARIO * 0+: ASSUMING * 1+: * 1+: GIVEN diff --git a/debian/control b/debian/control index 4d7bf8a..5944b5d 100644 --- a/debian/control +++ b/debian/control @@ -19,5 +19,5 @@ Description: blackbox testing of Unix command line programs cmdtest is aimed specifically at testing non-interactive Unix command line programs, and tries to make that as easy as possible. . - Also included is a "story testing" tool, yarn. + Also included is a "scenario testing" tool, yarn. Homepage: http://liw.fi/cmdtest/ diff --git a/simple.story b/simple.scenario index 995318c..8061e47 100644 --- a/simple.story +++ b/simple.scenario @@ -1,12 +1,12 @@ -A simple story test -=================== +A simple test scenario +====================== -This is a very simple story test, which exists only to test -the story test runner itself. +This is a very simple test scenario, which exists only to test +the scenario test runner itself. - STORY a simple story + SCENARIO a simple scenario -The following is the actual test in this story: +The following is the actual test in this scenario: GIVEN a clean slate WHEN nothing happens @@ -35,35 +35,37 @@ class YarnRunner(cliapp.Application): def setup(self): self.ts = ttystatus.TerminalStatus(period=0.001) self.ts.format( - '%ElapsedTime() %Index(story,stories): %String(story_name): ' + '%ElapsedTime() %Index(scenario,scenarios): ' + '%String(scenario_name): ' 'step %Index(step,steps): %String(step_name)') def process_args(self, args): - stories, implementations = self.parse_stories(args) - self.connect_implementations(stories, implementations) + scenarios, implementations = self.parse_scenarios(args) + self.connect_implementations(scenarios, implementations) - self.ts['stories'] = stories - self.ts['num_stories'] = len(stories) - logging.info('Found %d stories' % len(stories)) + self.ts['scenarios'] = scenarios + self.ts['num_scenarios'] = len(scenarios) + logging.info('Found %d scenarios' % len(scenarios)) start_time = time.time() - failed_stories = [] - for story in stories: - if not self.run_story(story): - failed_stories.append(story) + failed_scenarios = [] + for scenario in scenarios: + if not self.run_scenario(scenario): + failed_scenarios.append(scenario) duration = time.time() - start_time self.ts.clear() self.ts.finish() - if failed_stories: + if failed_scenarios: raise cliapp.AppException( - 'Test suite FAILED in %s stories' % len(failed_stories)) + 'Test suite FAILED in %s scenarios' % len(failed_scenarios)) - print ('Story test suite PASS, with %d stories, in %.1f seconds' % - (len(stories), duration)) + print ( + 'Scenario test suite PASS, with %d scenarios, in %.1f seconds' % + (len(scenarios), duration)) - def parse_stories(self, filenames): + def parse_scenarios(self, filenames): mdparser = yarnlib.MarkdownParser() for filename in filenames: mdparser.parse_file(filename) @@ -71,56 +73,56 @@ class YarnRunner(cliapp.Application): block_parser = yarnlib.BlockParser() block_parser.parse_blocks(mdparser.blocks) - return block_parser.stories, block_parser.implementations + return block_parser.scenarios, block_parser.implementations - def connect_implementations(self, stories, implementations): - for story in stories: - for step in story.steps: - self.connect_implementation(story, step, implementations) + def connect_implementations(self, scenarios, implementations): + for scenario in scenarios: + for step in scenario.steps: + self.connect_implementation(scenario, step, implementations) - def connect_implementation(self, story, step, implementations): + def connect_implementation(self, scenario, step, implementations): matching = [i for i in implementations if step.what == i.what and re.match('(%s)$' % i.regexp, step.text, re.I)] if len(matching) == 0: raise cliapp.AppException( - 'Story %s, step "%s %s" has no matching ' + 'Scenario %s, step "%s %s" has no matching ' 'implementation' % - (story.name, step.what, step.text)) + (scenario.name, step.what, step.text)) if len(matching) > 1: s = '\n'.join( 'IMPLEMENTS %s %s' % (i.what, i.regexp) for i in matching) raise cliapp.AppException( - 'Story "%s", step "%s %s" has more than one ' + 'Scenario "%s", step "%s %s" has more than one ' 'matching implementations:\n%s' % - (story.name, step.what, step.text, s)) + (scenario.name, step.what, step.text, s)) assert step.implementation is None step.implementation = matching[0] - def run_story(self, story): - logging.info('Running story %s' % story.name) - self.ts['story'] = story - self.ts['story_name'] = story.name - self.ts['steps'] = story.steps + def run_scenario(self, scenario): + logging.info('Running scenario %s' % scenario.name) + self.ts['scenario'] = scenario + self.ts['scenario_name'] = scenario.name + self.ts['steps'] = scenario.steps datadir = tempfile.mkdtemp() - cleanup = [s for s in story.steps if s.what == 'FINALLY'] - normal = [s for s in story.steps if s not in cleanup] + cleanup = [s for s in scenario.steps if s.what == 'FINALLY'] + normal = [s for s in scenario.steps if s not in cleanup] ok = True for step in normal: - exit = self.run_step(datadir, story, step) + exit = self.run_step(datadir, scenario, step) if exit != 0: ok = False break for step in cleanup: - exit = self.run_step(datadir, story, step) + exit = self.run_step(datadir, scenario, step) if exit != 0: ok = False break @@ -129,7 +131,7 @@ class YarnRunner(cliapp.Application): return ok - def run_step(self, datadir, story, step): + def run_step(self, datadir, scenario, step): logging.info('Running step "%s %s"' % (step.what, step.text)) logging.info('DATADIR is %s' % datadir) self.ts['step'] = step @@ -157,12 +159,12 @@ class YarnRunner(cliapp.Application): if exit != 0: self.ts.error( - 'ERROR: In story "%s"\nstep "%s %s" failed,\n' + 'ERROR: In scenario "%s"\nstep "%s %s" failed,\n' 'with exit code %d:\n' 'Standard output from shell command:\n%s' 'Standard error from shell command:\n%s' % - (story.name, step.what, step.text, exit, self.indent(stdout), - self.indent(stderr))) + (scenario.name, step.what, step.text, exit, + self.indent(stdout), self.indent(stderr))) return exit @@ -15,17 +15,17 @@ .\" .TH YARN 1 .SH NAME -yarn \- story testing of Unix command line tools +yarn \- scenario testing of Unix command line tools .SH SYNOPSIS .SH DESCRIPTION .B yarn -is a story testing tool: -you write a story describing how a user uses your software +is a scenario testing tool: +you write a scenario describing how a user uses your software and what should happen, and express, using very lightweight syntax, -the story in such a way that it can be tested automatically. -The story has a simple, but strict structure: +the scenario in such a way that it can be tested automatically. +The scenario has a simple, but strict structure: .IP .nf GIVEN some setup for the test @@ -33,7 +33,7 @@ WHEN thing that is to be tested happens THEN the post-conditions must be true .fi .PP -As an example, consider a very short test story for verifying that +As an example, consider a very short test scenario for verifying that a backup program works, at least for one simple case. .IP .nf @@ -46,12 +46,12 @@ THEN the data case be restored (Note the addition of AND: you can have multiple GIVEN, WHEN, and THEN statements. The AND keyword makes the text be more readable.) .PP -Stories are meant to be written in somewhat human readable language. +Scenarios are meant to be written in somewhat human readable language. However, they are not free form text. In addition to the GIVEN/WHEN/THEN structure, the text for each of the steps needs a computer-executable implementation. This is done by using IMPLEMENTS. -The backup story from above might be implemented as follows: +The backup scenario from above might be implemented as follows: .IP .nf IMPLEMENTS GIVEN some live data in a directory @@ -103,10 +103,10 @@ that quite a good test suite can be written. .SH EXAMPLE To run .B yarn -on all the stories in your current directory: +on all the scenarios in your current directory: .IP .nf -yarn *.story +yarn *.scenario .fi .PP All the files will be treated together as if they had been one file. diff --git a/yarn.tests/simple.script b/yarn.tests/simple.script index 83714f9..3382af7 100755 --- a/yarn.tests/simple.script +++ b/yarn.tests/simple.script @@ -2,4 +2,4 @@ set -eu -./yarn simple.story +./yarn simple.scenario diff --git a/yarnlib/__init__.py b/yarnlib/__init__.py index 07045fc..88bf46f 100644 --- a/yarnlib/__init__.py +++ b/yarnlib/__init__.py @@ -17,5 +17,5 @@ from mdparser import MarkdownParser -from elements import Story, StoryStep, Implementation +from elements import Scenario, ScenarioStep, Implementation from block_parser import BlockParser, BlockError diff --git a/yarnlib/block_parser.py b/yarnlib/block_parser.py index 1a600ee..84074e2 100644 --- a/yarnlib/block_parser.py +++ b/yarnlib/block_parser.py @@ -26,16 +26,16 @@ class BlockError(cliapp.AppException): pass -# Parse a sequence of textual blocks into Story and Implementation +# Parse a sequence of textual blocks into scenario and Implementation # objects, and their constituent objects. class BlockParser(object): def __init__(self): - self.stories = [] + self.scenarios = [] self.implementations = [] self.line_parsers = { - 'STORY': self.parse_story, + 'SCENARIO': self.parse_scenario, 'GIVEN': self.parse_given, 'WHEN': self.parse_when, 'THEN': self.parse_then, @@ -75,15 +75,15 @@ class BlockParser(object): raise BlockError("Syntax error: unknown step: %s" % line1) - def parse_story(self, line, blocks): - self.stories.append(yarnlib.Story(line)) + def parse_scenario(self, line, blocks): + self.scenarios.append(yarnlib.Scenario(line)) return blocks def parse_simple(self, what, line, blocks): - if not self.stories: - raise BlockError('Syntax errror: %s before STORY' % what) - step = yarnlib.StoryStep(what, line) - self.stories[-1].steps.append(step) + if not self.scenarios: + raise BlockError('Syntax errror: %s before SCENARIO' % what) + step = yarnlib.ScenarioStep(what, line) + self.scenarios[-1].steps.append(step) return blocks def parse_given(self, line, blocks): @@ -99,13 +99,13 @@ class BlockParser(object): return self.parse_simple('FINALLY', line, blocks) def parse_and(self, line, blocks): - if not self.stories: - raise BlockError('Syntax errror: AND before STORY') - story = self.stories[-1] - if not story.steps: + if not self.scenarios: + raise BlockError('Syntax errror: AND before SCENARIO') + scenario = self.scenarios[-1] + if not scenario.steps: raise BlockError( 'Syntax errror: AND before what it would continue') - step = story.steps[-1] + step = scenario.steps[-1] assert step.what in self.line_parsers return self.line_parsers[step.what](line, blocks) diff --git a/yarnlib/block_parser_tests.py b/yarnlib/block_parser_tests.py index 975444f..c988479 100644 --- a/yarnlib/block_parser_tests.py +++ b/yarnlib/block_parser_tests.py @@ -27,52 +27,52 @@ class BlockParserTests(unittest.TestCase): self.parser = yarnlib.BlockParser() def test_is_initially_empty(self): - self.assertEqual(self.parser.stories, []) + self.assertEqual(self.parser.scenarios, []) self.assertEqual(self.parser.implementations, []) def test_parses_simple_elements(self): self.parser.parse_blocks( - ['STORY foo', 'GIVEN bar', + ['SCENARIO foo', 'GIVEN bar', 'WHEN foobar\nTHEN yoyo\nFINALLY yay\nAND yeehaa']) - self.assertEqual(len(self.parser.stories), 1) + self.assertEqual(len(self.parser.scenarios), 1) self.assertEqual(len(self.parser.implementations), 0) - story = self.parser.stories[0] - self.assertEqual(story.name, 'foo') - self.assertEqual(len(story.steps), 5) - self.assertEqual(story.steps[0].what, 'GIVEN') - self.assertEqual(story.steps[0].text, 'bar') - self.assertEqual(story.steps[1].what, 'WHEN') - self.assertEqual(story.steps[1].text, 'foobar') - self.assertEqual(story.steps[2].what, 'THEN') - self.assertEqual(story.steps[2].text, 'yoyo') - self.assertEqual(story.steps[3].what, 'FINALLY') - self.assertEqual(story.steps[3].text, 'yay') - self.assertEqual(story.steps[4].what, 'FINALLY') - self.assertEqual(story.steps[4].text, 'yeehaa') + scenario = self.parser.scenarios[0] + self.assertEqual(scenario.name, 'foo') + self.assertEqual(len(scenario.steps), 5) + self.assertEqual(scenario.steps[0].what, 'GIVEN') + self.assertEqual(scenario.steps[0].text, 'bar') + self.assertEqual(scenario.steps[1].what, 'WHEN') + self.assertEqual(scenario.steps[1].text, 'foobar') + self.assertEqual(scenario.steps[2].what, 'THEN') + self.assertEqual(scenario.steps[2].text, 'yoyo') + self.assertEqual(scenario.steps[3].what, 'FINALLY') + self.assertEqual(scenario.steps[3].text, 'yay') + self.assertEqual(scenario.steps[4].what, 'FINALLY') + self.assertEqual(scenario.steps[4].text, 'yeehaa') def test_normalises_whitespace(self): - self.parser.parse_blocks(['STORY foo bar ']) - self.assertEqual(self.parser.stories[0].name, 'foo bar') + self.parser.parse_blocks(['SCENARIO foo bar ']) + self.assertEqual(self.parser.scenarios[0].name, 'foo bar') def test_handles_empty_line(self): - self.parser.parse_blocks(['STORY foo\n\nGIVEN bar\nTHEN foobar']) - self.assertEqual(len(self.parser.stories), 1) + self.parser.parse_blocks(['SCENARIO foo\n\nGIVEN bar\nTHEN foobar']) + self.assertEqual(len(self.parser.scenarios), 1) def test_raises_error_for_unknown_step(self): self.assertRaises( yarnlib.BlockError, self.parser.parse_blocks, - ['STORY foo\nblah']) + ['SCENARIO foo\nblah']) - def test_raises_error_for_step_outside_story(self): + def test_raises_error_for_step_outside_scenario(self): self.assertRaises( yarnlib.BlockError, self.parser.parse_blocks, ['GIVEN foo']) - def test_raises_error_for_AND_before_story(self): + def test_raises_error_for_AND_before_scenario(self): self.assertRaises( yarnlib.BlockError, self.parser.parse_blocks, @@ -82,7 +82,7 @@ class BlockParserTests(unittest.TestCase): self.assertRaises( yarnlib.BlockError, self.parser.parse_blocks, - ['STORY foo\nAND bar']) + ['SCENARIO foo\nAND bar']) def test_parses_implements_in_a_block_by_itself(self): self.parser.parse_blocks(['IMPLEMENTS GIVEN foo\ntrue']) diff --git a/yarnlib/elements.py b/yarnlib/elements.py index a1dea0d..9eeb1b7 100644 --- a/yarnlib/elements.py +++ b/yarnlib/elements.py @@ -16,9 +16,9 @@ # =*= License: GPL-3+ =*= -# This is a step in a story: GIVEN, WHEN, THEN, etc. +# This is a step in a scenario: GIVEN, WHEN, THEN, etc. -class StoryStep(object): +class ScenarioStep(object): def __init__(self, what, text): self.what = what @@ -26,9 +26,9 @@ class StoryStep(object): self.implementation = None -# This is the story itself. +# This is the scenario itself. -class Story(object): +class Scenario(object): def __init__(self, name): self.name = name diff --git a/yarnlib/mdparser.py b/yarnlib/mdparser.py index 6d16772..6708369 100644 --- a/yarnlib/mdparser.py +++ b/yarnlib/mdparser.py @@ -25,7 +25,7 @@ from markdown.treeprocessors import Treeprocessor # # Classes for Markdown parsing. See python-markdown documentation # for details. We want to find all top level code blocks (indented -# four spaces in the Markdown), which we'll parse for story test +# four spaces in the Markdown), which we'll parse for scenario test # stuff later on. We create a Python markdown extension and use # "tree processor" to analyse the parsed ElementTree at the right # moment for top level <pre> blocks. @@ -50,7 +50,7 @@ class GatherCodeBlocks(Treeprocessor): # gatherer at the right time. It stores the list of top level # code blocks as the blocks attribute. -class ParseStoryTestBlocks(markdown.extensions.Extension): +class ParseScenarioTestBlocks(markdown.extensions.Extension): def extendMarkdown(self, md, md_globals): self.blocks = [] @@ -64,7 +64,7 @@ class MarkdownParser(object): self.blocks = [] def parse_string(self, text): - ext = ParseStoryTestBlocks() + ext = ParseScenarioTestBlocks() f = StringIO.StringIO() markdown.markdown(text, output=f, extensions=[ext]) self.blocks = ext.blocks |