Skip to content

TestSuite

Jump to bottom
Romain Deltour edited this page Mar 2, 2020 · 6 revisions

1. EPUBCheck Test Suite

1.1. Introduction

The primary purpose of EPUBCheck is to assert whether an EPUB publication (or compoenent thereof) conforms to the EPUB specifications. EPUBCheck is used daily by ebook authors, publishers, and distributors, as a quality assurance tool. It is crucial that it gives the correct results, so that it can be trusted by the digital publishing industry and community.

Software bugs in EPUBCheck, or flaws in its logic, could result in false positives, or false negatives, or runtime errors:

  • A false positive is when EPUBCheck reports that the input does not conform to EPUB specificiations when it is in fact perfectly conforming. This could cause a valid EPUB to be unjustifiably rejected from a reseller’s ingestion pipeline, for instance.
  • A false negative is when EPUBCheck reports that the input conforms to the EPUB specifications when it is in fact non-conforming. This could cause interoperability issues where an invalid EPUB could not be read correctly on a Reading System expecting valid content.
  • A runtime error is when EPUBCheck terminates before even being able to report if the input is or is not conforming to the specifications. This is also known as a software crash. The issue here is that users are simply denied an answer to their question about the conformance of their publication.

These three kinds of errors all have obvious negative impacts.

In order to verify that EPUBCheck runs without errors and returns the expected results, the source code includes a series of automated tests. These tests can be grouped into the following categories:

  • Functional Tests: the tests which verify that EPUBCheck correctly evaluates the conformance to EPUB specifications
  • Unit Tests: the tests which verify the functionality of a specific part of the code, such as a utility class or an API.
  • Integration Tests: the tests that verify that the entire system, executed as a whole, functions as expected.

The tests are executed automatically at build time. They are run for instance by our Continuous Integretion build server whenever new code is pushed to the master branch. Tests can also be executed on demand during the development, individually or not. For more information on how the tests can be executed, refer to the section How to Run Tests.

1.2. Functional Tests

1.2.1. Overview

EPUBCheck’s functional tests form the biggest and most crucial part of the test suite. Their role is to verify that EPUBCheck correctly evaluates the conformance criteria defined in the EPUB specifications.

These tests follow a “black box” testing approach: the principle is to run EPUBCheck on an input file (or set of files), and then to inspect the resulting output, without having to know exactly how the logic is implemented internally.

In order to make the functional tests as readable as possible, and also understandable and maintainable by people who aren’t necessarily familiar with the Java language, this part of the test suite relies on the Gherkin language and the Cucumber tool. The Gherkin language allows test cases to be specified in a natural language syntax in plain text files (.feature files). Cucumber is the tool that parses the Gherkin test case specifications and execute the associated Java code to evaluate if the tests pass or fail.

The following sections describe:

1.2.2. A Cucumber Test

Test cases, also known as scenarios in Cucumber lingo, are defined in feature files (files with a .feature extension) as a set of readable instructions, called steps. Here is an example of the content of a simple feature file:

Feature: Validate EPUB 3
  
  EPUBCheck should check conformance to the latest EPUB 3 specifications,
  available at: https://www.w3.org/publishing/epub3/

  Background: 
    Given EPUB test files located at '/epub3/files/epub/'
    And EPUBCheck with default settings

  Scenario: valid EPUB
    When checking EPUB 'minimal'
    Then no errors or warnings are reported

  Scenario: invalid EPUB
    When checking EPUB 'schema-error'
    Then error RSC-005 is reported
    And no other errors or warnings are reported

This feature file contains two tests, one for a valid EPUB, the other for an invalid EPUB (due to a schema error). It has a specific structure, and some lines start with a keyword (Feature, Scenario, Given, When, Then, And, etc), but is generally fairly readable. Let’s take it apart and describe what it means, piece by piece.

The first line of a feature file consists of the keyword Feature, followed by a colon and a title:

Feature: Validate EPUB 3

The title (here, "Validate EPUB 3") should briefly identify the topic or feature being tested. It will be present in the test report produced by Cucumber, and help to quickly know what the tests are about.

The next lines contain a longer description of the topic or feature being tested:

  EPUBCheck should check conformance to the latest EPUB 3 specifications,
  available at: https://www.w3.org/publishing/epub3/

This longer description is optional. It can be of any lengh, and ends when the next line starts with a keyword (typically Background or Scenario). In the EPUBCheck test suite, the feature description is used to explain what part of the EPUB specifications is tested.

After the feature description comes a Background section, here containing two steps:

  Background: 
    Given EPUB test files located at '/epub3/files/epub/'
    And EPUBCheck with default settings

A step is any line starting with any of the keywords Given, When, Then, And, or But. Each step is associated with some Java code (see the section Underlying Java Code); Cucumber will execute this code in sequence, one step at a time, when running the tests. The steps included in a Background section will be run before each scenario. This section is therefore useful to define the initial steps shared by all the tests in the feature file.

In the EPUBCheck test suite, Background section is used to define the location where to find the test files, and to specify the EPUBCheck configuration shared by all the tests in the feature. These steps and their syntax are described at length in the Steps Reference.

The rest of the feature file basically consists of Scenario sections. Each scenario describes one test case, as a sequence of steps. It also has a title, consisting of the text string after the colon character following the Scenario keyword. The scenario titles should describe what the test is about; they are notably displayed in the test report, and can be used to identify the purpose of the test case.

For instance, the first Scenario in our example feature file is named "valid EPUB" and checks that no errors or warnings are reported when EPUBCheck is run on a minimalistic valid EPUB:

  Scenario: valid EPUB
    When checking EPUB 'minimal'
    Then no errors or warnings are reported

Because the feature file contains a Background section (as seen earlier), the sequence of steps that are actually executed when evaluating this test case are the following:

    Given EPUB test files located at '/epub3/files/epub/'
    And EPUBCheck with default settings
    When checking EPUB 'minimal'
    Then no errors or warnings are reported

This sequence of steps follows a typical pattern that is found in all the scenarios:

  1. description of the initial context (steps Given and And above)
  2. description of an action (step When)
  3. description of the expected result (step Then)

Note that a scenario can have any number of steps. The keywords And and But can be used to improve the reading flow. In fact, the step keywords are not actually differentiated in the underlying implementation, they just improve the readability. A sequence "given a context, when this happens and this also happens, then this is expected but this isn't expected" is more readable than the equivalent sequence "given a context, when this happens when this also happens, then this is expected then this isn't expected", or even the also equivalent but illegible sequence "then a context, then this happens then this also happens, then this is expected then this isn't expected".

A scenario can also start with free-form text. Free-form text made of any line preceding the list of steps and not starting with a keyword. This can be used to provide a longer detailed description of the feature, when the step sequence is not descriptive enough by itself.

Now let's look at the second scenario in our sample feature file. By reading the sequence of steps, we understand that it tests that EPUBCheck reports a single RSC-005 error for the specific invalid EPUB named schema-error.

  Scenario: invalid EPUB
    When checking EPUB 'schema-error'
    Then error RSC-005 is reported
    And no other errors or warnings are reported

At this point, it is important to note that the text of the steps (for instance "checking EPUB 'schema-error'" or "error RSC-005 is reported") is what is used by Cucumber to identify what piece of Java code to execute in order to run the test. Each step should correspond to a matching step definition, i.e. a Java method that implements the step’s logic.

However, test authors typically do not have to write any line of Java code themselves, as the EPUBCheck test suite already includes a collection of step definitions which intend to cover the variety of EPUBCheck test cases. Writing a new test case for EPUBCheck consists of describing a scenario in a plain text feature file, using any of the readily available steps listed in the Steps reference below.

1.2.3. Steps reference

Note: the steps in this reference are described using the Cucumber expression used in the matching step definition. In this notation, text in parenthesis is optional text, text in curly brackets identify typed parameters, and text around a slash character denotes alternative spellings.

Note: If you’re writing a test and find a use case for a new step that is not in this collection of steps, please file an issue or submit a pull request!

1.2.3.1. Setting the location of test files

Step: (EPUB )test files located at {string}

This step is used to set a base directory where test files can be found. The {string} parameter is the path of the base directory, relative to the src/test/resources directory. Test files are the EPUB content used in the test cases; they can be expanded EPUB file sets, packaged EPUBs, single content documents, package documents, etc.

Examples:

Given EPUB test files located at '/epub3/files/epub/'

Given test files located at '/epub3/files/content-document-xhtml/'

1.2.3.2. Configuring EPUBCheck

Step: EPUBCheck with default settings

This step is essentially a no-op: it indicates that EPUBCheck is instantiated with its default configuration.

Step: EPUBCheck configured to check a(n) {checkerMode}

This step is used to configure the type of file expected as input to EPUBCheck. The {checkerMode} parameter can be any of the following values (case insensitive):

  • full publication
  • Media Overlays Document
  • Navigation Document
  • Package Document
  • SVG Content Document
  • XHTML Content Document

Examples:

Given EPUBCheck configured to check a full publication

Given EPUBCheck configured to check an XHTML Content Document

Given EPUBCheck configured to check a navigation document

Step: EPUBCheck configured with the ('){profile}(') profile

This step is used to configure the validation profile used by EPUBCheck. A profile is used to run enable the checks related to a specific EPUB satellite specification (like EDUPUB, EPUB Indexes, EPUB Dictionaries, etc.). The {profile} parameter can be any of the following values (case insensitive):

  • default (the default value), used to validate against the main EPUB specifications
  • dict, used to validate against the EPUB Dictionaries and Glossaries specification
  • edupub, used to validate against the EPUB for Education specification (aka EDUPUB)
  • idx, used to validate against the EPUB Indexes specification
  • preview, used to validate against the EPUB Previews specification

Examples:

Given EPUBCheck configured with the default profile

Given EPUBCheck configured with the EDUPUB profile

Given EPUBCheck configured with the 'dict' profile

Step: the reporting level is set to {checkerMode}

This step is used to configure the severity of the reported messages. The {severity} parameter can be any of the following values (case insensitive):

  • fatal
  • error
  • warning
  • info (default)
  • usage
  • suppressed

Examples:

And the reporting level is set to Usage

Given the reporting level set to Error

1.2.3.3. Running EPUBCheck

Step: checking EPUB/document/file {string}

This step executes EPUBCheck with the given document as input. The {string} parameter is the path of the test document, relative to the base directory of test files. Given the flat organization of test files, it is typically just the name of the test file (or directory, for expanded EPUBs).

Examples:

When checking EPUB 'minimal'

When checking EPUB 'minimal.epub'

When checking document 'valid-content.xhtml'

When checking file 'invalid-file.foo'

Note that it is possibe to use a Scenario Outline (also known as a scenario template) to test several input files using the same step sequence. For instance:

Scenario Outline: valid EPUB <EPUB>
  When checking EPUB '<EPUB>'
  Then no errors or warnings are reported

  Examples:
    | EPUB               |
    | minimal.epub       |
    | fallbacks.epub     |
    | mediaoverlays.epub |

1.2.3.4. Expecting EPUBCheck results

Step: no( other) warning(s)/error(s) or error(s)/warning(s) are/is reported

This step asserts that the stack of warnings and error messages reported by EPUBCheck is empty. It may be used either:

  • as the first assertion step, to verify that no errors or warnings at all were reported by EPUBCheck, i.e. that the input is fully conforming to the specifications. In that case, the word "other" should be ommitted ("no warnings or errors are reported").
  • following assertion steps which checked that some isses were reported, to verify that no other issues were found. In that case, the word "other" should be present ("no other errors or warnings are reported").

It is important to note that the assertion steps consume the messages. For example, when a test description to use a sequence of assertions like:

Then error X is reported
And warning Y is reported
And no other warnings or errors are reported

the first step asserts that error X exists in the list of reported messages and consumes it; the next step asserts that warning Y exists and consumes it; finally the last step verifies that the list of messages is left with no other warnings or errors.

Examples:

Then no errors or warnings are reported

Then no error is reported

And no other errors or warnings are reported

And no other warnings are reported

Step: (the ){severity} {messageId} is reported

This step asserts that a specific message (typically a warning or error) was reported by EPUBCheck. The {severity} parameter must be a case-insensitive match to either ERROR, WARNING, INFO, USAGE, or FATAL. The messageId parameter must be a cases-sensitive match to an EPUBCheck message ID, as defined in the MessageId.java class (for example, RSC-005, or PKG-010, or OPF-086b). Message IDs can use a hyphen ('-') or an underscore ('_') interechangeably.

Examples:

Then error RSC-005 is reported

Then WARNING RSC-009 is reported

Step: (the ){severity} {messageId} is reported {int} time(s)

This step asserts that a specific message (typically a warning or error) was reported several times by EPUBCheck. The {severity} parameter must be a case-insensitive match to either ERROR, WARNING, INFO, USAGE, or FATAL. The messageId parameter must be a cases-sensitive match to an EPUBCheck message ID, as defined in the MessageId.java class (for example, RSC-005, or PKG-010, or OPF-086b). Message IDs can use a hyphen ('-') or an underscore ('_') interechangeably. The {int} paraameter must be an integer, equals to the number of times the message is expected to be reported.

It is important to note that this step asserts that the message is reported at least the given number of times. In order to ensure that it is not reported more than that, the step must be combine with a "no other warnings or errors are reported" step.

Examples:

Then error RSC-005 is reported 3 times

And warning RSC-009 is reported 1 time

Step: (the )following {severity}( ID)(s) are/is reported

This step asserts that a list of messages of a same severity (typically error or warning) are reported by EPUBCheck. The {severity} parameter must be a case-insensitive match to either ERROR, WARNING, INFO, USAGE, or FATAL. A list of expected message IDs must be given as a data table, in the fist column. Each element in the list must be a cases-sensitive match to an EPUBCheck message ID, as defined in the MessageId.java class (for example, RSC-005, or PKG-010, or OPF-086b). Message IDs can use a hyphen ('-') or an underscore ('_') interechangeably. A second column of the data table can be used to specify a text string that the reported message is expected to contain.

Examples:

Then the following warning IDs are reported
      | RSC-017 |
      | RSC-018 |

Then the following errors are reported
      | RSC-005 | attribute "foo" not allowed here |
      | RSC-005 | element "bar" not allowed here   |

Step: (the )message is {string}

This step asserts that the text of the last consumed message is equal to the given content. The {string} parameter is the expected text of the message.

This step should only be used after a step that verified that a given message ID was reported.

Examples:

Then error RSC-005 is reported
And the message is 'attribute "foo" not allowed here'

Step: (the )message contains {string}

This step asserts that the text of the last consumed message contains the given content. The {string} parameter is the text that the message is expected to contain.

This step should only be used after a step that verified that a given message ID was reported.

Examples:

Then error RSC-005 is reported
And the message contains 'Duplicate ID'

1.2.4. Test Resources Organization

1.2.4.1. Directory layout

The source code in EPUBCheck is following the ususal Maven standard directory layout:

  • Java test sources are located in the src/test/java directory
  • test resources are located in the src/test/resources directory.

The test resources related to the functional tests consist of the Gherkin feature files and all the test files used as input for EPUBCheck. The directory layout for the test resources is the following:

  1. test resources for EPUB 2 and EPUB 3 are located in distinct top-level directories, respectively named epub2 and epub3, hereafter called version directories.
  2. each version directory contains:
  • a list of Gherkin feature files (.feature extension) describing the tests for the related version of the EPUB specifications. How the tests are organized in feature files is further defined in the Feature files organization section below.
  • a files directory containing subdirectories for each type of test files used as input for EPUBCheck (e.g. whole EPUB or single Content Document). All the test files are directory contained within these subdirectories. The list of subdirectories and the naming convention used for test files are further defined in the Test files organization section below.

To give an example, the directory layout for functional tests looks like the following file and directory structure:

src/test/resources
├── epub2
│   ├── container-epub.feature
│   ├── content-document-xhtml.feature
│   ├── navigation-ncx-document.feature
│   ├── package-document.feature
|   ├── …
|   └── files
│       ├── epub
│       ├── content-document-xhtml
│       ├── ncx-document
│       ├── package-document
│       └── …
└── epub3
    ├── core-container-epub.feature
    ├── core-content-document-xhtml.feature
    ├── core-navigation-document.feature
    ├── core-navigation-epub.feature
    ├── …
    └── files
        ├── epub
        ├── content-document-xhtml
        ├── navigation-document
        ├── package-document
        └── …

1.2.4.2. Feature files organization

The EPUBCheck test suite includes hundreds of tests for each version of EPUB. It would be impractical to include all of them in a single massive feature file. Instead, tests are distributed in various feature files, organized by tested topic.

As the EPUB format is actually defined in a family of specficiations, the first way to categorize tests is by related specifications. For example, some feature files describe the tests for the rules defined in the EPUB Packages specification (for example, core-package-epub.feature), and other feature files describe the tests for the rules defined in the EPUB Content Document specification (for example, core-content-document-xhtml.feature). Satellite specifications (some of which are not widely adopted, but still supported by EPUBCheck) are also covered in distinct feature files (for example, indexes-epub.feature or edupub-epub.feature). In addition, some sections within the specifications are cohesive enough to justify having their own feature file (for example, core-navigation-epub.feature, or core-fixedlayout-epub.feature). Finally, a same topic can be split in several feature files depending on the nature of the tested input document (for example, core-navigation-epub.feature contains the navigation-related tests based on full EPUB publications, while core-navigation-document.feature contains the tests based on validating a single Navigation Document).

By convention, the name of feature files follow this pattern:

{specification}-{topic?}-{input-type}-{specialization?}

where:

  • {specification} identifies the specification (or family of specifications) containing the rules being tested, which can be any of:
    • core: the core family of EPUB specifications, including EPUB, EPUB Packages, EPUB Content Documents, and EPUB Open Container Format.
    • dictionaries: the EPUB Dictionaries specification
    • edupub: the EDUPUB specification
    • indexes: the EPUB Indexes specification
    • mediaoverlays: the EPUB Media Overlays specification
    • not specified for EPUB 2.0.1, which doesn’t have satellite specifications
  • {topic} may optionally be used to identify the sub-topic being tested, which can be any of:
    • container: tests related to the EPUB Open Container Format (OCF)
    • content: tests related to EPUB Content Documents
    • fixedlayout: tests related to Fixed Layout EPUBs
    • navigation: tests related to the EPUB Navigation Document (or NCX in EPUB 2.0.1)
    • package: tests related to EPUB Packages
    • resources: tests related to Publication Resources
  • {input-type} is the kind of document being used as input for the tests, which can be any of:
    • epub: tests based on the validation of full EPUB publications, either packages (.epub files) or unpackages (directories)
    • content-document: tests based on the validation of single EPUB Content Documents (e.g. .xhtml files)
    • mediaoverlays-document: tests based on the validation of single EPUB Media Overlays Documents (.smil files)
    • navigation-document: tests based on the validation of single EPUB Navigation Documents
    • ncx-document: tests based on the validation of single NCX document (EPUB 2.0.1)
    • package-document: tests based on the validation of single EPUB Package Documents (.opf files)
  • {specialization} may optionally be used to further differentiate the type of the input document being tested, for instance:
    • xhtml: for XHTML Content Documents
    • svg: for SVG Content Documents
    • dtbook: for DTBook Content Documents (EPUB 2.0.1)

when two consecutive segments of the file name are identical, they are not repeated. For example, the feature file for tests related to the core Navigation Document rules and using single Navigation Documents as input is named core-navigation-document.feature instead of core-navigation-navigation-document.feature.

Note: this naming convention is merely used to establish a consistent organization and help with manual test lookup. It is not relied upon for the execution of the tests; all the feature files will be automatically detected by Cucumber when running the test suite.

The feature files describing the tests related to the EPUB 3 specifications are:

  • core-container-epub.feature
  • core-content-document-svg.feature
  • core-content-document-xhtml.feature
  • core-content-epub.feature
  • core-fixedlayout-epub.feature
  • core-navigation-document.feature
  • core-navigation-epub.feature
  • core-package-document.feature
  • core-package-epub.feature
  • core-resources-epub.feature
  • dictionaries-content-document.feature
  • dictionaries-epub.feature
  • dictionaries-package-document.feature
  • edupub-content-document.feature
  • edupub-epub.feature
  • edupub-package-document.feature
  • indexes-content-document.feature
  • indexes-epub.feature
  • mediaoverlays-document.feature
  • mediaoverlays-epub.feature

The feature files describing the tests related to the EPUB 2.0.1 specifications are:

  • container-epub.feature
  • content-document-dtbook.feature
  • content-document-svg.feature
  • content-document-xhtml.feature
  • content-epub.feature
  • navigation-ncx-document.feature
  • navigation-epub.feature
  • package-document.feature
  • package-epub.feature
  • resources-epub.feature

Note: this categorization of tests in feature files may not be definitive. The organization can easily be adapted and may evolve depending on test writers and developers feedback. Feel free to file an issue or submit a pull request to add a new feature file, or suggest an improvement to this organization.

1.2.4.3. Test input files organization

All the functional tests for EPUBCheck are based on running EPUBCheck on a sample input file or directory, and inspecting that the returned messages are conforming to the expectations. As a result, the test resources directory contains many test files, that are used by the tests described in the feature files. This section describes the directory structure and naming convention used to to organize this big collection of test material in a consistent and maintainable manner.

By convention, names of test input files follow this pattern:

{specification}-{description}-{conformance}

where:

  • {specification} identifies the specification (or family of specifications) containing the rules being tested, which can be any of:
    • core: the core family of EPUB specifications, including EPUB, EPUB Packages, EPUB Content Documents, and EPUB Open Container Format.
    • dictionaries: the EPUB Dictionaries specification
    • edupub: the EDUPUB specification
    • indexes: the EPUB Indexes specification
    • mediaoverlays: the EPUB Media Overlays specification
    • not specified for EPUB 2.0.1, which doesn’t have satellite specifications
  • {description} identifies the feature or issue covered by this test file. The description is a sequence of words, separated by hyphens ("-"). It should be as explicit as possible, but also reasonably terse to avoid overly long file names. The topic or area of the spec (e.g. schema, fixedlayout, renditions, etc.) being tested should be listed first, in order for the files to be grouped together with alphabetical sorting.
  • {conformance} must be either valid, or a message severity like error or warning.

For example:

  • core-content-valid
  • core-schema-relaxng-error
  • core-schema-duplicate-id-error
  • core-aria-role-valid
  • core-content-switch-warning
  • core-fallback-missing-error

1.2.5. Underlying Java code

_Note: this section is targeted at EPUBCheck developers; it is not required reading for people who only want to contribute new tests

The Java code used by Cucumber to run the functional tests is in the org.w3c.epubcheck.test package, located in usual src/test/java directory.

1.2.5.1. Test Runners

The test suite comes with two Cucumber JUnit runners, which can be used to run the Cucmber tests like any other JUnit tests. These two runners are defined in the following Java classes:

  • RunCucumberTest.java: the default runner, which will execute all the Gherkin scenarios found in the tests classpath.
  • DebugCucumberTest.java: a utility runnerm which will only execute the scenarios marked with the @debug tag. See the section “Tips to run and debug a single test” below for more details.

1.2.5.2. Step definitions

The step definitions are the Java methods which are invoked for each step in Gherkin scenarios. These methods are mapped to the steps using an expression given as a method annotation. In the EPUBCheck test suites, the steps are implemented in two classes:

  • ExecutionSteps.java: contains all the steps related to instantiating and running EPUBCheck
  • AssertionSteps.java: contains all the steps related to evaluating expectations on EPUBCheck results.

These classes are injected with a TestReport object, which can be used as a shared state by the different step definitions. The PicoContainer library is used as a minimalistic dependency injection solution.

1.2.5.3. Type configuration

Cucumber uses a type registry to know how to convert Gherkin step parameters and data table types to Java types. This registry is configured in the class TypeRegistryConfiguration, which notably supports conversion from:

  • severity levels (error, warning, etc) to the Severity enumeration defined in EPUBCheck.
  • message IDs (RSC-005, PKG-009, etc) to the MessageId` enumeration defined in EPUBCheck.
  • test input types (full publication, navigation document, etc) to an enumeration defined in the step implementation classes.

1.2.5.4. Domain classes

The test classes rely on two primary domain classes:

  • TestReport: this class implements EPUBCheck’s Report interface, and is used by EPUBCheck to report all the messages produed during the validation of a file. In addition to the Report methods, it contains lookup methods used in assertion steps to verify that the reported messages match the expectations. Each EPUBCheck instance, created for each test case scenario, is configured with a new instance of a TestReport. The TestReport is injected in the step implementation classes; it is used as a shared state during the step evaluation.
  • MessageInfo: a simple representation of an EPUBCheck message, containing information like the message ID, its severity, the message text, etc.

1.3. Unit tests

The list of test classes containing unit tests for the various APIs or helper classes is:

  • com.adobe.epubcheck.api.ApiConstructorsTest.java
  • com.adobe.epubcheck.ocf.OCFFilenameCheckerTest.java
  • com.adobe.epubcheck.opf.MetadataSetTest.java
  • com.adobe.epubcheck.util.DateParserTest.java
  • com.adobe.epubcheck.util.OPFPeekerTest.java
  • com.adobe.epubcheck.util.PathUtilTest.java
  • com.adobe.epubcheck.util.SourceSetTest.java
  • com.adobe.epubcheck.vocab.PrefixParsingTest.java
  • com.adobe.epubcheck.vocab.PropertyTest.java
  • com.adobe.epubcheck.vocab.VocabTest.java
  • org.idpf.epubcheck.util.css.CssInputStreamTest.java
  • org.idpf.epubcheck.util.css.CssParserTest.java
  • org.idpf.epubcheck.util.css.CssScannerTest.java

1.4. Integration tests

1.5. How to Run Tests

1.5.1. Running tests from the command line

The EPUBCheck tests can be executed during the automated build process, based on the Maven tool.

The following command can be used to run a clean build and execute the whole EPUBCheck test suite:

mvn clean verify

This will run all the functional, unit, and integration tests.

The following command can be used to run a clean build and execute the all the tests but the integration tests:

mvn clean test

It is also possible to run all the tests from a single test class using the test property:

mvn -Dtest=VocabTest test

Or

mvn -Dtest=TestCircle#mytest test

For each test class that is executed, the test results will be logged to the standard output, like in the following example:

Running org.idpf.epubcheck.util.css.CssParserTest
Tests run: 123, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec - in org.idpf.epubcheck.util.css.CssParserTest

A summary of all the test results is also logged after all the tests have been run:

Results :

Tests run: 1231, Failures: 0, Errors: 0, Skipped: 0

Individual test reports can also be found as files in the target/surefire-reports directory.

For more information on running tests with the Maven build tool, refer to the documentation of the Maven Surefire plugin.

1.5.2. Running tests from the development environment

Some development environments support running tests from their UI, either natively or with extension plugins.

In Eclipse, the default JUnit plugin can be used to run all the tests, including the Cucumber functional tests (invoked via the JUnit Cucumber test runner). See the official documentation on running JUnit tests with Eclipse.

A Cucumber plugin for Eclipse is also available, which includes formatting for Gherkin feature files, as well as the ability to run Cucumber directly from feature files.

1.5.3. Tips to run and debug a single test

Remember that the EPUBCheck test suite includes two JUnit runners for Cucumber tests: RunCucumberTest and DebugCucumberTest. When running RunCucumberTest, all the scenarios described in all the feature files (found in the tests classpath) will be evaluated. In other words, this will run the entire suite of functional tests. However, when writing tests, you may want to run only the tests from a single feature, or even a single test in the feature. This is doable with the DebugCucumberTest runner.

The DebugCucumberTest runner will only execute the features or scenarios annotated with a @debug tag. The tag must be set on a single line above the line starting with the Feature or Scenario keywords.

For instance, the following scenario will be enabled when using the debug runner:

  @debug
  Scenario: invalid EPUB
    When checking EPUB 'schema-error'
    Then error RSC-005 is reported
    And no other errors or warnings are reported

The DebugCucumberTest runner will also write EPUBCheck’s log to the standard output. This is useful to inspect the exact messages reported by the tool when running a scenario.

When you've finished writing or debugging a test or group of tests, please remove all the @debug tags, to not commit them to the code base.

1.5.4. Troubleshooting

1.5.4.1. UndefinedThrowable exception

When writing new functional tests in Gherkin, you may find that an UndefinedThrowable exception is raised by Cucumber when running the tests. For instance:

io.cucumber.junit.UndefinedThrowable: The step "error RSC-005 is reported" is undefined

This exception is raised when no Java step definition was found to match a specific step in your Gherkin scenario description. In the example above, the step which lacked a definition is error RSC-005 is reported.

Please check that you’re using steps as documented in the Steps reference. If your test scenario cannot be sufficiently described using the steps in that default collection, please please file an issue or submit a pull request!

Clone this wiki locally