TESTING

Running the tests
-----------------

- All tests reside in the tests/ directory tree.

- Running the tests is as simple as:
    $ make check

- Alternatively, the test scripts can be run directly:
    $ cd tests
    $ ./run_tests.py

- To run the only the tests associated with language X ("cpp", "java", etc.):
    $ cd tests
    $ ./run_tests.py X  

Adding new tests
----------------

- A test is defined by adding a line in one of the tests/*.test
  files: you must pick the .test file for the proper programming
  language, i.e. pick the X.test file which defines tests which use
  inputs in language X.

- A test has a (unique) number (a.k.a. test name) -- it does not have
  to be a sequential number, but it might be handy to treat it like
  that.

- The X.test entry (line) has this format:

    <test.name> <config.file> <input.file>

  where the fields are separated by an arbitrary non-zero amount of
  whitespace.

- As each test references a config file and an input file (the latter
  written in programming language X), these are assumed to exist in
  the paths tests/config/<config.file> and tests/<input.file>
  respectively. It is _strongly_ advised to put input files in their
  own subdirectory, so a 'C' source formatting test entry might look
  like this (in 'c.test'):

    99902 ger-full-cmt-reflow.cfg c/cmt-not-a-boxed-cmt.c

  where 'ger-full-cmt-reflow.cfg' is stored at
  tests/config/ger-full-cmt-reflow.cfg and 'c/cmt-not-a-boxed-cmt.c'
  will be fetched from tests/c/cmt-not-a-boxed-cmt.c when you run
  the tests.
  

Features
--------

- You can re-use config files and input files for multiple tests. So
  one might observe test sets like these:

  # clark's style - blank lines before and after flow control,
  indented comments:
  
    20100 clark.cfg c/i2c-core.c
    20101 clark.cfg c/comment-indent.c
    20102 clark.cfg c/indent.cpp
    20103 clark.cfg c/output.cpp

  and

    21051 return-1.cfg c/nl_return_expr.c
    21052 return-2.cfg c/nl_return_expr.c
    21053 return-3.cfg c/nl_return_expr.c
    21054 return-4.cfg c/nl_return_expr.c

- Reference output files (the reference against which the uncrustify
  test run output is compared) are to be stored in the path
  tests/output/<X>/<NNN-TTTTT.TTT> where <X> is the 'language
  directory' part of the input file, e.g. 'c' for input file
  'c/nl_return_expr.c', <NNN> is the test name (number), e.g. '21051',
  and <TTTTT.TTT> is the file name part of the input test filespec,
  e.g. 'nl_return_expr.c' for input file spec 'c/nl_return_expr.c'.

  This means that for test

    21051 return-1.cfg c/nl_return_expr.c

  the accompanying reference output is:

    tests/output/c/21051-nl_return_expr.c

- When starting out with a new test, you don't need to have a
  'reference output' yet: the test will simply be reported as a
  'fail'ed test until you do.

- Tip: the easiest way to produce 'reference output' is to copy the
  test output (from tests/results/...etc... ) to
  tests/output/...etc... once you've ascertained that those tests
  produce the desired (correct) output. The helper script
  tests/fixtest.sh will copy the results file to the output folder:

    $ fixtest.sh 30014 00110  # copy tests 30014 and 00110

- Tip: the helper script tests/difftest.sh runs 'diff' on the result
  and output versions to point out what changed.

- Tip: There are also a few options to run_tests.py that can
  help. Most interesting is '-d' will run 'diff' if the test fails.