Tests
=====

Running tests
-------------

We currently have two to four coexisting
test mechanisms:

- Traditional shell script tests using ``TEST`` scripts; these are being phased out
- Perl scripts; these has been phased out
- Runtest based scripts, see: http://runtest.readthedocs.org
- CTest

CTest wraps around runtest. Runtest is for
individual tests. CTest ties them all together and produces a test runner.  The
preferred way to run tests is through CTest inside the build directory.

You can run all tests::

  $ ctest

all tests in parallel::

  $ ctest -jN

You can match test names::

  $ ctest -R somename

or labels::

  $ ctest -L essential

to see all options, type::

  $ man ctest

To see all labels, browse ``cmake/Tests(LS)DALTON.cmake``.

You can also run tests individually, for this execute the individual python
``test`` scripts and point it to the correct build directory (try ``./test -h``
to see all options).

.. warning::

  We should here describe what tests we require to pass before pushing
  anything to master.


Writing tests
-------------

Have a look here: http://runtest.readthedocs.org.

And have a look at other test scripts for inspiration. You have there full
Python freedom. The important part is the return code. Zero means success,
non-zero means failure.

To make a new test you have to make a folder, which is the test name, and put a
mol and a dal file in that folder. You also have to include a reference output
in a sub-folder "result/".  Then you can just copy a "test" file from one of the
other test directories and modify it according to your needs (there is no for
this).

You have to add your new test to
``cmake/Tests(LS)DALTON.cmake``. Then do::

  $ cmake ..

in your build directory. Finally you can run your new test with the command::

  $ ctest -R my_new_test

Your test folder will not be cluttered with output files if you are running the
test from the build directory.

You can also adjust the test and execute it directly in the test directory but
then be careful to not commit generated files.


Nightly testing dashboard
-------------------------

We have two testing dashboards, https://testboard.org/cdash/?project=Dalton, and
https://testboard.org/cdash/?project=LSDalton.

This is the place to inspect tests which are known to fail (ideally none).

By default CTest will report to https://testboard.org/cdash/?project=Dalton. You
can change this by setting ``CTEST_PROJECT_NAME`` to "LSDALTON" (or some other dashboard)::

  $ export CTEST_PROJECT_NAME=LSDALTON

By default the build name that appears on the dashboard is set to::

  "${CMAKE_SYSTEM_NAME}-${CMAKE_HOST_SYSTEM_PROCESSOR}-${CMAKE_Fortran_COMPILER_ID}-${BLAS_TYPE}-${CMAKE_BUILD_TYPE}"

If you don't like it you can either change the default, or set the build name
explicitly::

  $ ./setup -D BUILDNAME='a-better-build-name'

Then run CTest with -D Nightly or Experimental::

  $ ctest -D Nightly      [-jN] [-L ...] [-R ...]
  $ ctest -D Experimental [-jN] [-L ...] [-R ...]

If you want to test your current code, take Experimental. If you want to set
up a cron script to run tests every night, take Nightly.

By default CTest will report to https://testboard.org/cdash/?project=Dalton. You
can change this by setting ``CTEST_PROJECT_NAME`` to "LSDALTON" (or some other dashboard)::

  $ export CTEST_PROJECT_NAME=LSDALTON

On a mobile device you may find this useful:
https://testboard.org/cdash/iphone/project.php?project=Dalton


Testing with coverage
---------------------

To compile and test for coverage is to collect statistics on which source lines
are executed by the test suite. This accomplished at the setup stage::

   $ ./setup --type=debug --coverage

and by executing the tests with::

   $ ctest -D Experimental

This will execute the tests and upload the statistics to the CDash, and listed
under *Coverage*. For each source file one obtains the percentage of executable
lines that were executed by the test suite or the selected tests. To be able to
see which lines where executed and which were not executed, one has to be
authorized and logged in to the CDash pages.


Viewing coverage results locally (GNU)
--------------------------------------

This section applies for the GNU compiler suite. It can be useful if one wants to check things locally without submitting each run to the public site.

* The setup is the same as above, the ``--coverage`` compiler option is a synonym for the compiler flags ``-fprofile-arcs -ftest-coverage`` and link flag ``-lgov``.

* During the compile stage, for each source file ``xxx.F`` a file ``xxx.F.gcno`` is generated

* When the tests are run, for each source file ``xxx.F`` a file ``xxx.F.gcda`` is generated

* The ``gcov`` program is used to obtain text files with coverage results. This can be used for individual files: `e.g.`, in the build directory run the command::

    $ gcov CMakeFiles/dalton.dir/DALTON/yyy/xxx.F.gcno
    File '.../dalton/DALTON/yyy/xxx.F'
    Lines executed:86.37% of 653
    Creating 'xxx.F.gcov'

This generates a copy of the source file where each source line is preceded by
exexecution count and line number. In particular, lines that have not been
executed by the tests are labeled ``#####``

* The ``lcov`` program is a graphical frontend and the following steps can be used to generate html locally::

    $ lcov -o xxx.info -c -d CMakeFiles/dalton.dir/DALTON
    $ genhtml -o result xxx.info

Open result/index.html in your browser and you have a local graphical view of
coverage statistics down to the source-line level.