124 lines
4.6 KiB
ReStructuredText
124 lines
4.6 KiB
ReStructuredText
|
|
.. _testing:
|
|
|
|
Testing
|
|
=======
|
|
|
|
Spicy's testing & CI setup includes several pieces that we discuss in
|
|
the following.
|
|
|
|
TLDR: If you make changes, make sure that ``make check`` runs through.
|
|
You need the right ``clang-format`` (see :ref:`clang_format`) and
|
|
``clang-tidy`` (see :ref:`clang_tidy`) versions for that (from Clang
|
|
>=10). If you don't have them (or want to save time), run at least
|
|
``make test``.
|
|
|
|
BTest
|
|
-----
|
|
|
|
Most tests are end-to-end tests that work from Spicy (or HILTI) source
|
|
code and check that everything compiles and produces the expected
|
|
output. We use `BTest <https://github.com/zeek/btest>`_ to drive
|
|
these, very similar to Zeek. ``make test`` from the top-level
|
|
directory will execute these tests. You get the same effect by
|
|
changing into ``tests/`` and running ``btest -j`` there (``-j``
|
|
parallelizes test execution).
|
|
|
|
The most important BTest options are:
|
|
|
|
* ``-d`` prints debugging output for failing tests to the console
|
|
|
|
* ``-f diag.log`` records the same debugging output into ``diag.log``
|
|
|
|
* ``-u`` updates baselines when output changes in expected ways
|
|
(don't forget to commit the updates)
|
|
|
|
There are some alternatives to running just all tests, per the
|
|
following:
|
|
|
|
.. rubric:: Running tests using installation after ``make install``
|
|
|
|
By default, btests are running completely out of the source & build
|
|
directories. If you run ``btest -a installation``, BTest will instead
|
|
switch to pulling everything from their installation locations. If you
|
|
have already deleted the build directory, you also need to have the
|
|
environment variable ``SPICY_INSTALLATION_DIRECTORY`` point to your
|
|
installation prefix, as otherwise BTest has no way of knowing where to
|
|
find Spicy.
|
|
|
|
Unit tests
|
|
----------
|
|
|
|
There's a growing set of units test. These are
|
|
using ``doctest`` and are executed through ``btest`` as well, so just
|
|
running tests per above will have these included.
|
|
|
|
Alternatively, the test binaries in the build directory can be executed to
|
|
exercise the tests, or one can use the ``check`` build target to execute all
|
|
unit tests.
|
|
|
|
Sanitizers
|
|
----------
|
|
|
|
To build tools and libraries with support for Clang's address/leak
|
|
sanitizer, configure with ``--enable-sanitizer``. If Clang's ``asan``
|
|
libraries aren't in a standard runtime library path, you'll also need
|
|
to set ``LD_LIBRARY_PATH`` (Linux) or ``DYLD_LIBRARY_PATH`` (macOS) to
|
|
point there (e.g., ``LD_LIBRARY_PATH=/opt/clang9/lib/clang/9.0.1/lib/linux``).
|
|
|
|
When using Zeek's Spicy support, and Zeek hasn't been compiled
|
|
with sanitizer support, you'll also need to set ``LD_PRELOAD`` (Linux)
|
|
or ``DYLD_INSERT_LIBRARIES`` (macOS) to the shared ``asan`` library to
|
|
use (e.g.,
|
|
``LD_PRELOAD=/data/clang9/lib/clang/9.0.1/lib/linux/libclang_rt.asan-x86_64.so``).
|
|
Because you probably don't want to set that permanently, the test
|
|
suite pays attention to a variable ``ZEEK_LD_PRELOAD``: If you set
|
|
that before running ``btest`` to the path you want in ``LD_PRELOAD``,
|
|
the relevant tests will copy the value for running Zeek.
|
|
|
|
To make the sanitizer symbolize its output you need to set the
|
|
``ASAN_SYMBOLIZER_PATH`` environment variable to point to the
|
|
``llvm-symbolizer`` binary, or make sure ``llvm-symbolizer`` is in
|
|
your ``PATH``.
|
|
|
|
.. note::
|
|
|
|
As we are running one of the CI build with sanitizers, it's ok not
|
|
to run this locally on a regular basis during development.
|
|
|
|
Code Quality
|
|
------------
|
|
|
|
Our CI runs the :ref:`clang_format` and :ref:`clang_tidy` checks, and
|
|
will fail if any of that doesn't pass. To execute these locally, run
|
|
the make target ``format`` and ``tidy``, respectively. Don't forget to
|
|
set ``CLANG_FORMAT`` and ``CLANG_TIDY`` to the right version of the
|
|
binary if they aren't in your ``PATH``.
|
|
|
|
|
|
CI also runs `pre-commit <https://pre-commit.com>`_ with a
|
|
configuration pre-configured in ``.pre-commit-config.yaml``. To run that
|
|
locally on every commit, install pre-commit and then put its git hook
|
|
in place through executing ``pre-commit install``; see the
|
|
`installation instructions <https://pre-commit.com/#install>`_ for
|
|
more details.
|
|
|
|
Docker Builds
|
|
-------------
|
|
|
|
We are shipping a number of Docker files in ``docker/``; see
|
|
:ref:`docker` for more information. As part of our CI, we make sure
|
|
these build OK and pass ``btest -a installation``. If you have Docker
|
|
available, you can run these individually yourself through ``make
|
|
test-<platform>`` in ``docker/``. However, usually it's fine to leave
|
|
this to CI.
|
|
|
|
|
|
How Test Your Branch
|
|
--------------------
|
|
|
|
If you run ``make check`` in the top-level directory you get the
|
|
combination of all the btests, formatting, and linting. That's the
|
|
best check to do to make sure your branch is in good shape, in
|
|
particular before filing a pull request.
|