.. _testing: Testing ======= Spicy's testing & CI setup includes several pieces that we discuss in the following. TLDR: If you make changes, make sure that ``make check`` runs through. You need the right ``clang-format`` (see :ref:`clang_format`) and ``clang-tidy`` (see :ref:`clang_tidy`) versions for that (from Clang >=10). If you don't have them (or want to save time), run at least ``make test``. BTest ----- Most tests are end-to-end tests that work from Spicy (or HILTI) source code and check that everything compiles and produces the expected output. We use `BTest `_ to drive these, very similar to Zeek. ``make test`` from the top-level directory will execute these tests. You get the same effect by changing into ``tests/`` and running ``btest -j`` there (``-j`` parallelizes test execution). The most important BTest options are: * ``-d`` prints debugging output for failing tests to the console * ``-f diag.log`` records the same debugging output into ``diag.log`` * ``-u`` updates baselines when output changes in expected ways (don't forget to commit the updates) There are some alternatives to running just all tests, per the following: .. rubric:: Running tests using installation after ``make install`` By default, btests are running completely out of the source & build directories. If you run ``btest -a installation``, BTest will instead switch to pulling everything from their installation locations. If you have already deleted the build directory, you also need to have the environment variable ``SPICY_INSTALLATION_DIRECTORY`` point to your installation prefix, as otherwise BTest has no way of knowing where to find Spicy. Unit tests ---------- There's a growing set of units test. These are using ``doctest`` and are executed through ``btest`` as well, so just running tests per above will have these included. Alternatively, the test binaries in the build directory can be executed to exercise the tests, or one can use the ``check`` build target to execute all unit tests. Sanitizers ---------- To build tools and libraries with support for Clang's address/leak sanitizer, configure with ``--enable-sanitizer``. If Clang's ``asan`` libraries aren't in a standard runtime library path, you'll also need to set ``LD_LIBRARY_PATH`` (Linux) or ``DYLD_LIBRARY_PATH`` (macOS) to point there (e.g., ``LD_LIBRARY_PATH=/opt/clang9/lib/clang/9.0.1/lib/linux``). When using Zeek's Spicy support, and Zeek hasn't been compiled with sanitizer support, you'll also need to set ``LD_PRELOAD`` (Linux) or ``DYLD_INSERT_LIBRARIES`` (macOS) to the shared ``asan`` library to use (e.g., ``LD_PRELOAD=/data/clang9/lib/clang/9.0.1/lib/linux/libclang_rt.asan-x86_64.so``). Because you probably don't want to set that permanently, the test suite pays attention to a variable ``ZEEK_LD_PRELOAD``: If you set that before running ``btest`` to the path you want in ``LD_PRELOAD``, the relevant tests will copy the value for running Zeek. To make the sanitizer symbolize its output you need to set the ``ASAN_SYMBOLIZER_PATH`` environment variable to point to the ``llvm-symbolizer`` binary, or make sure ``llvm-symbolizer`` is in your ``PATH``. .. note:: As we are running one of the CI build with sanitizers, it's ok not to run this locally on a regular basis during development. Code Quality ------------ Our CI runs the :ref:`clang_format` and :ref:`clang_tidy` checks, and will fail if any of that doesn't pass. To execute these locally, run the make target ``format`` and ``tidy``, respectively. Don't forget to set ``CLANG_FORMAT`` and ``CLANG_TIDY`` to the right version of the binary if they aren't in your ``PATH``. CI also runs `pre-commit `_ with a configuration pre-configured in ``.pre-commit-config.yaml``. To run that locally on every commit, install pre-commit and then put its git hook in place through executing ``pre-commit install``; see the `installation instructions `_ for more details. Docker Builds ------------- We are shipping a number of Docker files in ``docker/``; see :ref:`docker` for more information. As part of our CI, we make sure these build OK and pass ``btest -a installation``. If you have Docker available, you can run these individually yourself through ``make test-`` in ``docker/``. However, usually it's fine to leave this to CI. How Test Your Branch -------------------- If you run ``make check`` in the top-level directory you get the combination of all the btests, formatting, and linting. That's the best check to do to make sure your branch is in good shape, in particular before filing a pull request.