pk/zeek
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
zeek/auxil/spicy/CHANGES
Patrick Kelley 8fd444092b initial
2025-05-07 15:35:15 -04:00

9881 lines
366 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

1.13.0-dev.214 | 2025-04-25 12:33:30 +0200
* Prevent escape of non-HILTI exception in lower-level driver functions. (Benjamin Bannier, Corelight)
While we already did some handling of non-HILTI exceptions in Spicy's
high-level drivers[^spicy-driver] (even though this would allow escape
of exceptions not deriving from `std::exception`), we did not do the
same for the lower level processing functionality in the actual driver.
This is the interface used by e.g., Zeek, and already an
`std::exception` thrown from a parser would lead to the exception
propagating into Zeek and ultimately causing a `std::terminate` there,
see e.g., [this oss-fuzz
report](https://oss-fuzz.com/testcase-detail/6088469254569984). In that
particular case a call to `std::vector::reserve` exceeding the available
memory caused raising of a `std::length_error`.
This patch extends the exception handling in the driver so all
exceptions are handled by now transforming any non-HILTI exceptions into
HILTI exceptions like expected by callers.
[^spicy-driver]: https://github.com/zeek/spicy/blob/32251850bd8808d18c5dd9306dc0d56c7ebd5d9d/spicy/toolchain/bin/spicy-driver.cc#L331-L334
1.13.0-dev.212 | 2025-04-24 08:31:03 -0400
* Limit `reserve` calls on vectors to prevent OOM. (Evan Typanski, Corelight)
1.13.0-dev.210 | 2025-04-23 15:32:56 +0200
* CI: No more Ubuntu 20.04 (Arne Welzel, Corelight)
EOL in April 2025, remove it now.
1.13.0-dev.208 | 2025-04-23 12:20:45 +0200
* Bump `justrx`. (Robin Sommer, Corelight)
1.13.0-dev.206 | 2025-04-22 17:40:06 +0200
* Reserve vectors with repeat counts. (Evan Typanski, Corelight)
This improves performance of large vectors with a known size by about
10-20% while not hurting performance of small vectors.
* Fix minor comment issues in `vector.h`. (Evan Typanski, Corelight)
1.13.0-dev.203 | 2025-04-22 17:38:22 +0200
* Prevent unneeded copy. (Benjamin Bannier, Corelight)
1.13.0-dev.201 | 2025-04-17 14:03:12 -0400
* Avoid attribute temporaries for constants. (Evan Typanski, Corelight)
* GH-2016: Fix `&size` expressions evaluating multiple times. (Evan Typanski, Corelight)
Fixes #2016
If the attribute is evaluated multiple times, some cases with side
effects may unknowingly get evaluated multiple times. This effect would
be different depending what happens - if there's a parse error, it could
even trigger an integer overflow when calculating offsets.
This solution uses a temporary and passes it around the parser builder
when it's needed. The downside is this is only applicable to `&size`,
but, luckily, that seems to be all that it currently applies to.
There are some alternative solutions, but they don't seem right. One
would be adding the temporary to the attribute, but that relies on only
retrieving that temporary if you're in the correct scope. Making the
attribute itself scope-aware seems wrong, as does removing the stored
expression when it would be out of scope.
It could also technically apply to the state, but that ended up being
more cumbersome than seemed necessary.
1.13.0-dev.198 | 2025-04-17 16:52:19 +0200
* Bump justrx to pull in CCL speed-up. (Robin Sommer, Corelight)
1.13.0-dev.196 | 2025-04-16 13:42:21 +0200
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Only disable clang-tidy if CMake version supports it. (Benjamin Bannier, Corelight)
1.13.0-dev.193 | 2025-04-16 10:38:34 +0200
* GH-1955: Split attributes between HILTI and Spicy. (Robin Sommer, Corelight)
A previous change had moved knowledge of Spicy attributes into the
HILTI layer. This splits that out again to separate concerns. The main
challenge is that we were using C++ enums, which could not be extended
by Spicy to add its own attributes. We now replace those enums values
with a custom helper class so that we can define constants at
multiple places.
Other than that new helper class, most of the changes are pretty
mechanical, with the main objective being that Spicy only refers to
its own attributes for anything that can appear in Spicy source code,
and uses HILTI attributes only when interacting with the HILTI layer.
1.13.0-dev.191 | 2025-04-15 10:06:29 +0200
* Drop unreachable lexer rules. (Benjamin Bannier, Corelight)
1.13.0-dev.189 | 2025-04-15 08:30:10 +0200
* Bump 3rdparty/filesystem from `076592c` to `9fda7b0` (dependabot[bot])
* Bump 3rdparty/utfcpp from `b26a5f7` to `65701fe` (dependabot[bot])
1.13.0-dev.186 | 2025-04-10 16:06:16 +0200
* Remove spurious include. (Robin Sommer, Corelight)
1.13.0-dev.184 | 2025-04-10 10:14:16 +0200
* GH-2013: Disallow illegal characters in hook IDs and a couple other places. (Robin Sommer, Corelight)
The lexer wasn't handling all cases.
* Revert "Bring back the internal `__visit` method for generated structs." (Robin Sommer, Corelight)
This reverts commit 479b2d4bfb88c3010aba21f63db4c2c04a65216a. We now
*really* don't need this anymore since Zeek has been adapted
accordingly.
* GH-2007: Fix using `&try` with `&max-size`, and potentially other cases. (Robin Sommer, Corelight)
In the generated code, a temporary was declared inside the `try { ...
}` block that we create for `&try`, but then used afterwards outside
of the block. A similar issue probably could have happened in
other situations as well.
The fix is to move more of the logic that was so far following the
`try`-block into the block. This comes with potential slight semantic
changes where in case of a parse error, before we would have executed
some post-field logic that now skip. The main example is hooks:
before, if a field with a `&try` had a hook, this would still run in
case of a parse error; now it's not running anymore—--which indeed
seems the better semantics.
* Let optimizer and dependency tracker ignore unresolved declarations. (Robin Sommer, Corelight)
This normally doesn't happen because validation catches unresolved
declarations early. However, for debugging it can be helpful to
support this when running with `--skip-validation` (e.g., to get to
the output of `--output-hilti` for debugging broken code generation).
1.13.0-dev.177 | 2025-04-09 12:33:40 +0200
* Bump 3rdparty/utf8proc from `5568eff` to `3de4596` (dependabot[bot])
1.13.0-dev.175 | 2025-04-08 14:22:30 +0200
* Make sure `scope_exit` never throws on teardown. (Benjamin Bannier, Corelight)
* Restructure fold expression for Coverity. (Benjamin Bannier, Corelight)
* Prevent unneeded copies. (Benjamin Bannier, Corelight)
1.13.0-dev.171 | 2025-04-07 11:31:12 +0200
* Add `bytes::ends_with`. (Benjamin Bannier, Corelight)
* Add `string::ends_with`. (Benjamin Bannier, Corelight)
* Add `hilti::rt::endsWith`. (Benjamin Bannier, Corelight)
1.13.0-dev.167 | 2025-04-02 14:13:29 +0200
* Fix a path in the CMake config for the recent benchmark. (Robin Sommer, Corelight)
1.13.0-dev.165 | 2025-04-02 12:37:53 +0200
* Fine-tune recent CMake 4 tweak. (Robin Sommer, Corelight)
1.13.0-dev.163 | 2025-04-02 10:28:07 +0200
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Always build benchmarks. (Benjamin Bannier, Corelight)
* Never build 3rdparty code with `-Werror`. (Benjamin Bannier, Corelight)
* Bump 3rdparty/benchmark to 1.9.2. (Benjamin Bannier, Corelight)
* Fix clang-tidy warnings on macos. (Benjamin Bannier, Corelight)
1.13.0-dev.157 | 2025-04-02 10:27:53 +0200
* Fix minor typos in comments and docs [skip CI]. (Evan Typanski, Corelight)
1.13.0-dev.155 | 2025-04-02 09:59:29 +0200
* Change runtime representation of tuples. (Robin Sommer, Corelight)
At runtime, we so far stored HILTI tuples simply as `std::tuple` of
the corresponding element types. However, that ignored that in HILTI
tuple elements can remain unset, which that runtime representation
could not express. This still worked so far because (a) unset elements
are rare (the main use case is the `.?` operator used inside a tuple
constructor); and (b) in the cases we needed it, we had C++-side
template code figure things out a compile time. However, the old
representation meant that a host application using introspection
through HILTI's runtime type information, could not receive/see unset
tuple elements at all.
This commit changes the runtime tuple representation to wrap all
elements into `std::optional`. So a HILTI tuple `tuple<T1, T2, T3>`
now gets turned into `std::tuple<std::optional<T1>, std::optional<T2>,
std::optional<T3>>`. While that comes with some additional overhead,
that can now fully capture all possible tuple values. The runtime type
information is adjusted accordingly as well. While that is a breaking
change, it presumably affects just Zeek as the only external user of
that API currently. A corresponding Zeek change is forthcoming.
The rest of the commit is mainly mechanical: it adapts code generator
and runtime to work with the new tuple representation.
* Provide `type_info::Value` constructor not taking a parent. (Robin Sommer, Corelight)
This leaves the internal member unset, saving an unnecessarily created
`shared_ptr`.
* Add mechanism to leave temporary codegen variables uninitialized. (Robin Sommer, Corelight)
Local variables now support a "special" init expression for
performance optimization: If the init expression is an instance of
`expression::Void`, the generated C++ code will not default-initialize
the variable. That means it's not safe to read from it before it has
been written to at least once. This avoids the overhead of the
creating the default value when it's not needed, but it disables
HILTI's safety property of forcing all runtime values to have
well-defined content at all times.
This is a work-around until we have flow-based optimization which
should be able to figure out automatically when it's safe to skip
computing the default.
* Add builder method to create an `add` expression. (Robin Sommer, Corelight)
* Remove AST node for deferred expressions. (Robin Sommer, Corelight)
This was no longer used anywhere now.
* Change FFI calling convention for `any`. (Robin Sommer, Corelight)
When translating HILTI functions to C++, we now convert any HILTI-side
arguments of type `any` into *two* C++-side arguments: the value (as
before) plus a `TypeInfo*` pointer describing the value's type.
Before, we did this manually in a few cases where we needed the type
information by adding an additional `type` parameter to the HILTI
function. Now this happens implicitly and is therefore automatically
available to all functions using `any` parameters. The advantage is
that we don't need to modify an call sites to provide type information
explicitly, in particular when they are inside generated code. The
downside is that we end up receiving that new parameter in a few cases
where we don't actually need it (though this could change in the
future if we ever started to move away from templates inside the C++
runtime).
* Add a few missing type information pieces. (Robin Sommer, Corelight)
- Add runtime type information for `Null` type.
- Generate `vector` type information for type `List` (because that's how we store lists).
- Fix type information for `result<void>`.
- dd `to_string()` support for `Nothing`.
* Make pre-defined type information objects constant. (Robin Sommer, Corelight)
* Report `ast-final` debugging output even if AST validation fails. (Robin Sommer, Corelight)
1.13.0-dev.142 | 2025-04-01 17:41:10 +0200
* Work-around CMake 4 no longer setting `CMAKE_OSX_SYSROOT` by default. (Robin Sommer, Corelight)
* Let `configure` work with CMake 4. (Robin Sommer, Corelight)
* GH-1580: Catch when functions aren't called. (Robin Sommer, Corelight)
There are only very few cases where usage of a function is permitted
without calling it.
* GH-1971: Extend unit `switch` based on look-ahead to support blocks of items. (Robin Sommer, Corelight)
1.13.0-dev.135 | 2025-03-28 09:44:11 +0100
* GH-1967: Fix inactive stack size check during module initialization. (Robin Sommer, Corelight)
* GH-1974: Add documentation on how to interpret stack traces involving fibers. (Robin Sommer, Corelight)
* GH-1964: Document use of `$$` inside `&{while,until,until-including}`. (Robin Sommer, Corelight)
* GH-1963: Document `assert-exception`. (Robin Sommer, Corelight)
* GH-1155: Document iteration over maps/set/vectors. (Robin Sommer, Corelight)
1.13.0-dev.128 | 2025-03-26 11:02:19 -0400
* Add simple parser benchmarking. (Evan Typanski, Corelight)
1.13.0-dev.126 | 2025-03-18 09:04:37 +0100
* Bring back the internal `__visit` method for generated structs. (Robin Sommer)
This reverts part of cb7336ff8d02a32fc: as it turns out, the Zeek
integration still needs `__visit`. I'd like to solve this differently
on the Zeek side, but have't quite gotten that to work yet. So
reverting this for now so that we don't block.
1.13.0-dev.124 | 2025-03-12 09:02:38 -0400
* Fix defaulted params with different return types. (Evan Typanski, Corelight)
The overload check for equivalent non-defaulted parameters was within a
check for non-equivalent return types, so it only applied if return
types differed. This fixes that to apply when return types are the same.
* Fix overload conflicts beyond with the first decl. (Evan Typanski, Corelight)
Previously it would only compare with the first declaration, but this
makes it compare with all previous declarations.
* Add reasons for rejecting invalid overloads. (Evan Typanski, Corelight)
* GH-1982: Reject equivalent functions. (Evan Typanski, Corelight)
Fixes #1982
* Reject function prototypes without `&cxxname`. (Evan Typanski, Corelight)
1.13.0-dev.118 | 2025-03-06 09:05:38 -0500
* Fix `for` loop snippet to run as-is. (Evan Typanski, Corelight)
* GH-1538: Implement compound statements (`{...}`). (Evan Typanski, Corelight)
Closes #1538
The functionality was there, just not available in the parser.
1.13.0-dev.115 | 2025-03-06 12:00:38 +0100
* GH-1981: Reject function overloads which only differ in default'd parameters. (Benjamin Bannier, Corelight)
Resolution of calls to functions who's signatures differ only in
defaulted parameters behaved non-intuitively in that when given with
less than the maximum possible number of parameters we resolved to the
overload with most parameters.
With this patch we reject such overloads where the only difference is in
defaulted parameters.
Closes #1981.
1.13.0-dev.113 | 2025-03-03 17:39:36 +0100
* GH-1983: Audit links in README. (Benjamin Bannier, Corelight)
Closes #1983.
1.13.0-dev.111 | 2025-03-03 17:15:46 +0100
* GH-1977: Disallow function redef with different return. (Evan Typanski, Corelight)
Fixes #1977
C++ doesn't allow different return types with equivalent parameters, so
Spicy should grab that and error before it gets to the C++ error.
1.13.0-dev.109 | 2025-03-03 17:13:37 +0100
* GH-1976: Fix unit `&max-size` not returning to proper loc. (Evan Typanski, Corelight)
Fixes #1976
&max-size was treated the same as &size - so it was going to the end of
the limited view rather than to the proper location based on what was
parsed.
1.13.0-dev.107 | 2025-02-21 11:05:36 +0100
* Fix possibly-incorrect custom host compile command (Evan Typanski)
The order of `ldflags` compared to the `-o` output matters - they should
be after. As-is, the documentation fails to compile, but with this
change, the command succeeds for me.
1.13.0-dev.105 | 2025-02-20 12:28:40 +0100
* Remove documentation of ``&nosub``. (Robin Sommer, Corelight)
Turns out we have never supported that in Spicy (only in HILTI). The
reasons seems to be a parsing challenge: cannot easily add attributes
to regular expression constants with our Spicy Bison parser.
* GH-1965: Add support for case-insensitive matching to regular expressions. (Robin Sommer, Corelight)
By adding an `i` flag to a regular expression pattern, it will no w be
matched case-insensitively (e.g. `/foobar/i`).
* GH-100: Switch regular expressions to new justrx API. (Robin Sommer, Corelight)
justrx no longer parses group match IDs out of the expression string,
but instead now offers an API to set them directly. This switches over
to using that new API.
As a result, setting a match ID inline inside a regexp doesn't work
anymore. An example of old syntax was: `/foobar${#100}/`. Instead, we
now allow IDs to be provided after the final slash. The new syntax is
`/foobar/$(100)`. While this is technically a breaking change, I doubt
anybody has been using this, and I don't think it's worth putting any
backwards-compatibility in.
Admittedly, that new ID syntax is bit arbitrary, and mostly chosen so
that it can be parsed easily. But it's not something users will
commonly use, so that seems fine. The Bison parser for that new syntax
already prepares for more pattern flags as well that we will want to
support.
* Bump justrx to pull in new API group matching. (Robin Sommer, Corelight)
* Refactor regexp patterns. (Robin Sommer, Corelight)
So far we were storing the regexp patterns as strings directly inside
the `RegExp` class. This factors out that storage into a new `Pattern`
helper class. Right now, that helper class likewise only stores a
string, just as before. But will extend that soon.
1.13.0-dev.99 | 2025-02-18 11:43:18 +0100
* Add a `to_string()` callback to our custom runtime type information. (Robin Sommer, Corelight)
Now a host application can render a value into a string that it has
retrieved through the runtime introspection API. The
`type_info::Value` class gains a `to_string()` method as the
user-facing entry point to that.
* Provide generated, C++-side structs and unions with explicit `__to_string` methods. (Robin Sommer, Corelight)
This replaces the `__visit` methods we had so far, which we only used
for implementing the free-standing ``to_string()`` overloads for structs and units.
This is primarily cleanup, simplifying our generated code by avoiding
the hard-to-read `__visit` indirection, and maybe also speeding up
compilation times a bit because we no longer need any templates.
We also rename the `__str__` method (used for implementing the `on
%print` hook) to `__hook_to_string` for consistency.
Note that external C++ code should continue to call the free-standing
`to_string()` function to render a struct/unit, not the new method,
because the method doesn't take the `on %print` hook into account.
* Move runtime function `lookupParser()` into global space. (Robin Sommer, Corelight)
This used to be a method of the Spicy `Driver` class, but isn't really
tied to it, and useful on its own as well. Moving it to global scope,
but keeping the existing method for backwards compatibility.
* Make `type` a type. (Robin Sommer, Corelight)
This is primarily for making it easier to pass type information to
host applications. It contains the following:
- In HILTI & Spicy, allow to declare variables to have type
`type`. These variables then store type information objects.
Initialization can come from either `typeinfo(TYPE)` (which
already existed in HILTI, but not Spicy), or coercion from an
existing type ID (e.g., `global T: type = MyStruct);` (which is
new in both HILTI and Spicy). Note that coercion from a basic
type does not work because of limitations of our Bison parser
(it can't parse, e.g., `t = bool`).
The effect of `type` was already achievable by declaring
variables to be of type
`__library_type("hilti::rt::TypeInfo*")`, but that was quite
hackish/internal.
- Add an attribute to `type::Library` that records whether the
underlying C++ type is always `const`, and needs to be declared
as such. We need this for `hilti::rt::TypeInfo`, which is always
constant. So far we had just added that `const` to C++-side type
itself, but that (1) was just a hack, and (2) could cause
trouble if the name was used in places where that `const`
confused the C++ compiler.
To be able to declare a `const` library type from inside
HILTI/Spicy, we add a new keyword `__library_type_const`.
- Include the C++ name of the library type into it's runtime type
information.
- Add `print` support for type information, printing the type's
`display` name.
* Do not include CXX flags into linker flags. (Robin Sommer, Corelight)
Unnecessary, and may confuse the linker.
* Map C++ types corresponding to public structs into public namespace in generated C++ prototypes. (Robin Sommer, Corelight)
* Fix generated C++ prototype header. (Robin Sommer, Corelight)
1.13.0-dev.91 | 2025-02-18 11:42:52 +0100
* Remove `TryCoercionWithinSameType` from all shortcut coercion styles. (Robin Sommer, Corelight)
Seems it wasn't needed anywhere there actually, mostly because it
combination with `TryCoercion` it wasn't adding anything.
* Compose a coercion bit. (Benjamin Bannier, Corelight)
* Reuse previously declared coercion bits for readability. (Benjamin Bannier, Corelight)
* Remove unused conversion bit. (Benjamin Bannier, Corelight)
* GH-1968: Fix coercion of function call arguments. (Robin Sommer, Corelight)
Before, we could unnecessarily coerce integer types when a direct
match was available. The fix takes out the `TryCoercionWithinSameType`
style for non-coercing function call arguments, which solves it. I
don't recall why that was in there in the first place, but it doesn't
seem to break anything to skip it ...
1.13.0-dev.85 | 2025-02-18 10:04:22 +0100
* Fix possessive "it's" in contexts docs. (Evan Typanski, Corelight)
* GH-1966: Reject anonymous units in variables and fields. (Evan Typanski, Corelight)
* Avoid creating external types unnecessarily during AST construction. (Robin Sommer, Corelight)
We could end up creating external types for freshly created AST nodes
just because we wanted them to have a certain constness/sideness. This
could then lead to the original types not being validated because they
weren't inserted into the AST anywhere.
This also extends the AST dump output for qualified types to indicate
whether it's an internal or external type.
1.13.0-dev.81 | 2025-02-07 09:21:54 +0100
* Drop freebsd-13 from default CI jobs. (Benjamin Bannier, Corelight)
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Bump freebsd-14 in CI. (Benjamin Bannier, Corelight)
* Bump freebsd-13 in CI. (Benjamin Bannier, Corelight)
* Bump opensuse-15 in CI. (Benjamin Bannier, Corelight)
1.13.0-dev.75 | 2025-01-23 10:29:10 +0100
* Regenerate usage documentation. (Benjamin Bannier, Corelight)
* Clean up minor warts in usage strings of some executables. (Benjamin Bannier, Corelight)
* Add `spicy-dump` option to enable profiling. (Benjamin Bannier, Corelight)
1.13.0-dev.71 | 2025-01-14 17:26:02 +0100
* Prevent unneeded copy in setting up unit `parseX` functions. (Benjamin Bannier, Corelight)
* Used fixed initialization for order for global static. (Benjamin Bannier, Corelight)
The `HILTI_PRE_INIT` macros work on the global
`_registered_preinit_functions` container. Since uses of the macro and
the global container are likely in different translation units their
initialization order can be undetermined (even though we have some
manual checks whether the `unique_ptr<vector>>` is null).
This patch cleans up initialization of the global so it should always be
fully initialized when accessed.
1.13.0-dev.68 | 2025-01-13 22:54:50 +0100
* GH-1958: Prevent copies in various places. (Benjamin Bannier, Corelight)
This adds a change incorrectly dropped from #1958.
1.13.0-dev.66 | 2025-01-13 22:09:19 +0100
* Prevent copies in various places. (Benjamin Bannier, Corelight)
1.13.0-dev.64 | 2025-01-10 09:55:22 +0100
* Address a FIXME. (Benjamin Bannier, Corelight)
* Prevent exception in generic `noexcept` function. (Benjamin Bannier, Corelight)
* Avoid theoretical exception in widely used helper function. (Benjamin Bannier, Corelight)
* Prevent destructor from throwing. (Benjamin Bannier, Corelight)
* Avoid throwing never handled exception in low-level helper function. (Benjamin Bannier, Corelight)
* Prevent exception escape. (Benjamin Bannier, Corelight)
* Drop dereference of end iterator. (Benjamin Bannier, Corelight)
* Prevent potential nullptr dereference. (Benjamin Bannier, Corelight)
* Prevent copies in various places. (Benjamin Bannier, Corelight)
This patch removes copies in different places by e.g., leveraging const
lifetime expansion or move semantics. We also clean up getters to return
new values less often.
* Prevent division by zero which is UB. (Benjamin Bannier, Corelight)
* Clearly mark up unreachable code. (Benjamin Bannier, Corelight)
* Add missing `break` in `switch` case. (Benjamin Bannier, Corelight)
* Drop redundant string copy. (Benjamin Bannier, Corelight)
* Remove copyright end year in source files [skip CI]. (Benjamin Bannier, Corelight)
* Remove copyright years [skip CI]. (Evan Typanski, Corelight)
1.13.0-dev.47 | 2025-01-08 17:48:23 +0100
* Fix update Fedora links in installation docs. (Benjamin Bannier, Corelight)
1.13.0-dev.45 | 2025-01-08 17:15:06 +0100
* GH-1936: Migrate `Attribute::Kind` to enum stringification framework. (Benjamin Bannier, Corelight)
This patch performs cleans up the recently added `Attribute::Kind` enum
so it is properly integrated.
- introduce a standalone enum `attribute::Kind` outside of the
`Attribute` class
- use enum helpers for conversion from and to string
Closes #1936.
1.13.0-dev.43 | 2025-01-08 09:39:45 +0100
* Drop unused includes. (Benjamin Bannier, Corelight)
* Move temporaries instead of creating copies. (Benjamin Bannier, Corelight)
* Avoid copies in implementation of trivial getters. (Benjamin Bannier, Corelight)
* Avoid copy when constructing node properties. (Benjamin Bannier, Corelight)
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Always initialize enum member to reasonable default. (Benjamin Bannier, Corelight)
* Always initialize pointer members. (Benjamin Bannier, Corelight)
* Fix use-after-move. (Benjamin Bannier, Corelight)
1.13.0-dev.34 | 2025-01-08 09:39:11 +0100
* Bump `3rdparty/filesystem` to pick up fixes for clang-tidy-19. (Benjamin Bannier, Corelight)
* Drop unused clang-format integration in `./ci/run-ci`. (Benjamin Bannier, Corelight)
This was already unused since we moved clang-format to a pre-commit
hook.
* Move `./ci/run-ci`'s clang-tidy check to CMake. (Benjamin Bannier, Corelight)
Instead of implementing this ourself instea use CMake with
`CMAKE_CXX_CLANG_TIDY`/`CMAKE_C_CLANG_TIDY`.
* Update CMake version in CI for clang-tidy. (Benjamin Bannier, Corelight)
* Bump LLVM toolchain used in CI. (Benjamin Bannier, Corelight)
* Fix up APIs in AST API which were not actually moving. (Benjamin Bannier, Corelight)
Many functions in the AST API were moving arguments only for them to
ultimately be copied; `clang-tidy` diagnosed most of these.
This patch cleans up the flagged useless passes by value and `std::move`
without effect.
* Fix up container helpers to ensure moving. (Benjamin Bannier, Corelight)
It is not immediately clear to me when the previous implementation could
move their values over. `clang-tidy` ended up flagging many of these as
not moving and forcing copies.
This patch rewrites the loops in these helpers to enforce iteration with
move.
* Disable clang-tidy on generated lexer/parser files. (Benjamin Bannier, Corelight)
* Reformat code not recognized as C++ by pre-commit hook. (Benjamin Bannier, Corelight)
* Remove some unneeded includes. (Benjamin Bannier, Corelight)
* Parenthesize expression when arithmetic mixes difference precedences. (Benjamin Bannier, Corelight)
`clang-tidy` suggests adding extra parentheses when mixing e.g.,
addition and multiplication. Apply these suggestions (automatically).
* Consistently pass `std::string_view` by value. (Benjamin Bannier, Corelight)
`std::string_view` is a type which is designed to be (very) cheap to
copy.
* Remove unneeded copies. (Benjamin Bannier, Corelight)
* Fix APIs taking `std::string_view` assuming them to be null-terminated. (Benjamin Bannier, Corelight)
In general `std::string_view` values do not need to be null-terminated,
e.g., if they reference subranges in a bigger null-terminated string. We
were previously passing `std::string_view` in a few places where we
assumed them to be null-terminated, e.g., for passing them on to C APIs
expecting null-terminated `const char*`. These worked well enough for us
for now, but these APIs were generally unsafe.
This patch switches to instead passing either `const char*` signalling
proper, null-terminated strings, or by passing in a `const std::string&`
where it should already be present in the caller.
* Consistently mark functions and values not exposed `static`. (Benjamin Bannier, Corelight)
* Locally suppress `clang-tidy` diagnostic. (Benjamin Bannier, Corelight)
This suppresses a `clang-tidy` lint suggesting to mark our default
`main` function as not exported. This function is shipped as part of
libhilti and intended to be available as a weak symbol.
* Prefer `std::max` over ad hoc implementations. (Benjamin Bannier, Corelight)
* Fix APIs where we assumed a string_view was null-terminated. (Benjamin Bannier, Corelight)
This either switches to using `const char*` to signify guaranteed null
termination, or locally suppress clang-tidy warnings about assuming
null-terminated string_views.
* Bring configuration for `clang-tidy` up-to-date. (Benjamin Bannier, Corelight)
This patch brings our clang-tidy setup up to date. Much of this was
built with `./ci/run-ci` `in mind with extra suppressions hardcoded
e.g., in `.clang-tidy.ignore`. With this patch we instead configure
`.clang-tidy` files so they can suppress diagnostics from certain
directories. We also set up 3rdparty code to work with CMake's
`clang-tidy` integration[1].
We also globally suppress some diagnostics which we have no intention of
addressing.
[^1]: https://cmake.org/cmake/help/latest/prop_tgt/LANG_CLANG_TIDY.html#prop_tgt:%3CLANG%3E_CLANG_TIDY
1.13.0-dev.14 | 2025-01-06 16:29:53 +0100
* Do not require owning strings for `startsWith`. (Benjamin Bannier, Corelight)
By passing in a `const std::string&` we would have incurred construction
costs of an owning string when all we needed was a non-owning view.
We also clean up the implementation of `startsWith` to use standard
library functionality.
* Move implementation of `Bytes::[lower|upper]` to cc file. (Benjamin Bannier, Corelight)
* GH-1788: Implement decoding and encoding UTF16 `Bytes`. (Benjamin Bannier, Corelight)
This adds two new charsets `UTF16LE` and `UTF16BE` for little and big
endian UTF16 respectively.
We also clean up use of the Unicode replacement character to make it
work consistently between UTF16 and UTF8.
Closes #1788.
* Add utfcpp submodule. (Benjamin Bannier, Corelight)
* Remove unneeded explicit conversion to HILTI enum in test. (Benjamin Bannier, Corelight)
* Add optional `errors` parameter to `strings::encode`. (Benjamin Bannier, Corelight)
This parameter defaults to `DecodeErrorStrategy::REPLACE` like the
previous implicit parameter used in the implementation.
* Move `Bytes` encoding into `string`. (Benjamin Bannier, Corelight)
This allows a cleaner separation between `Bytes` as "bags of bytes" and
`string` as "valid UTF8". Having such a clean separation will make
adding support for more encodings less duplicative.
* Use unified `DecodeErrorStrategy` for both `bytes` and `string`. (Benjamin Bannier, Corelight)
This was already exposed as a single type in Spicy and HILTI anyway. We
also move `Charset` into the `unicode` namespace.
* Update for spicy-1.12.0 release. (Benjamin Bannier, Corelight)
* Keep test `spicy.rt.time` working for the next 200 years. (Benjamin Bannier, Corelight)
This test still compares the current time against hardcoded times to
check ordering. Extend the upper value so this test passes for the next
~200 years.
1.13.0-dev.2 | 2024-12-19 17:08:09 +0100
* GH-1949: Fix codegen for string literals containing null bytes. (Benjamin Bannier, Corelight)
Our strings are UTF8 which can contain literal null bytes. We were
previously generating incorrect C++ for such literals. This was due to
us constructing C++ value via constructors taking `const char*` which is
expected to be null-terminated, i.e., everything after the terminator
was ignored even though it was emitted.
With this patch we switch to C++ literals for generating literal and
runtime strings.
Closes #1949.
1.13.0-dev.0 | 2024-12-13 09:26:36 +0100
* GH-1901: Add consistent validation for attributes. (Evan Typanski, Corelight)
This updates attribute validation to see if attributes are in the
right places. The goal here is to have one place (ie a big set) to
answer the question "Can I use X attribute on Y node?"
There are a lot more moving parts with attribute validation, but those
generally have to do with behavior. A lot of that requires extra context
in the validator, which is exactly what the validator is meant to do.
Much of that is pretty ad-hoc, so it could get cleaned up as well.
1.12.0-dev.240 | 2024-12-12 13:09:05 +0100
* Make sure autogen-docs pre-commit hook can always run in CI. (Benjamin Bannier, Corelight)
1.12.0-dev.238 | 2024-12-10 14:53:29 +0100
* Update tutorial to use spicy-driver's batch mode. (Robin Sommer, Corelight)
* Allow defining parser alias names when running spicy-driver. (Robin Sommer, Corelight)
``--parser-alias <alias>=<name>`` now makes the parser of name
`<name>` accessible under an additional name `<alias>`. The option can
be used multiple times.
This is particularly useful when using input batches: by adding an
alias of the form `PORT/PROTOCOL` (e.g., `--parser-alias
80/tcp=MyHTTP::Message`), this allows to define a parser to use on for
connections on that port (because the batch processing interprets any
ports in the trace as strings and tries to lookup a parser of the
corresponding name). Adding `%orig` or `%resp` to the alias name
limits it to the corresponding direction (e.g., ``--parser-alias
80/tcp%orig=MyHTTP::Request``).
See the `spicy-driver` documentation for more details on how to use
this with batches.
* Let `spicy-{dump,driver}` print out parser alias names with ``-ll``. (Robin Sommer, Corelight)
``-list-parsers`` prints the direct names of all known parsers.
However, internally, we sometimes register additional alias names that
can also be used to lookup a parser (e.g., if there's a `%mime-type`
or `%port` property). If one specifies ``-list-parsers`` twice now,
those aliases are printed as well.
* GH-1829: Catch integer shifts exceeding the width of the operand. (Robin Sommer, Corelight)
For constants, we now reject such shifts at compile time. At runtime,
we catch them by having the forked version of `SafeInt` throw an
overflow exception.
* Change `SafeInt` submodule to a fork including a fix. (Robin Sommer, Corelight)
We now use a fork of `SafeInt` that includes
https://github.com/dcleblanc/SafeInt/pull/64 (as well as general bump
to upstream `master`).
* Add `max-lines` option to Sphinx' `spicy-output` directive. (Robin Sommer, Corelight)
This limits the lines of output shown to the given number, adding a
`...` marker if truncated.
* GH-1928: Deprecate `&anchor` with regular expression constructors. (Robin Sommer, Corelight)
* Update auto-generated architecture diagram. (Robin Sommer, Corelight)
Looks like some dependency upgrade changed the content slightly.
1.12.0-dev.226 | 2024-12-09 17:28:27 +0100
* Fix `HILTI_CXX_FLAGS` for when multiple flags are passed. (Benjamin Bannier, Corelight)
* Add helper function to perform shell-like string splitting. (Benjamin Bannier, Corelight)
1.12.0-dev.223 | 2024-12-09 11:54:03 +0100
* Fix ruff 'ISC' lints. (Benjamin Bannier, Corelight)
* Fix ruff 'I' lints. (Benjamin Bannier, Corelight)
* Fix ruff 'C4' lints. (Benjamin Bannier, Corelight)
* Migrate Python linting to ruff. (Benjamin Bannier, Corelight)
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Upgrade Python to 3.9 with pyupgrade. (Benjamin Bannier, Corelight)
1.12.0-dev.216 | 2024-12-06 10:34:31 +0100
* Introduce `deprecated` helper function in Spicy validator. (Benjamin Bannier, Corelight)
* Add test for using both `[]` and `&count`. (Benjamin Bannier, Corelight)
* GH-1938: Deprecate `&count` attribute. (Benjamin Bannier, Corelight)
The preferred way to indicate how many elements should be parsed for a
vector has been vector syntax over `&count` for a long time, e.g.,
```
type X = unit {
: uint8[] &count=42; # AVOID.
: uint8[42]; # PREFER.
};
```
With this patch `&count` is now deprecated and we emit a warning. We
will remove support for `&count` in a future release.
Closes #1938.
1.12.0-dev.212 | 2024-12-05 10:31:19 +0100
* Fix doc code snippet that won't compile. (Evan Typanski, Corelight)
1.12.0-dev.210 | 2024-12-04 12:08:31 +0100
* Fix issue with type inference for `result` constructor. (Robin Sommer, Corelight)
`global x = result("foo");` would end up having type `string` instead
of `result<string>`.
* GH-1856: Disallow dereferencing a `result<void>` value. (Robin Sommer, Corelight)
Closes #1856.
* GH-1856: Teach `&requires` to accept condition-tests expressions. (Robin Sommer, Corelight)
This now allows creating custom error messages when a `&require`
condition fails. Example:
type Foo = unit {
x: uint8 &requires=($$ == 1 : error"Deep trouble!'");
# or, shorter:
y: uint8 &requires=($$ == 1 : "Deep trouble!'");
};
Internally, `&requires` now always wants a condition-test expression.
That remains transparent to the user, however, because the resolver
knows how to coerce a pure boolean `&requires` expression into a
condition test, crafting an error message that happens to look like
what we used to produce in the past.
Closes #1856.
* Teach `assert` to accept an `result<void>` value. (Robin Sommer, Corelight)
This allows to use the new condition-test operator with assert:
assert 3 == 4 : error"not equal"
Because of the RHS of a condition test coercing a `string` to `error`
automatically, this now also covers the existing syntax for `assert`.
So the above is now (externally and internally) the same as:
assert 3 == 4 : "not equal"
* Add new "condition-test" operator. (Robin Sommer, Corelight)
This adds a new expression operator to both HILTI and Spicy:
`COND : ERROR`
where `COND` is a boolean expression, and `ERROR` is an expression of
type `error`. When evaluated, the new operator yields a value of type
`result<void>`, which will be true if the `COND` is true, and set to
`ERROR` if `COND` is false. In other words, this is a short-cut to
both test a condition and provide an error message in case it fails.
1.12.0-dev.204 | 2024-11-28 13:57:57 +0100
* Spell all `Attribute::Kind` values in Pascal case. (Benjamin Bannier, Corelight)
We are not 100% consistent, but we seem to prefer PascalCase over
SNAKE_UPPERCASE for enum values. Rename the recently introduced
`Attribute::Kind` values to that style.
An intended side effect of this is that we now avoid collisions with a
possibly defined `DEBUG` macro (e.g., Zeek defines it).
1.12.0-dev.202 | 2024-11-26 18:31:48 +0100
* GH-1901: Prefer enum over string for attributes. (Evan Typanski, Corelight)
1.12.0-dev.200 | 2024-11-25 16:14:56 +0100
* GH-1920: Fix some inconsistencies in random access docs. (Evan Typanski, Corelight)
* GH-1914: Make `$$` documentation more precise. (Evan Typanski, Corelight)
The documentation would refer to `$$` in an attribute and hook with
similar language (the "parsed" value). But, in an attribute, it refers
to the value before any conversions. In a hook, it refers to the value
after any conversions.
Now, "parsed" is used to refer to the pre-conversion value, and "final"
is used to refer to the post-conversion value. Those terms were chosen
because they make sense when no conversions are applied (or when `$$` is
used *in* a conversion). But, "final" can still be a little confusing,
since you can write to `$$`.
1.12.0-dev.196 | 2024-11-25 08:30:19 +0100
* GH-1918: Fix potential segfault with stream iterators. (Robin Sommer, Corelight)
When trimming off the beginning of a stream, an existing iterator
could end up dereferencing its internal chunk pointer even if the
chunk now no longer existed. The issue was inside the
increment/decrement operations, which didn't check if the *current*
iterator offset was still valid (because only then the current chunk
is guaranteed to be valid too).
* GH-1918: Add regression tests triggering #1918. (Robin Sommer, Corelight)
1.12.0-dev.193 | 2024-11-21 14:48:06 +0100
* GH-1919: Validate that sets are sortable. (Evan Typanski, Corelight)
1.12.0-dev.191 | 2024-11-14 11:18:29 +0100
* Locally disable some warnings for >=gcc-13. (Benjamin Bannier, Corelight)
fedora-40 and fedora-41 come with GCC versions which trigger new
warnings.
- Some uses of `visitor::range` trigger dangling-reference false
positives, https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107532.
- The dtor of `Location` triggers maybe-uninitialized false positives in
`hilti::rt::filesystem::path` for `_file`.
Disable these warnings locally if we are building with >=gcc-13 for now.
* Bump Fedora versions in CI. (Benjamin Bannier, Corelight)
1.12.0-dev.188 | 2024-11-07 15:19:35 +0100
* Make units encompass attributes in location. (Evan Typanski, Corelight)
* GH-1898: Move type alias attribute validation to parser. (Evan Typanski, Corelight)
Closes #1898
This was originally in a validator, but there's some sense that this is
a parsing issue since attributes should not be allowed at all on most
constructs of that form.
Note that a lot of the cleanup here was meant to span removing the
`opt_attributes` field in `type_decl` altogether, but the fallout from
that is pretty large. It ended up being less cleanup and more
rearchitecting. This way still avoids the hacky test for type aliases in
the validator while not changing the AST structure.
1.12.0-dev.182 | 2024-11-06 14:19:55 +0100
* GH-1859: Improve error message when a unit parameter is used as a
field. (Robin Sommer, Corelight)
1.12.0-dev.180 | 2024-11-06 08:41:55 +0100
* GH-1913: Avoid redundant computation during literal parsing. (Robin Sommer, Corelight)
* GH-1910: Optimize parsing of literal bytes. (Robin Sommer, Corelight)
We now create the bytes instance representing the literal as a global
singleton to avoid instantiating it over and over again.
Closes #1910.
* Introduce `UnsafeConstIterator` for bytes instances. (Robin Sommer, Corelight)
An unsafe iterator offers fast but unchecked access to the data. We
also rename `bytes::Iterator` to `bytes::SafeConstIterator` so that
for bytes we now follow the same two-tiered iterator structure as for
streams. We then switch some library code over to now use unsafe
iterators, gaining a noticeable speed-up in some cases.
* Add infrastructure to create and cache global constants. (Robin Sommer, Corelight)
So far we had replicated low-level code a few times that cached
constants through a global declaration for reuse. This now factors out
that logic into a central method inside the Spicy code generator.
This isn't full interning, which I'm not sure we want/need, but covers
the performance use case that has now come up a few times.
* Replace our poor hash function with `std::hash()`. (Robin Sommer, Corelight)
1.12.0-dev.173 | 2024-10-28 10:22:29 +0100
* GH-1908: Fix performance regression when parsing bytes. (Robin Sommer, Corelight)
Turns out our improved error messages were adding additional overhead
because we were now constructing them through `fmt()` each time we
needed more data, independent of whether there was actually going to
be an error reported.
This adds a second version of `waitForInput()` that doesn't receive an
already prepared error message, but just returns false on error so
that the caller can throw itself.
* GH-1857: Support `&requires`for individual vector items. (Robin Sommer, Corelight)
1.12.0-dev.169 | 2024-10-25 10:09:50 +0200
* GH-1895: Do no longer escape backslashes when printing strings or bytes. (Robin Sommer, Corelight)
Escaping backslashes in `print` output seems both unnecessary and
confusing.
* Rename `expandEscapes()` to `expandUTF8Escapes`. (Robin Sommer, Corelight)
For better readability to avoid confusion about the semantics of the
function.
* Introduce style flags for functions rendering values into strings. (Robin Sommer, Corelight)
This is just cleanup for better readability. I also fixed a couple of
cases where we now escape control characters where previously we
wouldn't, but I believe should have. Otherwise no functional change.
* GH-1893: Encompass child node's location in parent. (Evan Typanski, Corelight)
When a child is added with `addChild`, the parent's location should
(generally) span over that child as well. This primarily helps for cases
where a node doesn't have much of a location until it gets children
added - like `AttributeSet`. The locations for those should encompass
all of the attributes within the set. That logic applies for any node
with a child: if it's the child, then its location should reflect that.
1.12.0-dev.163 | 2024-10-24 09:59:35 +0200
* GH-1803: Fix namespacing of `hilti` IDs in Spicy-side diagnostic output. (Robin Sommer, Corelight)
We now show them with a `spicy` prefix, which makes more sense for
users.
* Add printer plugin hook to customize ID printing. (Robin Sommer, Corelight)
Because IDs are no longer AST nodes, we did not have a way for a
compiler plugin's printing code to modify how it would like them to be
printed. This adds a corresponding hook. It's not used anywhere yet,
but will be soon.
In addition, we add a notion of "user-visibility" to the printer API
so that printing code knows whether the resulting output is something
that will be shown to the user (e.g., in diagnostics), or remains
internal (e..g, raw code output, debugging output). Likewise not used
yet, but will be soon.
We also clean up the printer API a little bit.
* Add scope guard utility class. (Robin Sommer, Corelight)
1.12.0-dev.159 | 2024-10-23 17:33:28 +0200
* Bump pre-commit hooks (Benjamin Bannier, Corelight)
* Remove stray removal of directory in `/tmp` in test. (Benjamin Bannier, Corelight)
1.12.0-dev.156 | 2024-10-21 15:50:46 +0200
* GH-1063: Document arguments to `new` operator. (Robin Sommer, Corelight)
* GH-1858: Fix the literals parsers not following coercions. (Robin Sommer, Corelight)
1.12.0-dev.152 | 2024-10-18 15:31:40 +0200
* GH-1891: Fix GCC 11 compilation failure. (Evan Typanski, Corelight)
1.12.0-dev.150 | 2024-10-17 10:16:57 +0200
* GH-1792: Prioritize error message reporting unknown field. (Robin Sommer, Corelight)
This suppresses some non-interesting follow-up errors.
* GH-1790: Provide proper error message when trying access an unknown unit field. (Robin Sommer, Corelight)
Returning `type::Unknown` instead of `type::Auto` will let the
resolver process stop, allowing an already existing error message to
kick in later. We were doing it this way already for structs, but not
for units.
This also includes a fix for a bug with finding bitfield ranges by ID,
which was triggered by the change.
* GH-1791: Fix usage of `&convert` with unit's requiring parameters. (Robin Sommer, Corelight)
The generated code needs to create a temporary instance of the type,
but doesn't have any arguments to provide to it. But that's ok, and we
now let the validation pass and just instantiate a default-constructed
instance.
However, this change now requires an additional validator check on the
Spicy side to ensure fields giving arguments to types do so correctly
Before we happened to check that implicitly on the HILTI-side through
code that now would let it pass if no arguments were given.
* Factor out logic to validate arguments given to a type. (Robin Sommer, Corelight)
This will allow Spicy's validator to use it as well.
We also add two options to (1) accept usages where no argument is given at
all even when the type would normally require it,;and (2) skip any
actual type checking, and just confirm argument count. In a subsequent
change, we'll use (1) to fall-back to a type's default constructor,
and (2) to check type usage inside unit fields where we haven't
fully coerced arguments yet.
1.12.0-dev.144 | 2024-10-16 16:10:40 +0200
* GH-1861: Disallow ignored attributes on type aliases. (Evan Typanski, Corelight)
This may make previously fine code invalid. Since that code would likely
be hiding a bug, that seems okay to me.
Closes #1861
1.12.0-dev.140 | 2024-10-15 16:21:58 +0200
* GH-1871: Fix `&max-size` on unit containing a `switch`. (Robin Sommer, Corelight)
We would advance our input too early, letting the subsequent
`&max-size` check fail.
1.12.0-dev.138 | 2024-10-14 16:16:08 +0200
* GH-1868: Associate source code locations with current fiber instead of current thread. (Robin Sommer, Corelight)
This fixes potential location mix-ups when switching between fibers.
Note that we still need a context-wide fallback location as well
because we're not always running inside a fiber.
I ran a performance comparison before/after and couldn't measure a
difference. Looks like using TLS storage was a case of premature
optimization.
1.12.0-dev.136 | 2024-10-14 14:20:16 +0200
* Disable cpplint iwyu check. (Benjamin Bannier, Corelight)
* Replace a few C-style casts with `static_cast`. (Benjamin Bannier, Corelight)
1.12.0-dev.133 | 2024-10-08 21:50:53 +0200
* Document `continue` statements. (Evan Typanski, Corelight)
1.12.0-dev.131 | 2024-10-08 17:09:30 +0200
* Fix "it's" used as a possessive. (Evan Typanski, Corelight)
1.12.0-dev.129 | 2024-10-07 13:02:38 +0200
* Stop using deprecated pre-commit stage names. (Benjamin Bannier, Corelight)
The names we used have been deprecated since some time, see
https://github.com/pre-commit/pre-commit/issues/2732.
1.12.0-dev.127 | 2024-10-05 13:21:27 +0200
* GH-1867: Fix infinite loops with recursive types. (Evan Typanski, Corelight)
Closes #1867
There are two different cases where infinite loops happen with recursive
types. First, a type may reference itself (`type Data = Data`). Second,
a type may reference itself inside some other type (`type Data =
vector<Data>`).
The first is fixed with a recursion limit. Since the type simply cannot
resolve, it doesn't get anywhere near codegen. You could detect cycles,
but that introduces some extra overhead and complexity that shouldn't
be needed in a "simple" function.
The second is fixed with an ad-hoc "occurs" check in type unification.
That just detects cycles and aborts if one is present. This could be
placed at some other place in the "resolve until convergence" loop, but
it seems best put closest to the source of the issue.
1.12.0-dev.124 | 2024-09-30 17:43:41 +0200
* GH-1875: Fix potential nullptr dereference when comparing streams. (Robin Sommer, Corelight)
Because we are operating on unsafe iterators, need to catch when one
goes out of bounds.
1.12.0-dev.122 | 2024-09-30 14:18:09 +0200
* GH-1874: Add new library function `spicy::bytes_to_mac`. (Robin Sommer, Corelight)
```
## Returns a bytes value rendered as a MAC address string (i.e., colon-separated hex bytes).
public function bytes_to_mac(value: bytes) : string;
```
* Optimize `spicy::bytes_to_hexstring` and `spicy::bytes_to_mac`. (Robin Sommer, Corelight)
1.12.0-dev.119 | 2024-09-30 13:37:10 +0200
* GH-1846: Fix bug with captures groups. (Robin Sommer, Corelight)
When extracting the data matching capture groups we'd take it from the
beginning of the stream, not the beginning of the current view, even
though the latter is what we are matching against.
* Add missing trim after matching a regular expression. (Robin Sommer, Corelight)
1.12.0-dev.116 | 2024-09-30 13:35:49 +0200
* GH-1842: Fix when input redirection becomes visible. (Robin Sommer, Corelight)
With `&parse-at/from` we were updating the internal state on our
current position immediately, meaning they were visible already when
evaluating other attributes on the same field afterwards, which is
unexpected.
1.12.0-dev.114 | 2024-09-30 10:24:42 +0200
* GH-1844: Fix nested look-ahead parsing. (Robin Sommer, Corelight)
When parsing nested vectors all using look-ahead, we need to return
control back to upper level when an inner look-ahead isn't found.
This may change the error message for "normal" look-ahead parsing (see
test baseline), but the new one seems fine and potentially even
better.
* Apply compiler suggestion. (Robin Sommer, Corelight)
1.12.0-dev.111 | 2024-09-26 20:00:42 +0200
* Remove `ast-stats` output from tests baselines. (Robin Sommer, Corelight)
These are too noisy as they update with every AST change. Originally I
included `ast-stats` into some test baselines to ensure the new AST
infrastructure isn't doing weird stuff, but seems that's working out
ok.
1.12.0-dev.109 | 2024-09-26 17:29:17 +0200
* Fix parsing ambiguity with properties. (Robin Sommer, Corelight)
In the past, we had special-cased properties in our Flex/Bison parser
so that when parsing an expression, they wouldn't be recognized.
However, that now led an error field hook of the form `x: bytes
&size=42 %error` to be parsed as `&size=(42 % error)`. We now switch
to white-listing all known properties, just as we already do for
attributes. That way conflicts should be extremely rare.
* Redo error handling docs (Robin Sommer, Corelight)
The old text was very outdated. This extends the content, documents
the new per-field `%error` handler, and moves it all into the
"parsing" section to have it closer to the error recovery content.
* GH-1824: Add support for field-local `%error` handlers. (Robin Sommer, Corelight)
We now support attaching an `%error` handler to an individual field:
type Test = unit {
a: b"A";
b: b"B" %error { print "field B %error", self; }
c: b"C";
};
With input `AxC`, that handler will trigger, whereas with `ABx` it
won't. If the unit had a unit-wide `%error` handler as well, that one
would trigger in both cases (i.e., for `b`, in addition to its field
local handler).
The handler can also be provided separately from the field:
on b %error { ... }
In that separate version, one can receive the error message as well by
declaring a corresponding string parameter:
on b(msg: string) %error { ... }
This works externally, from outside the unit, as well:
on Test::b(msg: string) %error { ... }
This is eebased on top of `topic/robin/optimize-type-parsing` so that
we get the peephole optimizer.
* Refactor field hook attributes. (Robin Sommer, Corelight)
Currently we only have two types of field hooks: standard hooks and
`foreach` hooks. To prepare for more types, this refactors the code
to represent the type with an `enum` instead of a `foreach` boolean. It
also moves validation of permitted attributes from the parser to
the validator.
1.12.0-dev.104 | 2024-09-26 14:14:36 +0200
* Fine-tune `bytes` literal parsing further. (Robin Sommer, Corelight)
This now optimizes the generated code for parsing a `bytes` literal
based on what we're going to use it for, or not (require literal vs.
look-ahead vs skip).
The standard case of a parsing an expected literal now looks like
this:
```
# Begin parsing production: Ctor: b1 -> b"abc" (const bytes)
(*self).b1 = spicy_rt::expectBytesLiteral(__data, __cur, b"abc", "../tests/spicy/types/bytes/parse-length.spicy:20:10-20:15", Null);
__cur = __cur.advance(3);
if ( __trim )
(*__data).trim(begin(__cur));
# End parsing production: Ctor: b1 -> b"abc" (const bytes)
```
* Add peephole optimization simplifying rethrowing catch blocks. (Robin Sommer, Corelight)
The optimization turns this:
```
function void foo() {
try {
...
} catch {
throw;
}
}
```
into
```
function void foo() {
{
...
}
}
```
It would be even nicer if we didn't need the braces around the
remaining block, but it's generally not safe to remove them because if
the block declares any locals their life-times and visibility would
change.
* GH-1592: Add peephole optimizer for final AST tuning. (Robin Sommer, Corelight)
We use this to remove two statement constructs that the main optimizer
may leave behind:
1. `default<void>()`
2. `(*self).__error = __error; __error = (*self).__error;`
The second case is a quite specific situation that eventually, once we
have CFG/DFG tracking, the main optimizer should be able to cover more
generically. However, for now, it's just not nice to always have
these blocks in the generated C++ code, so adding this special case
seems useful.
Couples notes on (2):
- Per #1592, case 2 may also have overhead. Closes #1592.
- Technically, this optimization isn't always correct: subsequent
code could assume that `(*self).__error` is set, whereas after
removal it's not (or not to the expected value). However,
`__error` is strictly-internal state, and we know that we don't
use it any different, so this seems ok until we have more
general optimizer logic.
* Disable check for reserved IDs with `--skip-validation`. (Robin Sommer, Corelight)
This allows to write tests that use internal IDs.
* Simplify parsing of literals. (Robin Sommer, Corelight)
There are two changes in here:
1. For grammars that don't use look-ahead, we skip the runtime check for a
pending look-ahead symbol, because we know we will never have one. This removes
generated code of the form `if ( _lah ) { ... }`. This change needs a bit of
machinery unfortunately because we need to get the information about look-ahead
usage over into the codegen for literal parsing.
2. For bytes literals, we now outsource their parsing to a runtime function to
make the generated code simpler. As a side effect this also provides more
informative error messages when the literal isn't found.
Taking the two together means that the code for parsing a plain `b"Foo"`
literal may look like this now:
spicy_rt::expectBytesLiteral(__data, __cur, b"Foo", "b.spicy:4:10-4:15", Null);
__cur = __cur.advance(3);
if ( __trim )
(*__data).trim(begin(__cur));
(*self).foo = b"Foo";
* Optimize parsing for `bytes &size=N`. (Robin Sommer, Corelight)
* Add mode for optimize types parsing. (Robin Sommer, Corelight)
This provides the surrounding infrastructure, but does not yet
implement it for any type.
* Refactor type parsing modes. (Robin Sommer, Corelight)
So far we had a boolean flag to differentiate between "normal" and
"try" parsing. We turn this flag into an enum now so that we can more
easily extend the set of modes later. No functional change otherwise.
* Refactor pre/post parsing logic. (Robin Sommer, Corelight)
This splits out the `&size/&max-size` handling so that we can
special-case that later.
* Cleanup: replace optionals with pointers. (Robin Sommer, Corelight)
For AST nodes, we can/should use nullptrs instead of unset optionals.
The versions using `optional` were still a left-over from the old AST
code.
1.12.0-dev.93 | 2024-09-26 13:07:41 +0200
* Disallow unit variables nested into other items. (Robin Sommer, Corelight)
Initialization would be ill-defined.
* GH-1839: Support `if`-blocks in unit. (Robin Sommer, Corelight)
We now support `if` around a block of unit items:
type X = unit {
x: uint8;
if ( self.x == 1 ) {
a1: bytes &size=2;
a2: bytes &size=2;
};
};
One can also add an `else`-block:
type X = unit {
x: uint8;
if ( self.x == 1 ) {
a1: bytes &size=2;
a2: bytes &size=2;
}
else {
b1: bytes &size=2;
b2: bytes &size=2;
};
};
* Add a new unit item `Block` that stores a sequence of subitems. (Robin Sommer, Corelight)
We now have a new unit item `Block` that represents a sequence of
sub-items. This goes along with a new production `Block` that the item
gets turned into inside the grammar.
The implementation reuses logic from `unit::item::switch_::Case`. We
then refactor `Case` to use the new `Block` instead of continuing to
maintain its own list of items.
We also refactor some other logic previously located with `Switch`
to now apply to `Blocks` as well.
Optionally, a `Block` can have a conditional expressions, an
"else"-block as well attributes with it. We don't use them yet, but
will soon; tests then coming as well. However, we already also
refactor some attribute logic that previously went just with the
`Switch` unit item to now also apply to the `Block` item, so that the
two can support the same attributes.
* Remove `production::Boolean`. (Robin Sommer, Corelight)
Turns out this was neither used nor fully implemented.
* Run a final resolver pass after optimization. (Robin Sommer, Corelight)
Optimization may leave node state unset, such as canonical IDs if we
created new nodes. The final resolver pass gets that back to a
well-defined state.
This also moves the debug output for `ast-final` out of the resolver
loop; it now executes at the very end after optimization. That's a
change I've been meaning to do for a while because the "final" AST
wasn't really final. `ast-resolved` now shows what used to be
`ast-final` (and no longer each iteration of the resolver).
Test coming with subsequent commit, where a new test would break
otherwise.
1.12.0-dev.87 | 2024-09-26 13:07:06 +0200
* Compile generated C++ with `-O0` in debug mode. (Robin Sommer, Corelight)
* Bump centos-stream in documentation. (Benjamin Bannier, Corelight)
* Bump latest release in docs. (Benjamin Bannier, Corelight)
1.12.0-dev.83 | 2024-09-19 08:58:15 +0200
* Fix attributes not getting validated. (Evan Typanski, Corelight)
This particularly fixes the case where a ctor within a field will not
have its attributes validated because it does not go through
`isParseableType`. The "generic attribute checks" are moved to apply to
all attributes, while type-specific ones remain in `isParseableType`
1.12.0-dev.81 | 2024-09-18 14:42:34 +0200
* Add internal error for unvalidatable vectors. (Evan Typanski, Corelight)
Vectors were changed to never be a vector-of-type, just a
vector-of-items that redirect to a type. This allows validating
attributes and whatnot properly, which only apply to fields, not types.
The consequence is that a vector-of-types should be impossible, or else
there is an error in the parser's workarounds to get to that state,
which is a bug.
Add an internal error so that case never surfaces without a big scary
error.
* Move legacy vector syntax check to new production. (Evan Typanski, Corelight)
* Remove vector builders for type/ids. (Evan Typanski, Corelight)
Currently, vectors need the inner type to actually point to a field.
This removes the repeat from builder methods that would otherwise need a
field but don't so that they aren't mistakenly added.
1.12.0-dev.77 | 2024-09-11 08:18:38 +0200
* GH-1847: Fix resynchronization issue with trimmed input. (Robin Sommer, Corelight)
When input had been trimmed, `View::advanceToNextData` could end up
returning a view starting ahead of the valid area.
* Add missing printer support for exception values. (Robin Sommer, Corelight)
* GH-1860: Fix parsing for vectors of literals. (Robin Sommer, Corelight)
This was broken in two ways:
1. with the `(LITERAL)[]` syntax, the parser would not recognize literals
using type constructors
2. with the syntax `LITERAL[]`, we'd try to store the parsed value
into a vector
* Fix error message. (Robin Sommer, Corelight)
1.12.0-dev.71 | 2024-09-06 16:29:02 +0200
* Move some `scoped_id` out for better locations. (Evan Typanski, Corelight)
* Fix `vector<>` syntax missing diagnostics. (Evan Typanski, Corelight)
* GH-1832: Fail for vectors with bytes but no stop. (Evan Typanski, Corelight)
Cases with parentheses were caught before this change. The comments
indicate behavior before the change, even though both cases should fail:
public type BytesVectors = unit {
: bytes[10] &size=5; # No diagnostic - oh no!
: (bytes)[10] &size=5; # Error - good
};
That's because a new `_anon` field is created with the parentheses. This
makes the `bytes` in an item, which can have its own `&size` and gets
checked later. But `bytes` by itself cannot have the `&size` attribute
since it's just a type - the vector will use its own `&size` attribute!
The fix makes all types in vectors use the anonymous field instead, so
their lack of attributes are properly modeled during validation.
1.12.0-dev.67 | 2024-09-04 12:44:42 +0200
* GH-1852: Fix `skip` with units. (Robin Sommer, Corelight)
For unit parsing with `skip`, we would create a temporary instance
but wouldn't properly initialize it, meaning for example that
parameters weren't available. We now generally fully initialize any
destination, even if temporary.
* Do not by default skip CI in update-changes cfg. (Benjamin Bannier, Corelight)
We rely on CI to create releases, so having `[skip CI]` in our
update-changes config was always a hassle which needed to be edited out
when creating a release (just tagging would autocommit with the message
template without a way to change the commit message before tagging).
Drop this part to make releases easier and less error prone.
1.12.0-dev.63 | 2024-08-26 17:10:07 +0200
* Bump 3rdparty/any from `7c76129` to `a05d5ad` (dependabot[bot])
1.12.0-dev.61 | 2024-08-14 14:04:27 +0200
* Suppress warning `invalid-offsetof`. (Robin Sommer, Corelight)
We now run into these in CI due to `-Werror`.
Also clean up existing duplication of
`-Wno-unused-command-line-argument`.
* GH-1835: Fix computation of declaration dependencies. (Robin Sommer, Corelight)
One more branch we had to follow to track down all dependencies
between global declarations: external qualified types.
1.12.0-dev.58 | 2024-08-09 15:57:52 +0200
* Add `string::lower` and `string::upper` (Evan Typanski, Corelight)
These already had a whole impl in the `string` namespace, so this just
hooks it up so you can use them reasonably on a string type in a spicy
file. It also switches those impls to use `string_view`
* Fix `split` and `split1` docs to be more accurate (Evan Typanski, Corelight)
The documentation for the bytes method seemed incorrect (or, at least,
confusing) where an empty separator did *not* split on white space. This
correctly documents the behavior with what the actual behavior is
(especially for split1).
Note, the behavior is a bit confusing, unintuitive, and maybe
inconsistent between the implementations of `split` and `split1`.
Namely, `split1` will sometimes but the entire value in the first
element of the tuple (if a match isn't found) or the second element (if
the separator is just an empty string).
For now this just updates the documentation, but I'd also welcome a
change in behavior.
* Add `string::split` and `string::split1` helpers (Evan Typanski, Corelight)
This doesn't do any work to fix the TODOs that were already in the Bytes
version. Also, there is a small unclear bit about the documentation for
the `split` op in both Bytes and now string, namely that an empty `sep`
argument should make the break take place at sequences of whitespace,
but it does not do that. It may also be misreading the documentation,
but one of the two should probably be clarified.
* Add `string::starts_with` helper (Evan Typanski, Corelight)
1.12.0-dev.53 | 2024-08-09 15:45:05 +0200
* GH-1831: Fix optimizer regression. (Robin Sommer, Corelight)
We were no longer marking types as used that are referenced through a
type name.
1.12.0-dev.51 | 2024-08-09 15:43:51 +0200
* Fix AST dependency computation. (Robin Sommer, Corelight)
Turns out the logic wasn't quite right yet. It worked well enough if
all units get wrapped into `ValueReference` because then we just need
a forward declaration to make that work; and those we would always
output first. But it wouldn't actually track dependencies correctly to
output them in the right order so that full struct are available where
needed. This commit reworks that logic. Turns out this also further
simplifies logic and state tracking, which is a good sign.
* Provide dependency information to AST transformations. (Robin Sommer, Corelight)
Before, we computed dependency information only once just before final
codegen, meaning that Spicy-to-HILTI lowering would not have it
available. This adds a separate dependency computation just before the
transformation pass. While this isn't used yet, the change prepares
for future functionality leveraging dependencies during lowering (like
selective application of `&on-heap`).
* Make AST dependency information deterministic. (Robin Sommer, Corelight)
The dependency information returned by the context wasn't sorted, with
order depending on pointer values. That means that users of that
information would end up processing dependencies in an undetermined
order, making their results non-deterministic.
* Fix an error message. (Robin Sommer, Corelight)
1.12.0-dev.45 | 2024-08-09 14:01:49 +0200
* Silence unused result warnings when expression has [[nodiscard]]. (Arne Welzel, Corelight)
The code generated by assert-exception *o == "", where o is an optional
results in an unused result warning due to operator== being
[[nodiscard]].
* Add -Wno-unused-command-line-argument to LD_FLAGS. (Arne Welzel, Corelight)
Suppress warnings when flags on the command line aren't used:
c++: error: -lspicy-rt-debug: 'linker' input unused
[-Werror,-Wunused-command-line-argument]
* Add Ubuntu 24.04 to CI. (Arne Welzel, Corelight)
* Avoid maybe-uninitialized warning. (Arne Welzel, Corelight)
Not sure, but Ubuntu 20's GCC 9.4 and OpenSUSE 15.5's GCC 7.3 are reporting
/hilti/runtime/include/hilti/rt/types/map.h:235:21: error: '<anonymous>'
may be used uninitialized in this function [-Werror=maybe-uninitialized]
* Fix sys/errno.h warning. (Arne Welzel, Corelight)
On Alpine 3.18:
/usr/include/sys/errno.h:1:2: error: #warning redirecting incorrect
#include <sys/errno.h> to <errno.h> [-Werror=cpp]
1 | #warning redirecting incorrect #include <sys/errno.h> to <errno.h>
* Avoid use-after-free warning with GCC-13.2. (Arne Welzel, Corelight)
Not exactly sure what's going on here. I *think* calling data->begin()
and data->end(), it sees/assumes one path freeing the underlying chain
of the SafeConstIterator and the other then accessing the freed pointer.
Maybe something about the evaluation of the arguments being unspecified.
It doesn't trigger when fetching offsets sequentially beforehand, so do
that...
inlined from 'spicy::rt::detail::printParserState(...)::<lambda()>'
at /src/spicy/runtime /src/parser.cc:81:19:
/src/hilti/runtime/include/hilti/rt/intrusive-ptr.h:43:20:
error: pointer used after 'void operator delete(void*, std::size_t)'
-Werror=use-after-free]
43 | if ( m && --m->_references == 0 )
| ~~~^~~~~~~~~~~$
* Use Bytes directly instead of uint8_t array for pack(). (Arne Welzel, Corelight)
GCC 12.2 on Debian 12 complains that raw might be unused. Replace
with Bytes/std::string and casting data() to uint8_t* instead.
Warning was:
inlined from 'hilti::rt::Bytes hilti::rt::Address::pack(
hilti::rt::ByteOrder) const'
at /src/hilti/runtime/src/types/address.cc:132:54:
/usr/include/c++/12/bits/char_traits.h:431:56: error:
'raw' may be used uninitialized [-Werror=maybe-uninitialized]
* Don't ignore alloca() return value in test. (Arne Welzel, Corelight)
Silence warning on Alpine:
fiber.cc:314:11: error: ignoring return value of 'void*
__builtin_alloca(long unsigned int)' declared with attribute
'warn_unused_result' [-Werror=unused-result]
* Disable -Werror when precompiling libspicy.h headers (Arne Welzel, Corelight)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=47857
* Use operator() from OptimizerVisitor via using. (Arne Welzel, Corelight)
This triggers -Werror=overloaded-virtual= on GCC 13.2 otherwise.
* Fix --enable-werror werror_flags. (Arne Welzel, Corelight)
Found while trying to test something with EXTRA_CXX_FLAGS.
This got lost in e27cd5ac0443659e7e9fdae2f3bbe5920bd3564b.
1.12.0-dev.33 | 2024-08-07 12:38:27 +0200
* Add an AST pass collecting dependencies between declarations. (Robin Sommer, Corelight)
* Rework emission of C++ code. (Robin Sommer, Corelight)
We now use the new AST dependency information to determine in which
order to output C++ declarations, including whether to output them at
all. Previously, we used a needed to get the desired order through
a pretty hacky, adhoc scheme that would prioritize some types
manually as needed.
This includes:
- We now only declare C++ types that are actually needed by the unit;
no more bulk import everything from an imported unit.
- Remove all the manual scheduling of when `cxx::declaration::*` are
to be emitted (no more forward declarations, no more manual
prioritization). The order of output is now generally determined
through dependencies. For example, before we output a type, we
output anything it depend on.
- Merge `cxx::Function` into `cxx::declaration::Function`. This means
we generally don't need to emit separate declaration vs
implementation anymore; instead we decide at emission time what
parts to output.
- Unify treatment of cxx::declaration::*: we go through emission
phases now where each type decides what to emit when.
- Remove various state from cxx::declaration::* that’s no longer
needed/used.
- We now unconditionally forward declare structs and unions; seems
cheap enough and makes things quite a bit easier.
- We new logic strives to maintain the same output where possible so
that we only get minimal baselines changes (mainly some small
ordering changes).
* Properly set parse extension when importing by path. (Benjamin Bannier, Corelight)
* Rename `cxx::type::Struct`'s `inlineCode` to `code`. (Benjamin Bannier, Corelight)
This function now generates not `inline` code, but instead code with
bodies which we would only want to include once in the generated C++.
* Tweak when we emit struct inline code. (Benjamin Bannier, Corelight)
We would previously emit inline C++ code for types into each unit module
where the type's module was imported. This meant that we potentially
compiled identical function bodies in many files. While this could in
theory lead to better local inlining decisions it also caused poor
scaling as the number of modules in a Spicy grammar grows (in
particular).
With this patch we now emit these bodies at most once, in particular
into the module of the type. To support this we make a number of
declarations and definitions not `inline` anymore so they can link
correctly. We also reduce which code gets included when importing a
module.
* Only emit full C++ body of functions in module of matching namespace. (Benjamin Bannier, Corelight)
* Simplify body of generated `operator<<`. (Benjamin Bannier, Corelight)
* Remove unused methods to retrieve module dependencies. (Robin Sommer, Corelight)
If we ever need this, we can now use the new declaration dependencies
instead.
* Fix `Node::tryAs_()`. (Robin Sommer, Corelight)
(Not used anywhere currently, but didn't work and want to keep.)
* Fully qualify runtime identifiers in generated code. (Robin Sommer, Corelight)
We weren't consistent doing that everywhere, which would lead to
trouble with upcoming changes.
1.12.0-dev.21 | 2024-08-07 10:49:10 +0200
* GH-1823: Don't qualify magic linker symbols with C++ namespace. (Robin Sommer, Corelight)
We need them at their original values because that's what the runtime
lbirary is hard-coded to expect.
1.12.0-dev.19 | 2024-08-06 15:01:21 +0200
* Fix use of move'd from variable. (Benjamin Bannier, Corelight)
Function parameters still shadown members in C++. This is a fixup of
c3abbbe8eb28d1e2cee5b58b19e1a30710b38853.
1.12.0-dev.17 | 2024-08-06 13:49:47 +0200
* GH-1817: Prevent null ptr dereference when looking on nodes without `Scope`. (Benjamin Bannier, Corelight)
Closes #1817.
1.12.0-dev.15 | 2024-08-06 13:21:57 +0200
* GH-3865: Add CMAKE_CXX_FLAGS to HILTI_CONFIG_RUNTIME_LD_FLAGS. (Arne Welzel, Corelight)
When building Spicy with CXXFLAGS='--coverage', a subsequent
spicy-driver invocation fails with:
$ ./build/bin/spicy-driver ./x.spicy
[error] failed to load library "/tmp/__library__c2d0d689ca0dbabb.hlto":
/tmp/__library__c2d0d689ca0dbabb.hlto: undefined symbol: __gcov_merge_add
Similarly for `spicyz` and Zeek.
CMake adds CMAKE_CXX_FLAGS to all its linker invocations [1] which is why
building Spicy's executables itself works out. Prepend EXTRA_CXX_FLAGS
and CMAKE_CXX_FLAGS to HILTI_CONFIG_RUNTIME_LD_FLAGS as a fix.
Unclear if building Spicy with `--coverage` should imply building produced
`.hlto` files with coverage, too, but that's what is done with other CXXFLAGS
already today.
Relates to zeek/zeek#3865.
[1] https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html
1.12.0-dev.13 | 2024-08-06 12:51:46 +0200
* Updates <bytes>.to_int() <bytes>.to_uint() documentation. (Arne Welzel, Corelight)
I'm going for RuntimeError for now, given that's at least not wrong
given what's thrown in various places.
* Fix undefined shifts of 32bit integer in toInt(). (Arne Welzel, Corelight)
1U is 32bit on a 64bit system and shifting it by more than 31 bits
is undefined. The following does currently produce -4294967296 instead
of -1:
b"\xff\xff\xff\xff".to_int(spicy::ByteOrder::Big)
* Fix to_uint(ByteOrder) for empty byte ranges. (Arne Welzel, Corelight)
to_uint() and to_int() for empty byte ranges throw when attempting to
convert printable decimals to integers. Do the same for the byte order
versions. The assumption is that it is really an error when the user
calls to_int() or to_uint() on an empty byte range.
1.12.0-dev.9 | 2024-08-02 12:30:03 +0200
* Simplify test. (Benjamin Bannier, Corelight)
This test was using complicated input data and parsing it in a
hard to follow way. This patch simplifies the test data and its
handling.
* GH-1815: Disallow expanding limited `View`s again with `limit`. (Benjamin Bannier, Corelight)
The documented semantics of `View::limit` are that it creates a new view
with equal or smaller size. In contrast to that we would still have
allowed to expand views with more calls `limit` again as well.
This patch changes the implementation of `View::limit` so it can only
ever make a `View` smaller.
We also tweak the implementation of the check for consumed `&size` when
used together with `&eod`: if the `&size` was already nested in a
limited view a larger `&size` value could previously extend the view so
the `&eod` effectively was ignored. Since we now do not extend the
`View` anymore we need to only activate the check for consumed `&size`
if `&eod` was not specified since in this case the user communicated that
they are fine with consuming less data.
Closes #1815.
1.12.0-dev.6 | 2024-07-31 14:24:26 +0200
* Start v1.12.0 development. (Benjamin Bannier, Corelight)
1.11.0-dev.287 | 2024-07-31 11:15:49 +0200
* Remember normalized paths when checking for duplicate files in driver. (Benjamin Bannier, Corelight)
* Remember files processed by the driver. (Benjamin Bannier, Corelight)
* GH-1813: Fix equality implementation of module UID. (Benjamin Bannier, Corelight)
1.11.0-dev.282 | 2024-07-30 17:02:31 +0200
* GH-1810: Fix nested look-ahead switches. (Robin Sommer, Corelight)
1.11.0-dev.279 | 2024-07-26 12:47:30 +0200
* GH-1808: Fix non-converging optimizer pass for used functions. (Benjamin Bannier, Corelight)
1.11.0-dev.277 | 2024-07-22 14:44:40 +0200
* Add rsync to CI Docker image. (Benjamin Bannier, Corelight)
`rsync` is used by `doc/scripts/autogen-docs` and without it the script
cannot run.
* Fix running of `autogen-docs` pre-commit hook. (Benjamin Bannier, Corelight)
If the `diagrams` package was not installed this hook would now fails.
Since it runs for changes to any type of file this was not ideal.
This patch adds a check for that dependency and stops the docs update
script if it is missing.
1.11.0-dev.274 | 2024-07-22 12:01:04 +0200
* GH-1795: Fix illegal stream content access through invalid view. (Robin Sommer, Corelight)
When a view's beginning ended up outside of the stream's valid range
(e.g., because the stream has been trimmed in the meantime), we
wouldn't catch that on access and attempt to access data that was no
longer valid. This commit extends the safety checks accordingly.
1.11.0-dev.272 | 2024-07-22 10:53:56 +0200
* GH-1800: Add test for invoked but unimplemented sink hooks. (Benjamin Bannier, Corelight)
This is a regression test for #1800.
* GH-1800: Guard access to sink hooks. (Benjamin Bannier, Corelight)
If no sink hooks were specified by the user trying to invoke them would
raise a `std::bad_function_call` previously since we did not have
explicitly guard the code. With #1736 we us infrastructure with less
implicit checks so we need to handle the absence of a value ourself
since otherwise we might end up dereferencing a nullptr.
1.11.0-dev.269 | 2024-07-22 10:53:15 +0200
* Do not search for Python in CMake setup. (Benjamin Bannier, Corelight)
We previously needed this since during the build we were generating
files with Python. This is not the case anymore.
1.11.0-dev.267 | 2024-07-22 10:50:22 +0200
* GH-1462: Defer start of feature-dependent transformations to a later point. (Benjamin Bannier, Corelight)
We previously would evaluate whether a unit field or method was required
by an optional feature before we had finished collecting all feature
requirements. This behavior was fine when we were visiting individual
modules' AST one by one, but breaks with #1462 were we changed using a
single AST to hold all modules.
This patch defers transformations until all feature requirements have
been collected.
1.11.0-dev.264 | 2024-07-22 09:27:15 +0200
* Update developer's documentation. (Robin Sommer, Corelight)
Includes:
- Rework and update developer documentation's architecture diagram.
This is now generated through Python's `diagrams` package (i.e.,
graphviz). Because the layout is automatically determined by `dot`,
it's not perfect; but seems good enough.
- Update debugging streams.
- Remove benchmarking section, as this was completely outdated.
1.11.0-dev.262 | 2024-07-15 11:49:46 +0200
* Remove unused `Engine` type and tracking. (Benjamin Bannier, Corelight)
* Remove API customization points for `spicy::Engine`. (Benjamin Bannier, Corelight)
* Remove support for customizable Spicy hook and field engine. (Benjamin Bannier, Corelight)
1.11.0-dev.258 | 2024-07-12 16:34:16 +0200
* GH-1784: Fix import HILTI vs Spicy confusion. (Robin Sommer, Corelight)
Our tracking of module dependencies could end up mixing `.spicy`
and `.hlt` modules when the former transitioned into the latter.
1.11.0-dev.255 | 2024-07-12 13:46:57 +0200
* Update lists of releases in docs [skip CI]. (Benjamin Bannier, Corelight)
* Bump latest release in docs [skip CI]. (Benjamin Bannier, Corelight)
* Drop docs around `try`/`catch` [skip CI]. (Benjamin Bannier, Corelight)
This feature is not on the horizon and having it documented prominently
painted an incorrect picture for users. Drop its docs for now until we
actually provide it.
1.11.0-dev.251 | 2024-07-09 18:16:15 +0200
* Drop unused include. (Benjamin Bannier, Corelight)
* GH-1785: Prevent defining already forwarded constants. (Benjamin Bannier, Corelight)
We previously would add both forward declarations as well as full
definitions for any constant in an imported module, i.e. we emitted
the same constant in multiple C++ files which the linker seems to have
cleaned up for us.
With this path we only emit a definition if we are not already emitting
a forward declaration.
Closes #1785.
* Remove duplicate emission of types. (Benjamin Bannier, Corelight)
This was a noop but still useless work.
1.11.0-dev.247 | 2024-07-09 10:38:23 +0200
* Add missing include. (Benjamin Bannier, Corelight)
1.11.0-dev.244 | 2024-07-09 09:23:01 +0200
* Document spicy-driver's `@gap` for batch input. (Robin Sommer, Corelight)
1.11.0-dev.242 | 2024-07-04 12:19:21 +0200
* Fix copy-n-paste error in GH action name. (Benjamin Bannier, Corelight)
* Fix when GH `email-failure` action sends notification. (Benjamin Bannier, Corelight)
In order to detect whether to send out a failure email we previously
would check that the action conclusion contained certain statuses. This
check seems to have been nonsensical as we only notified for benign
failure reasons (the check would have made a little more sense if
negated, but would have then also notified on successes).
With this patch we instead explicitly enumerate the failure conclusions
to trigger on, see
https://docs.github.com/en/webhooks/webhook-events-and-payloads#check_suite--check_suite-object
for the full list of conclusions.
1.11.0-dev.239 | 2024-07-04 12:13:40 +0200
* Drop macos versions unsupported by Cirrus. (Benjamin Bannier, Corelight)
1.11.0-dev.237 | 2024-07-04 12:13:03 +0200
* GH-1775: Extend validation of synchronization properties. (Robin Sommer, Corelight)
Closes #1775.
1.11.0-dev.235 | 2024-07-04 12:09:41 +0200
* GH-1777: Fix interning of regexps for `%skip*`. (Benjamin Bannier, Corelight)
If we used regexps for `%skip`, `%skip-pre` or `%skip-post` we would
previously compute unique identifiers in a way which clashed with
general regexp interning elsewhere[^1].
With this patch we now pick a unique stem for `%skip*`-related regexps
so they do not clash anymore.
[^1]: https://github.com/zeek/spicy/blob/90751715212c7268ebb86276d4365dfaedd94882/spicy/toolchain/src/compiler/codegen/parsers/literals.cc#L115
1.11.0-dev.233 | 2024-07-04 12:08:15 +0200
* GH-1774: Fix synchronization with symbol different from last lookahead token. (Benjamin Bannier, Corelight)
If `synchronize-[at|after]` used a lookahead symbol different from the
last lookahead symbol (e.g., normal parsing used a
literal, recovery a regexp), synchronization would clobber the lookahead
symbol so that after successful synchronization normal parsing could
still not succeed.
This patch sets up dedicated parser state for synchronization to prevent
the clobbering.
* Use symbol unique for each error recovery. (Benjamin Bannier, Corelight)
By using a constant symbol name we could run into situations where
different error recoveries on different regexps (e.g., in unrelated
units) could accidentally use the same symbol. This is due to regexps
being automatically interned.
With this patch we generate unique symbols for each position where we
recover.
* Add debug log of parser state during synchronization. (Benjamin Bannier, Corelight)
We have a similar debug log already when synchronizing on regexps, but
where missing it for other literals.
1.11.0-dev.229 | 2024-07-03 12:00:00 +0200
* Call `%sync_advance` hook also when entering sync mode. (Benjamin Bannier, Corelight)
This allows users to control the full synchronization lifecycle, from
entering it, to regular checks during the search (both via
`%sync_advance`, until it is left (via their explicit `confirm`, e.g.,
from a `%synced` hook).
1.11.0-dev.227 | 2024-06-28 13:20:55 +0200
* Improve doc wrt. `%error` hook in sync mode. (Jan Grashoefer, Corelight)
* Add draft release notes for spicy-1.11.0. [skip CI] (Benjamin Bannier, Corelight)
1.11.0-dev.224 | 2024-06-25 08:50:58 +0200
* GH-3779: Add `%sync_advance` hook. (Robin Sommer, Corelight)
This adds support for a new unit hook:
```
on %sync_advance(offset: uint64) {
...
}
```
This hook is called regularly (see below) during error recovery when
synchronization skips over data or gaps while searching for a valid
synchronization point. It can be used to check in on the
synchronization to, e.g., abort further processing if it just keeps
failing. `offset` is the current position inside the input stream that
synchronization just skipped to.
By default, "called regularly" means that it's called every 4KB of
input skipped over while searching for a synchronization point. That
value can be changed by setting a unit property
`%sync-advance-block-size = <number of bytes>`.
As an additional minor tweak, this also changes the name of what used
to be the `__gap__` profiler to now be called `__sync_advance` because
it's profiling the time spent in skipping data, not just gaps.
* Add stream method `statistics()` to retrieve input statistics. (Robin Sommer, Corelight)
This returns a struct of the following type, reflecting the input
seen so far:
```
type StreamStatistics = struct {
num_data_bytes: uint64; ## number of data bytes processed
num_data_chunks: uint64; ## number of data chunks processed, excluding empty chunks
num_gap_bytes: uint64; ## number of gap bytes processed
num_gap_chunks: uint64; ## number of gap chunks processed, excluding empty chunks
};
```
* Extend HILTI's stream implementation to track data statistics. (Robin Sommer, Corelight)
* Add unit method `stream()` to access current input stream. (Robin Sommer, Corelight)
We implement this by storing the current input stream inside the
internal parse struct as a new field `__stream`. Using `stream()` then
turns into an access to that field (i.e, `*(self.__stream)`). If that
field never gets accessed, the optimizer will remove it, so there's no
overhead if nobody ever uses `stream()` with a unit.
1.11.0-dev.219 | 2024-06-20 10:32:09 +0200
* Fix Spicy's support for `network` type. `network` was not
recognized by the scanner. (Robin Sommer, Corelight)
1.11.0-dev.217 | 2024-06-19 09:29:16 +0200
* Fix lints reported by clang-tidy-18 (Benjamin Bannier, Corelight)
1.11.0-dev.202 | 2024-06-18 12:31:36 +0200
* GH-1763: Only allow creation of `const` variables from literals. (Benjamin Bannier, Corelight)
Since it is hard to enforce that `const` variables are initialized in
correct order (order of decls usually is not significant in Spicy/HILTI
and works e.g., for `global`s, but this is not the case for `const`s due
to codegen), this patch disallows creating `const` values from anything
but literals. This completely removes the ordering issue.
1.11.0-dev.200 | 2024-06-14 10:56:04 +0200
* GH-1759: Fix `if`-condition with `switch` parsing. (Robin Sommer, Corelight)
The parser generator was ignoring `if` conditions attached to `switch`
constructs. While we actually had a test for this already, turns out
we had recorded a broken baseline. Plus, we were testing only one
variant of `switch` (expression-based, not look-ahead-based). This
implements and tests both variants now.
* Fix clang-tidy. (Robin Sommer, Corelight)
1.11.0-dev.197 | 2024-06-13 12:43:33 +0200
* GH-1750: Add `to_real` method to `bytes`. (Robin Sommer, Corelight)
This interprets the data as representing an ASCII-encoded floating
point number and converts that into a ``real``. The data can be in
either decimal or hexadecimal format. If it cannot be parsed as
either, throws an `InvalidValue` exception.
* GH-1608: Add `get_optional` method to maps. (Robin Sommer, Corelight)
This returns an optional either containing the map's element for the
given key if that entry exists, or an unset optional if it does not.
* Update some debug baselines. (Robin Sommer, Corelight)
1.11.0-dev.193 | 2024-06-13 12:10:22 +0200
* GH-1760: Fix generated code for huge `const` collections. (Benjamin Bannier, Corelight)
1.11.0-dev.191 | 2024-06-13 09:59:22 +0200
* GH-1598: Enforce that the argument `new` is either a type or a
ctor. (Robin Sommer, Corelight)
So far we allowed some more generic expressions as well, but it's hard
for the parser to support arbitrary expressions here due to parsing
ambiguities, leaving things inconsistent. So we now limit it to what
was pretty much the intent originally anyways.
Note that the error message for #1598 stays the same: it's not great,
but seems good enough. However, we now actually disallow the
workaround shown in the ticket as well for consistency. The new
work-around is shown in the changes to `hilti.codegen.type-info`.
* GH-90/GH-1733: Add `result` and `spicy::Error` types to Spicy to
facilitate error handling. (Robin Sommer, Corelight)
The `result` and `error` types were already implemented internally
HILTI-side, but not yet available to Spicy users. This exposes them to
Spicy as well. To avoid name clashes with existing code, we don't
introduce `error` as a new type keyword, but instead make it available
as a library type ``spicy::Error``.
Typical usage is something like this:
```
function foo() : result<int64> {
...
if ( everything_is_ok )
return 42;
else
return error"Something went wrong.";
}
if ( local x = foo() )
print "result: %d " % *x;
else
print "error: %s " % x.error();
```
The documentation has more specifics.
* Support `result<void>` to HILTI. (Robin Sommer, Corelight)
This allows to capture errors even if there's no actual result
otherwise. Example (HILTI syntax):
```
function result<void> x(bool b) {
if ( b )
return Null; # coerces to a successful result<void>
else
return error("trouble...");
}
assert x(True);
assert x(False).error() == error("trouble...");
```
* Add `==`/`!=` operators for HILTI `error` instances. (Robin
Sommer, Corelight)
1.11.0-dev.185 | 2024-06-11 18:47:19 +0200
* Bump centos-stream in CI. (Benjamin Bannier, Corelight)
1.11.0-dev.183 | 2024-06-07 10:35:53 +0200
* GH-1745: Fix C++ initialization of global constants through global functions. (Robin Sommer, Corelight)
The changes ordering of the emitted global declarations so that
functions now come first, allowing them be used inside subsequent
constant initializations.
1.11.0-dev.181 | 2024-06-04 10:18:25 +0200
* Clean up includes. (Benjamin Bannier, Corelight)
* Fix Flex linker error when building as part of Zeek. (Benjamin Bannier, Corelight)
When building Spicy as part of Zeek against the Homebrew flex-2.6.4 I
saw linker errors after f52325693aad8bd7931d51c4658027a8b7d7adae,
```
ld: Undefined symbols:
HiltiFlexLexer::LexerInput(char*, unsigned long), referenced from:
vtable for hilti::detail::parser::Scanner in driver.cc.o
HiltiFlexLexer::LexerOutput(char const*, unsigned long), referenced from:
vtable for hilti::detail::parser::Scanner in driver.cc.o
```
It is still not clear to me how this error comes about, but the change in
this patch seems to address the issue.
1.11.0-dev.178 | 2024-06-03 09:50:53 +0200
* Fix a typo in packing.rst (Tanner Kvarfordt)
Fix a typo in packing.rst where the template argument to unpack was (u)int
instead of real.
* Expand guidelines on improving compilation performance. [skip CI] (Benjamin Bannier, Corelight)
* Fix documented type mapping for integers. [skip CI] (Benjamin Bannier, Corelight)
1.11.0-dev.172 | 2024-05-17 12:51:34 +0200
* GH-1742: Unroll ctrs of big containers. (Benjamin Bannier, Corelight)
When generating C++ code for container ctrs we previously would directly
invoke the respective C++ ctrs taking an initializer list. For very big
initializer lists this causes very bad C++ compiler performance, e.g.,
compiling code constructing a vector with 10,000 elements could take
minutes.
With this patch we unroll such ctrs calls by calling a dedicated
initialization function. For huge containers this causes creating of big
functions instead of big initializer lists, but compiling functions
seems to behave more predictively.
Closes #1742.
1.11.0-dev.170 | 2024-05-17 12:51:06 +0200
* GH-1743: Use a checked cast for `map`'s `in` operator. (Benjamin Bannier, Corelight)
1.11.0-dev.168 | 2024-05-15 10:19:54 +0200
* Fix behavior for unset accept and decline hooks. (Benjamin Bannier, Corelight)
We did not properly initialize pointer values for accept and decline
hooks. In downstream code they then appeared to be set when they were in
fact not.
With this patch we initialize them with proper defaults.
1.11.0-dev.166 | 2024-05-14 11:54:16 +0200
* Docs: Add new section with guidelines and best practices. (Robin Sommer, Corelight)
This focuses on performance for now, but may be extended with other
areas alter.
Much of the content was contributed by Corelight Labs.
* Docs: Update Custom Extensions section. (Robin Sommer, Corelight)
The usage of `-P` wasn't up to date.
* Docs: Update feedback section. (Robin Sommer, Corelight)
1.11.0-dev.162 | 2024-05-13 14:42:08 +0200
* Update types.rst (Smoot)
1.11.0-dev.160 | 2024-05-13 11:45:04 +0200
* GH-1657: Update Spicy runtime driver to use new stream features for improved performance. (Robin Sommer, Corelight)
This does two things:
- When adding data to a stream, we now do that without copying
anything initially. For block input (e.g., UDP) that's always fine
because the parser will never suspend before it's fully done
parsing; hence we can safely delete it once the parser returns. For
stream input (e.g., TCP), we make the stream own its data later
if (and only if) the parser suspends.
- For block input (e.g., UDP) we now keep reusing the same stream for
subsequent blocks, instead of creating a new one each time. This
allows the stream to reuse an allocated chunk that it may have still
cached internally.
The result of this, plus the new chunk caching introduced earlier, is
that for a UDP flow, we never need to allocate more than one chunk,
and never need to copy any data; and for TCP it's the same as long as
parsing consumes all data before suspending (which should be a common
case), plus, when we allocate new storage we only copy data that didn't
get trimmed immediately anyways.
* Give stream a method to reset it into freshly initialized state. (Robin Sommer, Corelight)
This does not clear the internal chunk cache.
* Cache previously trimmed chunks inside stream for reuse. (Robin Sommer, Corelight)
A chain now retains one previously used but no longer needed chunk for
reuse, so that we can avoid constant cycles of creating/destructing
chunks (and their payload memory) in the common case of a parser
consuming full chunks without yielding. The caching is also geared
towards owning/non-owning semantics staying consistent across
subsequent append operations.
* Extend stream API to allow for chunks that don't own their data. (Robin Sommer, Corelight)
By default, we still copy data when creating chunks but we add a
parallel API that just stores pointers, assuming the data will stay
around as long as needed. If the stream owner cannot guarantee that,
they may at any point convert all not-owned data into owned data
through a corresponding `makeOwning()` stream method. To make that
method efficient even with long chains of chunks, we internally
maintain an invariant that only the last chunk of chain can be
non-owning: whenever we add a new chunk to a chain, we ensure that the
previous tail become owning at that point. In other words, we amortize
the work across all newly added chunks.
* Remove `std::functional` from `DeferredExpression`. (Robin Sommer)
* Revert "Remove support for deferred expressions." (Robin Sommer)
Turns out this is actually still being used by the Zeek
integration.
1.11.0-dev.151 | 2024-05-10 16:35:50 +0200
* Remove unused include headers. (Robin Sommer, Corelight)
* Remove all usage of `std::function` from toolchain. (Robin Sommer, Corelight)
* Remove usage of `std::function` from parser and sink runtime representations. (Robin Sommer, Corelight)
We switch to raw function pointers, which is easy enough.
* Remove use of `std::function` from `spicy::rt::Configuration`. (Robin Sommer, Corelight)
Callbacks are now classic function pointers.
* Remove unused functional header. (Robin Sommer, Corelight)
* Remove use of `std::function` from runtime vector class. (Robin Sommer, Corelight)
* Remove support for deferred expressions. (Robin Sommer, Corelight)
These weren't used anymore anywhere so we can remove the corresponding
code from toolchain and runtime.
1.11.0-dev.142 | 2024-05-06 15:14:40 +0200
* GH-1664: Fix `&convert` typing issue with bit ranges. (Robin Sommer, Corelight)
Turns out #1664 was only indirectly related to the `&convert` itself;
the real issue was that we couldn't assign one bitfield struct to
another if their field types didn't match exactly, even in cases where
at the C++ level there was no meaningful difference. In this case we
ended up with a field that had a C++ type `rt::Bool` in one type and
`bool` in another, leading to errors when assigning the latter to the
former. We now allow to creating instances of the former from the
latter through standard C++ type conversions on a per field basis.
* Suppress new `clang-tidy` warnings. (Robin Sommer, Corelight)
* Fix a Spicy scoping issue across imports. (Robin Sommer)
We could get a bogus "unknown ID" error for default arguments of
functions defined in an imported module if that default argument was
itself referring to an identifier inside yet another imported module.
The test case shows the exact situation that was broken.
1.11.0-dev.137 | 2024-04-25 13:38:42 +0200
* Remove Spicy parser support for unsupported `&priority` attribute. (Benjamin Bannier, Corelight)
* Make spelling of hook `priority` consistent across Spicy and HILTI. (Benjamin Bannier, Corelight)
We were naming the priority attribute differently in Spicy (`priority`)
and HILTI (`&priority`). While e.g., a Spicy `Hook` could correct
extract its priority, this still could have lead to potential issues if
we were attempting to access the priority of a Spicy hook from HILTI as
we do not perform any adjustment of this attribute when lowering to
HILTI. This distinction also made it hard to generate intended code from
the outside using our API (e.g., from Zeek) since one needed to be aware
at which level the attribute was injected (Spicy or HILTI).
With this patch we internally translate a Spicy `priority` attribute to
`&attribute` syntax.
1.11.0-dev.134 | 2024-04-25 13:36:59 +0200
* Fix incremental skipping. (Benjamin Bannier, Corelight)
We previously would incorrectly compute the amount of data to skip which
could have potentially lead to the parser consuming more data than
available. With this patch we correctly use the actually consumed amount
to compute what the trim from the input.
* GH-1724: Fix skipping in size-constrained units. (Benjamin Bannier, Corelight)
We previously could skip too much data if `skip` was used in a unit with
a global `&size`. This was due to the machinery moving the input forward
believing that `skip` units were that really field productions. While
this is true in a sense in this particular case it still lead to
incorrect behavior, in particular as far is input handling is concerned
`skip` productions largely behave like any other unit.
1.11.0-dev.131 | 2024-04-25 09:27:48 +0200
* Fix potential internal error in port string conversion. (Robin Sommer)
* GH-1284: GH-1693: Promote use of `-x` over `-c`, clean up `-P`. (Robin Sommer)
This includes these pieces:
- `-P` now requires a prefix argument that set's the C++ namespace, so
that generated prototypes match that of `-x`. Like with `-x`, the
prefix may be empty (`-P ""`) to get back to the old `hlt::`.
- For both `-P` and `-x` the prefix now must be a valid C++ identifier,
because we use it as such.
- `spicyc` usage message now refers to `-c` and `-l` as for debugging
use.
- Update documentation on host applications, switching to using `-x`
instead of `-{cl}`.
- Scanned for other use of `-c` and `-l` in the docs as well, the
remaining ones seem fine.
* Remove generated, in-code linker JSON meta data. (Robin Sommer)
This was originally to allow for compiling multiple Spicy modules
separately, with the meta data providing what's the necessary to add
any cross-module functionality. However, we've moved away from that
approach and now already require the compiler to always see all code,
so this is no longer needed/possible.
1.11.0-dev.125 | 2024-04-18 15:53:48 +0200
* GH-1501: Improve some error messages for runtime parse errors. (Robin Sommer, Corelight)
* GH-1586: Make skip productions behave like the production they are wrapping. (Robin Sommer, Corelight)
* GH-1719: Fix `new` passing a unit reference to an `inout` unit parameter. (Robin Sommer, Corelight)
One might debate whether this is something we should allow at all but
we do permit it elsewhere when passing unit parameters, and changing
that would probably break code, so for consistency this allows it for
`new` as well.
Internally, there was a more general inconsistency introduced by
automatic derefs of Spicy-level strong references. While we do need
that deref to happen for operator resolution, we now remove it after
generation of HILTI code so that HILTI-level resolution can then work
as expected on the value references introduced for units. I wouldn't
be surprised if the prior behavior was causing more trouble than just
#1719 and we just hadn't run into yet it.
* Mark automatic derefs inside the operator's AST node. (Robin Sommer, Corelight)
This allows to differentiate between explicit `deref` operations part
of the source code and implicit `deref` operations inserted by the
coercer. A subsequent commit will leverage this information, but I've
been meaning to do this anyways because it could be hard to track
where a particular `deref` was coming from.
* Unify code generation for `new`. (Robin Sommer, Corelight)
There's no functional change (afaict) but for consistency of
implementation and results, `new` should go through
`compileCallArguments()`.
* GH-1655: Reject joint usage of filters and look-ahead. (Robin Sommer, Corelight)
We cannot retrieve look-ahead information from units to which a filter
is connected, because parsing for outer and inner layers will be
operating on separate streams. While we could pass the lahead token
number upwards, we would also need the stream position for the end of
the lahead symbol, but we cannot tie an iterator on the filtered
stream back to a position on the original stream. So we now reject
this.
This shouldn't cause any actual backwards-incompatibility because this
wasn't working in the first place: it would reliably abort with an
invalid stream iterator exception during parsing.
* Add test confirming that `&parse-{at,from}` don't interfere with outer look-ahead parsing. (Robin Sommer, Corelight)
This makes sure fields with these attributes are ignored for
look-ahead computation of the unit containing them. We actually have a
related test for `&parse-from` already, but can't hurt to have it
covered further here.
1.11.0-dev.114 | 2024-04-16 09:38:38 +0200
* Bump baselines. (Benjamin Bannier, Corelight)
1.11.0-dev.112 | 2024-04-15 17:22:55 +0200
* Update NEWS with a list of incompatibilities compared to previous version. (Robin Sommer)
* Pretty print reference types in Spicy output. (Robin Sommer)
We now render `strong_ref<T>` as `T&`, as one would expect on
the Spicy side.
* Do not perform automatic deref on RHS of an assignment. (Robin Sommer)
This used to be accepted but had not the intended effect at runtime:
```
function f(s: string&) {
*s = new "xxx";
}
```
* Fix line numbers. (Robin Sommer)
Line numbers could be off in the presence of comments including '#'
characters.
* Fix auto-deref of LHS references. (Robin Sommer)
We were checking the constness of the reference, not of the wrapped
value.
* Improve error message. (Robin Sommer)
1.11.0-dev.105 | 2024-04-10 13:24:03 +0200
* Add missing file to git. (Robin Sommer, Corelight)
* Document generic operators. (Robin Sommer, Corelight)
* Refactor output logic in `spicy-doc-to-rst`. (Robin Sommer, Corelight)
* GH-1711: Fix forwarding of a reference unit parameter to a non-reference parameter. (Robin Sommer, Corelight)
* GH-1599: Fix integer increment/decrement operators require mutable arguments. (Robin Sommer, Corelight)
* Add tests checking sinks as unit parameters. (Robin Sommer, Corelight)
* GH-1710: Improve implementation of sink type. (Robin Sommer, Corelight)
So far we lowered Spicy's `sink` type into a `strong_ref` a HILTI
codegen time, meaning that a unit's `sink` field would have type
`sink` at the Spicy-level and then later `strong_ref<sink>` inside
HILTI (actually: `strong_ref<spicy_rt::Sink>`). This approach led some
inconsistencies because of the mismatch between the two levels, and it
also made the implementation more complex than necessary. We now let
`sink` fields simply have type `sink&` (i.e., `strong_ref<sink>`) in
Spicy; everything else then falls in place more easily. From a user
perspective, the change should remain largely invisible.
* Fix internal assertion potentially triggering erroneously during retrieval of parent node. (Robin Sommer, Corelight)
* GH-1618: Fix and clarify usage of references in Spicy. (Robin Sommer, Corelight)
References weren't fully consistent in their properties and
implementation, and they weren't documented either. This commit cleans
that up and adds documentation.
Changes:
- Constness 1: For a type `T&`, which internally is `strong_ref<T>`,
we now consistently make the inner `T` a mutable LHS type. That
seems most natural and useful from a user perspective. In
particular, this allows mutating objects passed into functions as
parameters without declaring them `inout`: (`foo (x: bytes&)`). This
is important because declaring the parameter `inout` makes the
reference itself mutable (not the contained object), which isn't
what one wants. Plus, the latter doesn't work for unit parameters
anyways. Accordingly, we also adapt our built-in operators to
use`T&` instead of `inout T&`.
- Constness 2: For a type `T&`, we now make the outer reference type
constant and non-mutable. This is mostly for consistency, it doesn't
really change anything as we didn't havemutating operations on
references anyways.
- Feature: We add `new BASIC_TYPE` as syntax for creating default
initialized values of basic types. So far we only had `new
NAMED_TYPE` and `new VALUE`.
Technical note to the reviewer: the changes to the `recreateAs*`
methods are necessary to avoid complex types being copied into the
newly created type, which would lead to name resolution problems. This
is tested through existing tests which fail otherwise
(`spicy.types.sink.filter-it.spicy`,
`spicy.rt.base64-filter-eod.spicy`). (Well, `recreateAsLHS()` is
tested that way, but the others would have the same issue.) This is a
good change anyways, because it keeps the AST smaller.
* GH-1515: Catch unsupported types for unit inout parameters. (Robin Sommer, Corelight)
We support only types that are (internally) passed around as
references.
* GH-1583: Disallow coercion when passing arguments to `inout` parameters. (Robin Sommer, Corelight)
1.11.0-dev.92 | 2024-04-10 09:40:32 +0200
* Update release version in documentation. (Robin Sommer, Corelight)
This hadn't been updated, leading to outdated links.
* Do not require all AST nodes to be destroyed before we begin compilation. (Robin Sommer)
If a 3rdparty (like Zeek) still retains a pointer to a `Node`, the
check would trigger, making usage awkward. Instead we now just
check at context destruction time that we no longer have any live
nodes.
1.11.0-dev.88 | 2024-04-09 09:33:03 +0200
* Bump 3rdparty/utf8proc from `1fe43f5` to `894e810` (dependabot[bot])
1.11.0-dev.86 | 2024-04-04 15:51:17 +0200
* Overhaul AST node memory management. (Robin Sommer)
We switch back to reference counting nodes, but through a custom
scheme that allows us to continue passing around raw pointers most
of the time.
* Remove unused class. (Robin Sommer)
* Reduce memory usage for operators with complex argument types. (Robin Sommer)
We now use external types for to store operands of name types, which
reduces the number of AST nodes created for operators substantially.
* Remove meta/location information from AST IDs. (Robin Sommer)
Turns out these aren't used anywhere.
* Use less memory for storing context IDs. (Robin Sommer)
32 bits should still be plenty. Because these are stored in many
nodes, the change is noticeable in overall memory consumption.
* Compute an ID's internal views on demand. (Robin Sommer)
We had previously optimized IDs by pre-computing some information for
quick access. However, for most IDs that information isn't actually
accessed at all, so we now compute it only on demand the first time
it's needed. This saves both CPU and memory.
* Move inherit-scope information into virtual method. (Robin Sommer)
This saves space, and is also more aligned with how other information is
managed as well.
* Change storage for AstContext. (Robin Sommer)
No need for `shared_ptr`, now using a `unique_ptr`.
* Do not auto-allocate storage for errors with each AST node. (Robin Sommer)
We now create the vector for error messages only when needed.
* Reuse `Meta` instances across nodes. (Robin Sommer)
We now store each `Meta` value only once globally. Saves about 15%
memory.
1.11.0-dev.75 | 2024-04-03 11:55:16 +0200
* Remove a few left-over unused variables. (Benjamin Bannier, Corelight)
* Fix a few instances where codegen was non-deterministic. (Benjamin Bannier, Corelight)
1.11.0-dev.72 | 2024-04-03 09:35:23 +0200
* Fix repeated evaluations of `&parse-at` expression. (Robin Sommer)
* GH-1316: GH-1635: Provide better error messages for some cases of unknown unit IDs. (Robin Sommer, Corelight)
* GH-1493: Support/fix public type aliases to units. (Robin Sommer, Corelight)
An alias like `public type Unit1 = Unit2` used to lead to C++-side
compiler errors, which this fixes. We also fully support this now by
making both `Unit1` and `Unit2` available for parsing to host
applications. Internally, the `Unit1` parser is just a small facade
pointing to the parsing functions for `Unit2`.
* GH-1661: Deprecate usage of `&convert` with `&chunked`. (Robin Sommer, Corelight)
Per discussion in #1661, this combination can lead to confusion and
can be worked around if really needed.
1.11.0-dev.64 | 2024-04-02 17:44:13 +0200
* Fix GCC false positive around `strncpy` use. (Benjamin Bannier, Corelight)
1.11.0-dev.62 | 2024-03-25 10:39:42 +0100
* Reimplement `IDBase` for better performance. (Robin Sommer)
Our codegen phase had quite some overhead due to repeated ID
operations recomputing the same information over and over again (e.g.,
subpaths and namespaces). This reimplements the class to compute
everything once upfront, cutting codegen time into half.
* Add more tests for `IDBase`. (Robin Sommer)
This includes a tiny semantic change for `IDBase::length()`: it now
returns zero for an empty ID, which seems more consistent. (This
doesn't seem to have an impact anywhere, all tests pass.)
1.11.0-dev.59 | 2024-03-21 12:35:06 +0100
* Rework order of C++ codegen. (Robin Sommer)
We used to potentially codegen modules multiple times. Now we cache a
module's generated C++ code inside the AST node the first time we
create it. From there, we can then easily reuse it later, in
particular when needing to import its declarations into another
module. Internally, we switch storage for `cxx::Unit` to shared
pointers, and tweak the debug logging a bit for better readability in
this new model.
* Cleanup: Remove flag to compile implementation from higher-level codegen method. (Robin Sommer)
Pushing down the condition that we used to pass in, to lower-level
code.
* Fix duplicates in module dependency tracking. (Robin Sommer)
We now use a set instead of a vector to unique dependencies
automatically.
* Fix file name case for fuzzer builds. (Benjamin Bannier, Corelight)
1.11.0-dev.54 | 2024-03-18 09:56:58 +0100
* Refresh CI platforms. (Benjamin Bannier, Corelight)
1.11.0-dev.50 | 2024-03-16 08:55:11 +0100
* Rework memory management for AST nodes. (Robin Sommer, Corelight)
We switch memory management of AST nodes to an arena/bump allocator
that releases them as a whole once the AST gets destroyed, not
individually/continiously through their own life-time scoping. This
then allows us to also switch them from `shared_ptr` to raw pointers
throughout the AST code. The result is a compiler speed up of about
20% for some complex analyzer.
* Centralize the `vector` type we use for storing AST nodes. (Robin Sommer, Corelight)
No functional change, this just unifies the various vectors of
`Node`-derived classes so that there's a central place where to define
the underlying vector type. For now this is just for easier
maintenance. In the future we could experiment with different vector
or allocator implementations.
* Make node tags `constexpr`. (Robin Sommer, Corelight)
1.11.0-dev.45 | 2024-03-15 15:11:26 +0100
* Fix broken f-string. (Benjamin Bannier, Corelight)
* Remove Moneterey Homebrew CI tasks. (Benjamin Bannier, Corelight)
* Add Cirrus tasks running non-Homebrew macos builds and tests. (Benjamin Bannier, Corelight)
1.11.0-dev.41 | 2024-03-15 12:47:42 +0100
* Fix fuzzer builds for reworked AST. (Benjamin Bannier, Corelight)
1.11.0-dev.39 | 2024-03-13 09:07:42 +0100
* Bump softprops/action-gh-release from 1 to 2 (dependabot[bot])
* Bump typos pre-commit hook. (Benjamin Bannier, Corelight)
* Bump clang-format. (Benjamin Bannier, Corelight)
* Modernize Python scripts with `pyupgrade`. (Benjamin Bannier, Corelight)
* Reformat Python with ruff-format. (Benjamin Bannier, Corelight)
* Fix generation of doc example code. (Benjamin Bannier, Corelight)
* Fix Python lints diagnosed by ruff. (Benjamin Bannier, Corelight)
1.11.0-dev.30 | 2024-03-12 12:02:43 +0100
* Get rid of more dynamic casts. (Robin Sommer, Corelight)
The main use of dynamic casts that's left now (other than for
debugging code) is inside the grammar's production hierarchy. That
logic isn't standing out during profiling, so seems fine to leave for
now.
* Introduce custom RTTI system for casting safely between node
types. (Robin Sommer, Corelight)
Our custom system is faster than C++'s `dynamic_cast<>`, resulting in
a noticeable speed-up for larger parsers.
We assign a unique integer tag to each `Node`-derived class. Each
class' constructors pass an array through to the top-level `Node`
class that contains a series of these integers describing the
inheritance path from the derived class back up to `Node`. The `Node`
class stores this path for fast type checks. In addition, each
`Node`-derived class `T` gets a couple constants as members: (1) a
copy of its own tag (`T::NodeTag`), and (2) a level indicating its
distance from `Node` in the inheritance tree (`T::NodeLevel`). The
level is used as index into an instance's tag array when performing
type checks. The result is that we can do `isA<T>` operation with just
a single comparison between an array element and a constant value.
This system is optimized for our very simple Node hierarchy: not very
deep (max. 4 entries in the tag array), single-inheritance only,
derived class are always the same distance from `Node`, and
all `Node`-derived classes are known upfront.
1.11.0-dev.27 | 2024-03-06 17:06:09 +0100
* Speed up `util::toIdentifier`. (Robin Sommer)
Turns out this is called a lot. This changes baselines due to slight
differences in generated IDs (changes are good because they better avoid
potential conflicts).
This also removes the option to ensure non-keyword IDs: I don't
think we need that because we check for that when we generate C++
code.
* GH-1675: Extend runtime profiling to measure parser input volume. (Robin Sommer, Corelight)
With `--enable-profiling` the output for Spicy units/fields now
includes a new `volume` column, like this:
```
#name count time avg-% total-% volume
[...]
spicy/unit/test::A 1 285500 43.96 43.96 8
spicy/unit/test::A/__gap__ 4 3167 0.12 0.49 0
spicy/unit/test::A/__synchronize__ 1 35500 5.47 5.47 4
spicy/unit/test::A::a 1 74833 11.52 11.52 -
spicy/unit/test::A::b 1 15333 2.36 2.36 1
spicy/unit/test::A::c 1 19125 2.94 2.94 1
spicy/unit/test::A::d 1 7583 1.17 1.17 1
spicy/unit/test::A::e 1 8042 1.24 1.24 1
```
Three different things here:
- The `volume` column for `spicy/unit/TYPE` and
`spicy/unit/TYPE::FIELD` augments the already existing timing
measurement and reports the total, aggregate number of bytes that
this unit/field got to parse over the course of the processing.
- For units going into synchronization mode, there are now additional
rows `spicy/unit/TYPE/__synchronize__` that report both CPU time and
volume spent in synchronization while processing that unit.
- For units encountering input gaps during synchronization, there are
now additional rows 'spicy/unit/TYPE/__gap__` that report total
aggregate gap size encountered while processing the unit.
All the volume measurements are taken as differences of two offsets
inside the input stream. For normal unit/field parsing, we subtract
the final offset after parsing an instance from the initial offset
where its parsing started.[1] For synchronization, it's the offset
where synchronization stopped successfully minus where it started.[2]
For gaps, it's the offset where we continued after the gap minus where
the gap started.[3] All these differences are then added up for each
row over the course of total input stream processing.
Note that volume isn't counted if parsing for some reason never
reaches the point where the end measurement would be taken (e.g., a
parser error prevents it from being reached; in the output above
that's the case for `spicy/unit/test::A::a`).
Closes #1675.
[1] This *includes any ranges that the unit spent in synchronization
mode trying to recover from parse errors.
[2] This does *not* include any gaps encountered because they don't
affect stream offsets.
[3] Little glitch: these values can currently by off by one due to some
internal ambiguity.
* Move inferring of a unit's context type into the resolver. (Robin Sommer)
No functional change here, just cleanup: this is where the logic
belongs.
* Infer constness of `<unit>.context()` from that of `<unit>`. (Robin Sommer)
Closes https://github.com/corelight/zeek-spicy-openvpn/issues/11.
* Ensure constness of result for const map/vector index operator. (Robin Sommer)
Not sure this actually changes anything, but it ensures they are correct.
* Simplify constness check for struct field assignments. (Robin Sommer)
Same treatment as for unit fields.
* Bump 3rdparty/filesystem from `2fc4b46` to `42ea4fc` (dependabot[bot])
1.11.0-dev.15 | 2024-03-05 10:43:27 +0100
* Bump 3rdparty/filesystem from `2fc4b46` to `42ea4fc` (dependabot[bot])
1.11.0-dev.11 | 2024-03-04 10:07:25 +0100
* New AST architecture. (Robin Sommer)
This is a large revamp of compiler internals, cleaning up and speeding
up lots of the AST pipeline. From a user perspective, nothing changes,
except that the new compiler is a tiny bit more strict: turns out that
in rare cases the old compiler ended up accepting some ill-formed
Spicy code that is now (rightfully) rejected. Specifically, two
instances of this are known where existing Spicy code may need
tweaking now:
- Identifiers from the (internal) `hilti::` namespace are no longer
accessible. Usually you can just scope them with `spicy::` instead,
and it'll work.
- The old compiler didn't always enforce constness as it should have.
In particular, function parameters could end up being mutable even
when they weren't declared as `inout`. Now `inout` is required for
supporting any mutable operations on a parameter, so make sure to
add it where needed.
1.11.0-dev.9 | 2024-03-01 18:10:49 +0100
* Simplify `while` loops over constant conditions. (Benjamin Bannier, Corelight)
While loops with constant conditions are unlikely to occur in code
generated by us, but could appear in user code. Cleaning them up is
simple and cheap, so this patch implements a pass which simplifies them.
1.11.0-dev.7 | 2024-02-27 17:44:22 +0100
* GH-1624: Enable optimizations when running `spicy-build`. (Benjamin Bannier, Corelight)
We previously would emit single C++ source files from `spicy-build`.
This made it impossible to enable optimizations since we could not know
whether individual code was used. Since optional features are by default
enabled and only disabled through an optimizer pass this meant that
`spicy-build` emitted code which performed a lot of work not done
usually.
This patch reworks `spicy-build` to use `spicyc`'s `-x` instead which
has a global view and can run optimizations, and emits all source files
into a prefix.
Closes #1624.
Closes #1625.
Closes #1622.
* Fix some shellcheck warnings in `spicy-build`. (Benjamin Bannier, Corelight)
* Fix handling of `spicyc -x` when passed a filename. (Benjamin Bannier, Corelight)
For the `-x` flag users can pass either the name of a directory or a
file. If passed a filename the file name is incorporated into the
identifier generated namespace identifier of e.g., the form
`__hlt_FILENAME` in the emitted C++ files. Previously when passing a
directory we would still emit a `_` separator, but since there was no
filename nothing after it so we generated namespace identifiers like
`__hlt_`.
With this patch we now only emit the separator if passed a filename.
* Do not optimize out public functions. (Benjamin Bannier, Corelight)
Optimizing out public functions breaks the assumption we have elsewhere
that something marked `public` is part of the public API and would not
get optimized out. This is e.g., the behavior we have for units.
1.11.0-dev.2 | 2024-02-26 12:16:17 +0100
* Fix stray Python escape sequence. (Benjamin Bannier, Corelight)
1.10.0-dev.149 | 2024-02-22 09:48:01 +0100
* GH-1585: Put closing of unit sinks behind feature guard. (Benjamin Bannier, Corelight)
This code gets emitted, regardless of whether a sink was actually
connected or not. Put it behind a feature guard so it does not enable
the feature on its own.
Closes #1585.
1.10.0-dev.147 | 2024-02-21 10:30:22 +0100
* GH-1667: Always advance input before attempting resynchronization. (Benjamin Bannier, Corelight)
When we enter resynchronization after hitting a parse error we
previously would have left the input alone, even though we know it fails
to parse. We then relied fully on resynchronization to advance the
input.
While this just pushed work downstream when synchronizing on literals,
it could cause us loosing input if synchronizing on regular expressions
if we happened to fail parsing due to a gap which is now at the front of
the input (parse errors from gaps are the most likely resynchronization
scenario when parsing genuine traffic); in this case the regular
expression would synchronize at the second byte after the input and we
would synchronize only at a later position.
With this patch we always forcibly advance the input to the next non-gap
position. This has no effect for synchronization on literals, but allows
it to happen earlier for regular expressions.
Closes #1667.
* Refactor test `spicy.types.unit.synchronize-on-gap`. (Benjamin Bannier, Corelight)
This refactoring cleans up how we feed gaps into the parser to testing
with more inputs simpler.
1.10.0-dev.144 | 2024-02-14 15:55:35 +0100
* GH-1652: Fix filters consuming too much data. (Benjamin Bannier, Corelight)
We would previously assume that a filter would consume all available
data. This only holds if the filter is attached to a top-level unit, but
in general not if some sub-unit uses a filter. With this patch we
explicitly compute how much data is consumed.
Closes #1652.
1.10.0-dev.142 | 2024-02-08 17:00:53 +0100
* GH-1668: Fix incorrect data consumption for `&max-size`. (Benjamin Bannier, Corelight)
We would previously handle `&size` and `&max-size` almost identical
with the only difference that `&max-size` sets up a slightly larger view
to accommodate a sentinel. In particular, we also used identical code to
set up the position where parsing should resume after such a field.
This was incorrect as it is in general impossible to tell where parsing
continues after a field with `&max-size` since it does not signify a
fixed view like `&size`. In this patch we now compute the next position
for a `&max-size` field by inspecting the limited view to detect how
much data was extracted.
Closes #1668.
1.10.0-dev.140 | 2024-02-08 13:22:52 +0100
* GH-1522: Drop overzealous validator. (Benjamin Bannier, Corelight)
This validator was intended to reject incorrect parsing of vectors but instead
ending up rejecting all vector parsing if the vector elements itself produced
vectors. Since this code has no test and it seems to have no clear purpose this
patch drops this validation.
Closes #1522.
1.10.0-dev.138 | 2024-02-08 13:20:31 +0100
* Remove now unused testing `random.seed`. (Benjamin Bannier, Corelight)
* GH-1659: Lift requirement that `bytes` forwarded from filter be mutable. (Benjamin Bannier, Corelight)
Since we did not declare the `bytes` argument to `unit::forward`
constant it was implicitly mutable so that it was impossible to e.g.,
pass literal bytes. This patch uses the originally intended type.
Closes #1659.
1.10.0-dev.135 | 2024-01-30 12:29:28 +0100
* GH-1665: Fix running of `codebase` tests. (Benjamin Bannier, Corelight)
1.10.0-dev.133 | 2024-01-29 10:59:23 +0100
* GH-1489: Deprecate &bit-order on bit ranges. (Arne Welzel, Corelight)
This does not appear to have any effect and allowing it may be
confusing to users. Deprecate it with the idea of eventual
removal.
* Add extensive bitfield test including endianness behavior. (Arne Welzel, Corelight)
* Add bitfield examples. (Arne Welzel, Corelight)
1.10.0-dev.129 | 2024-01-23 17:48:19 +0100
* Adjust end of Bison-generated locations. (Robin Sommer, Corelight)
The Bison-side end column points one beyond the element, which isn't
that nice in error messages, so we adjust them now.
* Extend location printing to include single-line ranges. (Robin Sommer, Corelight)
For a location of, e.g., "line 1, column 5 to 10", we now print
`1:5-10`, whereas we used to print it as only `1:5`, hence dropping
information.
* Fix Bison locations. (Robin Sommer, Corelight)
If a Bison rule started with an optional element, its location would
start at the end of the *previous* token if that optional element
wasn't present. We now skip token locations that are zero-sized at the
beginning of a rule.
1.10.0-dev.125 | 2024-01-23 16:27:43 +0100
* Bump 3rdparty/any from `e88b1bf` to `7c76129`
* Fix incorrect check_suite type in GH actions [skip CI]. (Benjamin Bannier, Corelight)
1.10.0-dev.122 | 2024-01-18 08:39:05 +0100
* Update documentation of `offset()`. (Benjamin Bannier, Corelight)
The originally documented caveats do not apply anymore, so remove them.
* Always store a valid begin iterator in units. (Benjamin Bannier, Corelight)
* Always pass a valid begin iterator to parse methods. (Benjamin Bannier, Corelight)
* GH-1648: Provide meaningful unit `__begin` value when parsing starts. (Benjamin Bannier, Corelight)
We previously would not provide `__begin` when starting the initial
parse. This meant that e.g., `offset()` was not usable if nothing ever
got parsed.
With this patch we provide a meaningful value now.
Closes #1648.
1.10.0-dev.117 | 2024-01-17 16:43:22 +0100
* GH-1640: Implement skipping for any field with known size. (Benjamin Bannier, Corelight)
This patch adds `skip` support for fields with `&size` attribute or of
builtin type with known size. If a unit has a known size and it is
specified in a `&size` attribute this also allows to skip over unit
fields.
* Add method to field to optionally return its size. (Benjamin Bannier, Corelight)
* Fix skipping of literal fields with condition. (Benjamin Bannier, Corelight)
1.9.0-111 | 2024-01-11 11:55:07 +0100
* GH-1645: Fix `&size` check. (Robin Sommer, Corelight)
The current parsing offset could legitimately end up just beyond the
`&size` amount.
1.9.0-109 | 2024-01-11 09:00:08 +0100
* GH-1634: Fix infinite loop in regular expression parsing. (Robin Sommer, Corelight)
1.9.0-107 | 2024-01-11 08:59:05 +0100
* Bump justrx to pull in bugfix. (Robin Sommer, Corelight)
* CI: Re-enable `clang17_ubuntu_debug_task`. (Robin Sommer, Corelight)
1.10.0-dev.103 | 2024-01-09 14:00:44 +0100
* Remove references to Cirrus cron task. (Benjamin Bannier, Corelight)
We do not schedule CI runs with cron anymore.
* Run CI ASAN task also for PRs. (Benjamin Bannier, Corelight)
* Skip tests around stack size on macos with ASAN. (Benjamin Bannier, Corelight)
These tests produce false positives on macos with ASAN which seem hard
to get rid of. Disable them on the smallest possible subset of setups.
* Fix ASAN false positives introduces with d332827aee7d70cdb642631d7a289751e1d8a36a. (Benjamin Bannier, Corelight)
Since the logging changes of d332827aee7d70cdb642631d7a289751e1d8a36a
ASAN reports false positives on e.g., macos-13.6.2 with its
clang-1500.0.40.1. This patch slightly reorganizes the code so these
false positiives are not triggered.
1.10.0-dev.98 | 2024-01-09 13:55:27 +0100
* Bump dependencies. (Benjamin Bannier, Corelight)
* GH-1632: Bump justrx to pull in bugfix. (Robin Sommer, Corelight)
Closes #1632.
* CI: drop freebsd-12, add freebsd-14. (Benjamin Bannier, Corelight)
* GH-1500: Add `+=` operator for `string`. (Benjamin Bannier, Corelight)
This allows appending to a `string` without having to allocate a new
string. This might perform better most of the time.
Closes #1500.
1.9.0-97 | 2024-01-09 10:13:30 +0100
* GH-1632: Bump justrx to pull in bugfix. (Robin Sommer, Corelight)
1.10.0-dev.91 | 2024-01-04 12:44:32 +0100
* Add unit tests for extracting from expanding Views. (Benjamin Bannier, Corelight)
* Add unit tests for extracting from Views with gaps. (Benjamin Bannier, Corelight)
* Add fast pass to noop unsafe stream iterator increment/decrement. (Benjamin Bannier, Corelight)
We already had this optimization for safe, but not for unsafe iterators.
* GH-1628: Avoid potentially inefficient data access in `View::extract`. (Benjamin Bannier, Corelight)
While we already had a fast path to extract data out of `View`s
consisting of a single chunk we still would use a potentially
inefficient approach when extracting from `View`s over multiple chunks.
This patch uses the same optimized handling for both cases.
Closes #1628.
1.10.0-dev.86 | 2024-01-02 16:38:08 +0100
* Suppress clang-tidy `misc-include-cleaner` lint. (Benjamin Bannier, Corelight)
This currently triggers a lot of issues. While it seems to be useful
getting us to a passing state seems to require substantial work.
* Remove outdated workaround for clang-tidy. (Benjamin Bannier, Corelight)
With newer clang-tidy versions this is not needed anymore.
* Reenable clang-tidy `readability-simplify-boolean-expr` lint. (Benjamin Bannier, Corelight)
This currently triggers no issues, and the lint seems to be generally
useful.
* Drop `;` after `#pragma`. (Benjamin Bannier, Corelight)
* Fix clang-tidy `readability-redundant-string-init` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `bugprone-exception-escape` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `misc-header-include-cycle` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `bugprone-use-after-move` lints. (Benjamin Bannier, Corelight)
* Suppress clang-tidy `bugprone-switch-missing-default-case` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `readability-avoid-unconditional-preprocessor-if` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `modernize-use-emplace` lints. (Benjamin Bannier, Corelight)
* Suppress clang-tidy `modernize-macro-to-enum` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `performance-avoid-endl` lints. (Benjamin Bannier, Corelight)
* Fix clang-tidy `modernize-type-traits` lints. (Benjamin Bannier, Corelight)
* Update clang-tidy suppression for newer clang-tidy versions. (Benjamin Bannier, Corelight)
* Remove nightly task. (Benjamin Bannier, Corelight)
This task originally tested against Zeek's `master` branch. We do not do
that anymore since some time so this task has lost its purpose.
* Remove `--rpath` flag to `ci/run-ci`. (Benjamin Bannier, Corelight)
Uses of this flag have been mirroring external setup of
`LD_LIBRARY_PATH` in `.cirrus.yml` for some time with no added benefit.
Drop it instead of investing work in keeping it consistent.
* Disable building benchmarks in CI. (Benjamin Bannier, Corelight)
On some platforms the CMake configure phase of 3rdparty/benchmark fails
with errors like
```
CMake Error at 3rdparty/benchmark/CMakeLists.txt:301 (message):
Failed to determine the source files for the regular expression backend
```
Since it is not executed in CI anyway disable it.
* Refresh LLVM version in CI Docker image. (Benjamin Bannier, Corelight)
* Remove unused Docker image args. (Benjamin Bannier, Corelight)
* Bump CI image to ubuntu-22.04. (Benjamin Bannier, Corelight)
1.10.0-dev.64 | 2024-01-02 15:50:26 +0100
* Add dependency of generated file `spicy-build` on source file. (Benjamin Bannier, Corelight)
Without this dependency we never updated the generated file.
* Remove use of `IntrusivePtr` over `const`. (Benjamin Bannier, Corelight)
It looks like using a `IntrusivePtr<const T>` interferes with default'ed
move constructors and assignment operators so that the non-move versions
are called. This can incur additional overhead.
This patch simply gets rid all `IntrusivePtr<const T>` in favor of
`IntrusivePtr<T>` which does not seem to show this issue. It might be
possible to instead adjust something e.g., `ManagedObject` so
`IntrusivePtr` does not regress when holding semantically `const`
values though.
For a big internal parser this shows runtime improvements up to 2%.
1.10.0-dev.61 | 2024-01-02 15:50:05 +0100
* Fix docs namespace for symbols from `filter` module. (Benjamin Bannier, Corelight)
We previously would document these symbols to be in `spicy` even though
they are in `filter`.
1.10.0-dev.59 | 2023-12-15 13:29:41 +0100
* Add move ctr for `stream::SafeConstIterator`. (Benjamin Bannier, Corelight)
* Use unchecked operations for `View::unsafeEnd`. (Benjamin Bannier, Corelight)
We previously would call `end()` to compute `unsafeEnd` which incurs
overheads this function is explicitly designed to avoid. We now switch
this implementation of this function to completely unsafe code avoiding
these overheads.
* Add move ctr for `stream::View`. (Benjamin Bannier, Corelight)
In C++17 move ctors are not by default generated, add them explicitly.
We also get rid of the `View` dtr since it adds nothing (e.g., the class
is declared `final`), but declaring them could e.g., make the class not
POD anymore (this is not the case here though).
1.10.0-dev.55 | 2023-12-12 18:38:24 +0100
* GH-1617: Fix handling of `%synchronize-*` attributes for units in lists. (Benjamin Bannier, Corelight)
We previously would not detect `%synchronize-at` or `%synchronize-from`
attributes if the unit was not directly in a field, i.e., we mishandled
the common case of synchronizing on a unit in a list.
With this patch we now handle these attributes, regardless of how the
unit appears.
1.10.0-dev.53 | 2023-12-12 13:05:41 +0100
* Avoid unnecessary copy when constructing `Stream` from `Bytes`. (Benjamin Bannier, Corelight)
* Back Chunk with a std::string instead of std::variant. (Arne Welzel, Corelight)
This is probably a bit funky due to the trailing \0 and a Chunk not
being a string, but hey, hilti::rt::Bytes is backed by std::string, too.
Looking at the creation of Chunk::Chunk(), there's actually a memset()
operation visible due to the variant holding some 40 bytes that are
zero initialized upon construction. std::string supports SSO, so we
can leverage even if the actual size is out of our control. The
std::get_if()s do show up very marginally for the View::size() calls.
1.10.0-dev.50 | 2023-12-11 12:08:12 +0100
* GH-1615: Optimize C++ tuple element coercions. (Benjamin Bannier, Corelight)
This patch introduces an optimization to pass C++ tuples through if they
do not need C++-side coercion. This emits better code if a tuple ctor
already has the right types. We also optimize codegen if we are coercing
from a temporary tuple in that we now capture it in a temporary and
coerce elements from that instead of repeatedly emitting the same tuple
ctor (we scale with the number of tuple elements now not its square
anymore).
1.10.0-dev.48 | 2023-12-08 15:55:12 +0100
* Use deterministic destruction of Spicy runtime in unit tests. (Benjamin Bannier, Corelight)
We previously would shut down the runtimes in units tests with
library destructors. Due to recent changes ASAN correctly flags this as
potential use-after-free errors (e.g., it might run after the
configuration global has already been destructed).
With this patch implement our own doctest main function and
deterministically tear down runtimes from that.
1.10.0-dev.46 | 2023-12-08 09:56:14 +0100
* Remove redundant forward decl. (Benjamin Bannier, Corelight)
* GH-1611: Silence ASAN warning. (Benjamin Bannier, Corelight)
1.10.0-dev.43 | 2023-12-08 09:55:13 +0100
* Allocate Vector::_control lazily. (Arne Welzel, Corelight)
* Allocate Bytes::_control lazily. (Arne Welzel, Corelight)
1.10.0-dev.40 | 2023-12-06 12:31:27 +0100
* GH-1605: Allow for unresolved types for set `in` operator. (Benjamin Bannier, Corelight)
1.10.0-dev.38 | 2023-12-06 12:28:27 +0100
* Avoid reallocating Bytes when appending from view (Arne Welzel, Corelight)
When copying a full view into a Bytes instance, avoid potential
reallocations and memcpy() by pre-allocating enough capacity in
the underlying string.
* Allocate vector of correct size right away when constructing Chunks. (Benjamin Bannier, Corelight)
* Avoid one extra std::string copy in Stream::append. (Arne Welzel, Corelight)
For a chunk of significant size, seems that would result in an extra
malloc and copy of the input data.
* Reduce safe iterator use in internal code for `Bytes`. (Benjamin Bannier, Corelight)
The safe iterator for bytes dynamically allocates which can cause
overhead. Use index-based or at least string iterators to reduce that
overhead where possible.
* Use unsafe iterators in `View::extract`. (Benjamin Bannier, Corelight)
* Avoid unneeded parameter copy in parse functions. (Benjamin Bannier, Corelight)
* Streamline `View::firstBlock`. (Benjamin Bannier, Corelight)
* Reduce allocations when connecting sinks by mime-type. (Benjamin Bannier, Corelight)
* Reduce allocations for `Sink` debug logging. (Benjamin Bannier, Corelight)
* Deprecate `builder::string`. (Benjamin Bannier, Corelight)
* Reduce allocations when creating exceptions. (Benjamin Bannier, Corelight)
* Reduce allocations in `builder::addAssert`. (Benjamin Bannier, Corelight)
* Reduce allocations in `ParserBuilder::waitForInput`. (Benjamin Bannier, Corelight)
* Reduce allocations in `Builder::startProfiler`. (Benjamin Bannier, Corelight)
* Reduce allocatons in `Builder::addDebug*` methods. (Benjamin Bannier, Corelight)
* Reduce allocations in `ParserBuilder::parseError`. (Benjamin Bannier, Corelight)
* Reducing copying when forwarding data to sinks. (Benjamin Bannier, Corelight)
* Reduce lookup overhead in indent/dedent logger functions. (Benjamin Bannier, Corelight)
* Emit C++ string literals for HILTI string literals. (Benjamin Bannier, Corelight)
When emitting literals for HILTI strings (string ctors) we would
previously explicitly force creation of `std::string`. This was almost
always an unnecessary pessimisation over emitting string literals since
even if their C++ uses expected `std::string` string literals can
convert to this type implicitly; at the same time it made it impossible
to make effective use of APIs accepting `std::string_view`.
With this patch we now emit C++ string literals for HILTI string
literals.
* Emit locations as generated strings. (Benjamin Bannier, Corelight)
* Avoid allocations when creating parsers. (Benjamin Bannier, Corelight)
* Use non-owning strings for `fmt`. (Benjamin Bannier, Corelight)
* Use non-owning strings for `printParserState`. (Benjamin Bannier, Corelight)
* GH-1591: Use non-owning strings in the logging framework. (Benjamin Bannier, Corelight)
Closes #1591.
* Add `to_string_for_print` for string literals. (Benjamin Bannier, Corelight)
* GH-1589: Reduce string allocations on hot parse path. (Benjamin Bannier, Corelight)
When waiting for input we pass down strings for a possible error message
and the triggering location. In generated code these are always
literals.
With this patch we do not take them as owning strings, but instead as
views into existing strings to minimize allocations. In the case of
error messages the created low-level exception objects already had used
string_views, so this also aligns the APIs.
Closes #1589.
* Bump pre-commit hooks (Benjamin Bannier, Corelight)
1.10.0-dev.10 | 2023-12-06 10:40:22 +0100
* Allow unsafe Vector iteration over underlying std::vector. (Arne Welzel, Corelight)
1.10.0-dev.8 | 2023-11-17 14:04:16 +0100
* doc: Fix typo and outdated info for host applications (Anthony VEREZ)
* Add placeholder section for 1.10 to NEWS. (Benjamin Bannier, Corelight)
1.9.0 | 2023-10-24 16:28:18 +0200
* Release 1.9.0.
1.9.0-dev.159 | 2023-10-24 16:26:42 +0200
* Bump 3rdparty/utf8proc (dependabot[bot])
1.9.0-dev.157 | 2023-10-23 10:58:57 +0200
* Fix spicy-build to correctly infer library directory. (Robin Sommer, Corelight)
Closes https://github.com/zeek/zeek/issues/3384.
1.9.0-dev.155 | 2023-10-23 10:54:13 +0200
* GH-1565: Disable capturing backtraces with HILTI exceptions in non-debug builds. (Robin Sommer, Corelight)
They can be expensive to capture, and aren't used anywhere by default
unless explicitly requested.
We change it so that the exception class still remains ABI
compatible between release and debug builds. It's the compilation
settings of the code including `exception.h` that determines if a
backtrace it captured.
Closes #1565.
1.9.0-dev.152 | 2023-10-23 10:49:19 +0200
* GH-1567: Speed up runtime calls to start profilers. (Robin Sommer, Corelight)
We now cache the profiler tags to avoid frequent re-computation.
Closes #1567.
1.9.0-dev.150 | 2023-10-23 10:48:59 +0200
* GH-1568: Fix bitfield's lack of declaring its C++-side type dependencies. (Robin Sommer, Corelight)
Closes #1568.
1.9.0-dev.148 | 2023-10-23 10:10:39 +0200
* Remove check of Zeek docs in Cirrus cron config. (Benjamin Bannier, Corelight)
This is a follow-up to 2b92da596631ff1a29b7deac05e00faaad9305f3.
1.9.0-dev.146 | 2023-10-13 12:26:24 +0200
* Remove Zeek-specific documentation. (Robin Sommer, Corelight)
This now lives Zeek-side.
This keeps the section structure so that we make things easier to
find, and put in pointers to the new Zeek-side documentation
It leaves everything in the development section as is; hard to piece
apart and seems fine to leave it here.
1.9.0-dev.144 | 2023-10-13 11:47:35 +0200
* GH-1571: Remove trimming inside individual chunks. (Benjamin Bannier, Corelight)
Trimming `Chunk`s (always from the left) causes a lot of internal work
with only limited benefit since we manage visibility with `stream::View`s
on top of `Chunk`s anyway.
This patch removes trimming inside `Chunk`s so now any trimming only
removes `Chunk`s from `Chain`s, but does not internally change
individual `Chunk`s anymore. This might lead to slightly increased memory
use, but callers usually have that data in memory anyway.
Closes #1571.
1.9.0-dev.142 | 2023-10-09 11:10:35 +0200
* Explicitly set ASM compiler. (Benjamin Bannier, Corelight)
When not setting an ASM compiler we might end up configuring Clang as C
compiler, but GCC as ASM compiler. The code in `3rdparty/fiber` then
causes Clang-only flags like `-Weverything` to be passed to GCC (which
does not understand it).
The way we need to do this is subtle. We need to rely on
`3rdparty/fiber` to explicitly `enable_language(ASM)` and not set it as
a project language to work around `find_package(BISON)` breaking if
`CMAKE_ASM_COMPILER` is set. By removing ASM from project languages and
setting the ASM config after Bison was found, but before we configure
`3rdparty/fiber` things seem to work as intended.
1.9.0-dev.140 | 2023-10-06 12:34:02 +0200
* Set `ParserState::begin` on all possible paths into a parser. (Benjamin Bannier, Corelight)
We previously would sometimes leave `ParserState::begin` unset, e.g.,
when parsing beginning parsing of a top-level production. With this
patch we now set its value on all possible paths.
* Fix spicy-rt-tests for freebsd-12. (Benjamin Bannier, Corelight)
* GH-1089: Make `offset()` independent of random access functionality. (Benjamin Bannier, Corelight)
With this patch we store the value returned by `offset()` directly in
the unit instead of computing it on the fly when requested from `cur -
begin`. With that `offset()` can be used without enabling random access
functionality on the unit.
Closes #1089.
* Move `offset()` into its own feature. (Benjamin Bannier, Corelight)
This patch makes `__position` depend on `uses_offset` so uses of it
(`position()`, `offset()`) can be tracked separately. Since `offset()`
still requires random access `uses_offset` enables `uses_random_access`
for that use case; we plan to relax that in a follow-up patch.
* Consistently wrap `ParserState::begin` in optional. (Benjamin Bannier, Corelight)
1.9.0-dev.134 | 2023-10-04 11:20:46 +0200
* Bump 3rdparty/fiber from `f75b2f9` to `ada36b2` (dependabot[bot])
Bumps [3rdparty/fiber](https://github.com/simonfxr/fiber) from `f75b2f9` to `ada36b2`.
- [Commits](https://github.com/simonfxr/fiber/compare/f75b2f93aa312b9922f9c977021e5470ff7715fa...ada36b254c10d487eb4d8d108a3cb156538e1885)
---
updated-dependencies:
- dependency-name: 3rdparty/fiber
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* Bump 3rdparty/benchmark from `0d98dba` to `7736df0` (dependabot[bot])
Bumps [3rdparty/benchmark](https://github.com/google/benchmark) from `0d98dba` to `7736df0`.
- [Release notes](https://github.com/google/benchmark/releases)
- [Commits](https://github.com/google/benchmark/compare/0d98dba29d66e93259db7daa53a9327df767a415...7736df03049c362c7275f7573de6d6a685630e0a)
---
updated-dependencies:
- dependency-name: 3rdparty/benchmark
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* Bump 3rdparty/SafeInt from `4cafc91` to `925235c` (dependabot[bot])
Bumps [3rdparty/SafeInt](https://github.com/dcleblanc/SafeInt) from `4cafc91` to `925235c`.
- [Release notes](https://github.com/dcleblanc/SafeInt/releases)
- [Commits](https://github.com/dcleblanc/SafeInt/compare/4cafc9196c4da9c817992b20f5253ef967685bf8...925235c06de490865f871218fa2bd8ac38241a95)
---
updated-dependencies:
- dependency-name: 3rdparty/SafeInt
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* GH-1549: GH-1554: Fix potential infinite loop when trimming data before stream. (Benjamin Bannier, Corelight)
Previously we would trigger an infinite loop if one tried to trim before
the head chunk of a stream. In praxis this seem to have been no issue
due to #1549 and us emitting way less calls to trim than possible.
This patch adds an explicit check whether we need to trim anything, and
exits the low-level function early for such cases.
Closes #1554.
1.9.0-dev.128 | 2023-09-29 13:27:28 +0200
* GH-1550: Replace recursive deletion with explicit loop to avoid stack overflow. (Benjamin Bannier, Corelight)
1.9.0-dev.126 | 2023-09-29 13:26:53 +0200
* GH-1549: Add feature guards to accesses of a unit's `__position`. (Benjamin Bannier, Corelight)
Access of `__position` triggers a random access functionality. In order
to distinguish our internal uses from accesses due to user code, most
access in our generated code should be guarded with a feature constant
(`if` or ternary).
In this patch add proper guards for a couple instances where we did not
do that correctly. That mishap caused all units with containers to be
random access (even the root unit) which in turn could have lead to
e.g., unbounded memory growth, or runtime overhead due to generation and
execution of unneeded code, or expensive cleanup on very large untrimmed
inputs.
Closes #1549.
* Refactor feature guard helper to directly take unit ID instead of type. (Benjamin Bannier, Corelight)
1.9.0-dev.123 | 2023-09-27 09:41:04 +0200
* Escape rendered bitfield fields. (Benjamin Bannier, Corelight)
* Enforce that `Bitfield`s always own their fields. (Benjamin Bannier, Corelight)
* Fix typos. (Benjamin Bannier, Corelight)
1.9.0-dev.119 | 2023-09-27 09:39:35 +0200
* GH-1542: GH-1547: Bump dependencies. (Benjamin Bannier, Corelight)
1.9.0-dev.111 | 2023-09-26 16:05:25 +0200
* Add GH dependabot config. (Benjamin Bannier, Corelight)
1.9.0-dev.109 | 2023-09-22 16:35:02 +0200
* Add explicit dependency for local pre-commit hook. (Benjamin Bannier, Corelight)
* Fix file selection in clang-format pre-commit hook. (Benjamin Bannier, Corelight)
1.9.0-dev.106 | 2023-09-22 12:19:21 +0200
* GH-1533: Fix access to anonymous bitfield element through a constant value. (Robin Sommer, Corelight)
1.9.0-dev.104 | 2023-09-22 11:04:19 +0200
* Artificially limit the number of open files. (Benjamin Bannier, Corelight)
This works around a silent failure in reproc where it would refuse to
run on systems which huge rlimits for the number of open files. We have
seen this hit on huge production boxes.
1.9.0-dev.102 | 2023-09-22 11:04:01 +0200
* GH-1478: Add regression test for #1478. (Benjamin Bannier, Corelight)
* Add begin to parser state. (Benjamin Bannier, Corelight)
This patch adds the current begin position to the parser state, and
makes the corresponding changes to generated parser functions so it is
passed down.
We already modelled the semantic beginning of the input in the unit, but
had no reliable way to keep this up-to-date across non-unit contexts
like `&parse-from`. This would then for certain setups lead to generated
code where `input` and `position` would point to different inputs which in
turn caused `offset` (modelled as `position - input`) to be incorrect.
* Factor computation of feature const into separate function. (Benjamin Bannier, Corelight)
* Expand validator error message. (Benjamin Bannier, Corelight)
* Add trait class to bitfield types. (Robin Sommer, Corelight)
This lets C++ templates test if a class T is a bitfield.
1.9.0-dev.95 | 2023-09-20 12:14:50 +0200
* Add trait class to bitfield types. (Robin Sommer, Corelight)
1.9.0-dev.93 | 2023-09-20 09:26:29 +0200
* Declare Spicy pygments extension as parallel-safe. [skip CI] (Benjamin Bannier, Corelight)
1.9.0-dev.91 | 2023-09-19 11:50:32 +0200
* Skip validating links to https://www.icir.org/hilti. (Benjamin Bannier, Corelight)
* Document using anonymous field for extracting TCP messages. (Benjamin Bannier, Corelight)
1.9.0-dev.88 | 2023-09-15 10:47:53 +0200
* Use find_package(Python) with version. (Arne Welzel, Corelight)
Zeek's configure sets Python_EXECUTABLE has hint, but Spicy is using
find_package(Python3) and would only use Python3_EXECUTABLE as hint.
This results in Spicy finding a different (the default) Python executable
when configuring Zeek with --with-python=/opt/custom/bin/python3.
Switch Spicy over to use find_package(Python) and add the minimum
version so it knows to look for Python3.
1.9.0-dev.86 | 2023-09-14 10:36:25 +0200
* Add support for passing arbitrary C++ compiler flags. (Benjamin Bannier, Corelight)
This adds a magic environment variable `HILTI_CXX_FLAGS` which if set
specifies compiler flags which should be passed during C++ compilation
after implicit flags. This could be used to e.g., set defines, or set
low-level compiler flags.
Even with this flag, for passing include directories one should still
use `HILTI_CXX_INCLUDE_DIRS` since they are searched before any
implicitly added paths.
1.9.0-dev.84 | 2023-09-07 17:12:00 +0200
* GH-1467: Support bitfield constants in Spicy for parsing. (Robin Sommer, Corelight)
One can now define bitfield "constants" for parsing by providing
integer expressions with fields:
type Foo = unit {
x: bitfield(8) {
a: 0..3 = 2;
b: 4..7;
c: 7 = 1;
};
This will first parse the bitfield as usual and then enforce that the
two bit ranges that are coming with expressions (i.e., `a` and `c`)
indeed containing the expected values. If they don't, that's a parse
error.
We also support using such bitfield constants for look-ahead parsing:
type Foo = unit {
x: uint8[];
y: bitfield(8) {
a: 0..3 = 4;
b: 4..7;
};
};
This will parse uint8s until a value is discovered that has its bits
set as defined by the bitfield constant.
(We use the term "constant" loosely here: only the bits with values
are actually enforced to be constant, all others are parsed as usual.)
Closes #1467.
* Extend bitfield type with per-item storage for constant values. (Robin Sommer, Corelight)
This allows to associate an expression with each bit value. We don't
use this from HILTI because there's isn't a good syntax to do so
(and/or: it's not worth adding), but we'll use (and test) this from
Spicy in a subsequent commit.
* Add bitfield constants. (Robin Sommer, Corelight)
It's now possible to initialize a bitfield value through an assignment
from a struct constructor expression:
type BF = bitfield(8) {
a: 0..3;
b: 4..7;
c: 4..5;
};
global BF bf = [$a = 1, $c = 2];
* Change internal tuple representation of bitfield to store optional values. (Robin Sommer, Corelight)
This will allow us to create bitfield constants where not at all
elements are set. We also add an operator to test if an element is
set (test forthcoming in a subsequent commit).
1.9.0-dev.79 | 2023-09-07 11:15:05 +0200
* GH-1520: Fix handling of `spicy-dump --enable-print`. (Benjamin Bannier, Corelight)
This flag was understood, but not handled since at least v1.1.0.
1.9.0-dev.77 | 2023-09-06 14:38:55 +0200
* Clarify error handling docs [skip CI]. (Benjamin Bannier, Corelight)
The docs previously made it sound as if exceptions could potentially be
swallowed with `on %error` handlers. Since this is currently not
supported, but a often requested feature, clarify the docs to not stir
up too much anticipation.
1.9.0-dev.75 | 2023-09-05 16:15:17 +0200
* Merge branch 'topic/bbannier/rtd-search-suppress-doxygen-results' (Benjamin Bannier, Corelight)
* GH-1516: Do not include generated Doxygen documentation in RTD search results [skip CI]. (Benjamin Bannier, Corelight)
Doxygen documentation is likely not relevant for users and hides results
they are looking for. Also, Doxygen already comes with its own search
which is reachable the developer documentation.
Closes #1516.
* Reduce ccache cache size in CI. (Benjamin Bannier, Corelight)
1.9.0-dev.71 | 2023-09-01 10:33:35 +0200
* Drop removed RTD config key. (Benjamin Bannier, Corelight)
1.9.0-dev.69 | 2023-09-01 09:57:22 +0200
* GH-1503: Handle anonymous bitfields inside `switch` statements. (Robin Sommer, Corelight)
We now map items of anonymous bitfields inside a `switch` cases into
the unit namespace, just like we already do for top-level fields. We
also catch if two anonymous bitfields inside those cases carry the
same name, which would make accesses ambiguous.
So the following works now:
```
switch (self.n) {
0 -> : bitfield(8) {
A: 0..7;
};
* -> : bitfield(8) {
B: 0..7;
};
};
```
Whereas this does not work:
```
switch (self.n) {
0 -> : bitfield(8) {
A: 0..7;
};
* -> : bitfield(8) {
A: 0..7;
};
};
```
The latter not working is reasonable I think, rather than trying to
figure out that the two cases have the same type and hence could be
mapped to a single data member in the resulting code (which we do, and
support, for cases of the same *name* and type).
Closes #1503.
* Polish string rendering of anonymous bitfields in structs. (Robin Sommer, Corelight)
We now print their ID as `<anon>`. This isn't perfect, it would be
nicer if we printed out the IDs of the bitfield's items at the
top-level (like `spicy-dump` does). However we don't have the
necessary type information available when rendering as a string, and
this seems good enough.
* Support `.?` and `?.` for items of anonymous bitfields. (Robin Sommer, Corelight)
1.9.0-dev.64 | 2023-08-28 17:05:18 +0200
* GH-1508: Fix returned value for `<unit>.position()`. (Benjamin Bannier, Corelight)
We declared this method to return an iterator, but potentially returned
an optional iterator. This patch add an additional deref so we get the
correct value.
1.9.0-dev.62 | 2023-08-28 12:11:40 +0200
* GH-1504: Use user-inaccessible chars for encoding `::` in feature variables. (Benjamin Bannier, Corelight)
When setting up feature tracking variables for the optimizer we
previously would normalize `:` as `_`, e.g., `mod::Unit` would lead to
feature variables `__feat%mod__Unit%...`. In various places in the
optimizer we would then use that knowledge to extract module and unit
ID. Since user-specified identifiers can contain literal `_` or `__`
this could confuse the optimizer.
With this patch we now normalize `:` as `@` which is not
user-accessible. We add a new codegen normalization so `@` becomes
`0x40`. This gives a more reliable encoding.
* Clean up extraction and handling of feature flag in optimizer. (Benjamin Bannier, Corelight)
1.9.0-dev.59 | 2023-08-22 15:10:07 +0200
* Add documentation for `export` extensions. (Robin Sommer, Corelight)
This goes with https://github.com/zeek/zeek/pull/3228.
* spicy-dump: Level up anonymous bitfields. (Robin Sommer, Corelight)
Spicy-dump now renders items of anonymous bitfields at the parent
level. Offsets are added accordingly as well if requested.
* spicy-dump: Make offsets in JSON output useful. (Robin Sommer, Corelight)
We were including offsets into spicy-dumps JSON output, but there was
no information in there saying which offsets belong to what field.
Turning `__offsets` from an array into a map indexed by field name now.
* spicy-dump: Change format how bitfields are printed. (Robin Sommer, Corelight)
We now render them as key/value form in both text and JSON output.
* Include a flag into a unit field's type information indicating if it's anonymous. (Robin Sommer, Corelight)
This preservers the information if a field had a name even though
internally we do carry a generated dummy name around in that case. For
now, we just adapt `spicy-dump` to not print the dummy name in that
case in its text output. Later, Zeek will leverage this when
auto-generating record types.
* GH-1468: Allow to directly access members of anonymous bitfields. (Robin Sommer, Corelight)
This works now:
type Foo = unit {
: bitfield(8) {
x: 0..3;
y: 4..7;
};
on %done {
print self.x, self.y;
}
};
* GH-1032: GH-1484: Switch Spicy over to use the new HILTI-side bitfield type. (Robin Sommer, Corelight)
* Implement remaining pieces to make `bitfield` a full HILTI type. (Robin Sommer, Corelight)
We keep representing bitfields as a C++ tuple behind the scenes, but
no longer map it into a HILTI-side tuple as well. Instead `Bitfield`
is now a regular type with corresponding operators and runtime
functionality.
The one change to the C++ tuple representation is that we add an
additional hidden tuple element at the end that stores the original
integer value. That allows for displaying it (which we will do for now
only inside Spicy debug output) and could later also facilitate modifying
fields and reconstructing an integer.
* Move `bitfield` type from Spicy to HILTI. (Robin Sommer, Corelight)
This prepares for making `bitfield` a fully-supported HILTI type. This
change only moves existing code over to HILTI without breaking
Spicy-side functionality. There are still HILTI-side pieces missing
for providing full support, which will come next.
* Fix comparison for resolved operators. (Robin Sommer, Corelight)
The `isEqual` method was implemented in the base class, which led to
wrong template types being used.
A subsequent commit will depend on this working correctly, including a test
breaking if not.
1.9.0-dev.47 | 2023-08-14 11:32:33 +0200
* GH-1384: Fix stringification of `DecodeErrorStrategy`. (Benjamin Bannier, Corelight)
This was pointed out by Simeon Miteff in #1384.
1.9.0-dev.45 | 2023-08-14 11:32:11 +0200
* Fix handling of `--show-backtraces` flag. (Benjamin Bannier, Corelight)
This flag was intended and always documented as a toggle, but instead
implemented as an argument taking a value (which was never evaluated).
This patch fixes the implementation to align with itend and
documentation.
1.9.0-dev.43 | 2023-08-14 11:31:35 +0200
* Bump FreeBSD versions in CI. (Benjamin Bannier, Corelight)
1.9.0-dev.41 | 2023-07-19 09:17:47 +0200
* Mention unit switch statements in conditional parsing docs. (Benjamin Bannier, Corelight)
Instead of decorating many fields with `if` conditions for certain
grammars a simpler approach can be to use a unit switch statement with
branch-by-expressions. Add a link to unit switch from the conditional
parsing section.
1.9.0-dev.39 | 2023-07-18 11:21:27 +0200
* GH-1485: Add validator rejecting unsupported multiple uses of attributes. (Benjamin Bannier, Corelight)
While for some attributes like e.g., `&requires` we support multiple
mentions, for many we actually only support at most one instance. We
previously would only interpret only one, even if multiple instances
where given.
This patch adds a validator which rejects code with multiple instances
of such attributes. We also add test coverage for `&requires` supporting
multiple instances.
Closes #1485.
1.9.0-dev.37 | 2023-07-07 12:56:37 +0200
* Fix typo: maccOS -> macOS (Ryan Schmidt)
1.9.0-dev.35 | 2023-07-07 11:22:41 +0200
* GH-1465: Support skipping explicit `%done` in external hooks. (Robin Sommer, Corelight)
Assuming `Foo::X` is a unit type, these two are now equivalent:
on Foo::X::%done { }
on Foo::X { }
* GH-1465: Produce better error message when hooks are used on a
unit field. (Robin Sommer, Corelight)
1.9.0-dev.32 | 2023-07-05 13:03:56 +0200
* GH-1475: Add regression test for #1475. (Benjamin Bannier, Corelight)
1.9.0-dev.30 | 2023-07-04 17:10:32 +0200
* GH-1475: Revert #1439. (Benjamin Bannier, Corelight)
The changes for #1439 were incorrect and introduced incorrect parsing with
possible undefined behavior at runtime when encountering a combination of
list parsing and `&parse-from` or `&parse-at`.
This revert reopens GH-1421 and GH-1089.
1.9.0-dev.27 | 2023-06-28 14:43:05 +0200
* Run a few pre-commit checks only at `commit` stage. (Benjamin Bannier, Corelight)
* GH-1466: Reject uses of `self` in unit `&size` and `&max-size` attribute. (Benjamin Bannier, Corelight)
Values in `self` are only available after parsing has started while
`&size` and `&max-size` are consumed before that. This means that any
use of `self` and its members in these contexts would only ever see
unset members, so it should not be the intended use.
This patch adds a validation which rejects uses of `self` for these unit
attributes.
* Refactor validation code for unit `&size` and `&max-size` attributes. (Benjamin Bannier, Corelight)
1.9.0-dev.23 | 2023-06-28 10:20:18 +0200
* Bump doc sphinx-rtd-theme dependency. (Benjamin Bannier, Corelight)
1.9.0-dev.21 | 2023-06-27 10:04:53 +0200
* Fix rendering in validation of `%byte-order` attribute. (Benjamin Bannier, Corelight)
* GH-1275: Add missing lowering of Spicy unit ctor to HILTI struct ctor. (Benjamin Bannier, Corelight)
1.9.0-dev.18 | 2023-06-27 10:04:11 +0200
* Bump freebsd12 version in CI. (Benjamin Bannier, Corelight)
1.9.0-dev.16 | 2023-06-26 14:48:32 +0200
* Factor out code to guard search during recovery. (Benjamin Bannier, Corelight)
* GH-1231: GH-1464: Add special handling for potential `advance` failure in trial mode. (Benjamin Bannier, Corelight)
In 7c18597f1b95747b0305918eb1777ed22c06b166 for #1231 we added code to
handle advance failures in trial mode when synchronizing on regexps, but
missed adding the same support for literals. This patch adds that
missing part.
1.9.0-dev.13 | 2023-06-26 14:39:10 +0200
* GH-1446: Initialize generated struct members in constructor body. (Benjamin Bannier, Corelight)
We previously would generate them with default member initializers. This
lead to issues since for some constructs we need to create temporaries
(e.g., mutable accessors to `self`). With this patch we now always emit
a constructor body which ensures we have a block to attach temporaries
to.
* Disable implicit conversions of `cxx::Block` to `bool`. (Benjamin Bannier, Corelight)
I ran into this when I incorrectly tried to directly output a
`cxx::Block` to an iostream which worked but output a boolean (via
implicit conversion).
* Add missing newline in spicy-driver usage string. (Benjamin Bannier, Corelight)
1.9.0-dev.9 | 2023-06-20 08:48:49 +0200
* Fix signedness warning. (Arne Welzel, Corelight)
1.9.0-dev.7 | 2023-06-15 14:40:25 +0200
* Fix autogeneration of zeek docs. (Benjamin Bannier, Corelight)
1.9.0-dev.5 | 2023-06-14 16:37:26 +0200
* GH-1435: Add bitwise operators `&`, `|`, and `^` for booleans. (Benjamin Bannier, Corelight)
1.9.0-dev.3 | 2023-06-14 16:35:45 +0200
* GH-1343: Include condition in `&requires` failure message. (Benjamin Bannier, Corelight)
With this patch we now include the `&requires` condition in the failure
message in case the condition is not met. Since we do not have direct
access to the Spicy source code anymore we render HILTI code for now,
but undo the most common desugaring by replacing `__dd` with `$$`.
* Update links to some release artifacts. (Benjamin Bannier, Corelight)
* Add placeholder section for 1.9 to NEWS. (Benjamin Bannier, Corelight)
1.8.0 | 2023-06-14 08:53:34 +0200
* Release 1.8.0.
1.8.0-dev.176 | 2023-06-14 08:52:14 +0200
* Update Zeek docs for Zeek 6.0. (Robin Sommer, Corelight)
1.8.0-dev.173 | 2023-06-09 10:01:28 +0200
* GH-1447: Do not forcibly make `strong_ref` `in` function parameters immutable. (Benjamin Bannier, Corelight)
We previously would always emit a `const` value when passing a
`strong_ref` as a function parameter. This lead to the unintuitive error
that non-const `strong_ref` parameters could not be modified or have
non-const methods called on them (this would actually fail at codegen
time).
With this patch we remove the special treatment of `strong_ref`
parameter mutability.
Closes #1447.
1.8.0-dev.171 | 2023-06-09 10:00:51 +0200
* GH-1452: Allow resolving of unit parameters before `self` is fully resolved. (Benjamin Bannier, Corelight)
When constructing the list of declarations in a scope we previously
would not emit any items for struct types without fully resolved type.
This lead to the situation where when resolving references to a unit
parameter we might end up resolving them to similarly named globals
since they were available before the struct type was resolved.
This patch makes sure that we emit at least unit parameters at struct
scope level, even if we cannot emit `self` yet. This improves the
situation around globals mentioned above and is likely okay since `self`
should only ever resolve to something at struct scope.
Closes #1452.
1.8.0-dev.169 | 2023-06-09 09:59:42 +0200
* Fix docs for long-form of `-x` flag to spicyc. (Benjamin Bannier, Corelight)
The long form of `-x` is `--output-c++-files` but we documented
`--output-c++` instead which is used for `-c` and takes no argument.
1.8.0-dev.167 | 2023-06-09 09:59:08 +0200
* Replace obsolete uses of `egrep` with `grep -E`. (Benjamin Bannier, Corelight)
1.8.0-dev.165 | 2023-06-07 17:23:04 +0200
* Clean up dependencies for ubuntu-22 image. (Benjamin Bannier, Corelight)
1.8.0-dev.163 | 2023-06-07 15:15:10 +0200
* Remove unsupported fedora-36, add fedora-38. (Benjamin Bannier, Corelight)
* Bump supported Alpine version to latest release. (Benjamin Bannier, Corelight)
* Refresh supported Ubuntu versions to upstream supported ones. (Benjamin Bannier, Corelight)
* Remove EOL debian-9. (Benjamin Bannier, Corelight)
1.8.0-dev.158 | 2023-06-05 11:26:41 +0200
* GH-1448: Adjust getting started docs for removed JIT compilation via Zeek. (Benjamin Bannier, Corelight)
* Fix markup for code blocks. (Benjamin Bannier, Corelight)
1.8.0-dev.155 | 2023-05-26 13:59:22 +0200
* Remove Zeek from Spicy CI. (Benjamin Bannier, Corelight)
1.8.0-dev.153 | 2023-05-11 16:26:49 +0200
* Profile exception activity. (Robin Sommer, Corelight)
1.8.0-dev.151 | 2023-05-11 16:19:28 +0200
* Add new `skip` keyword to let unit items efficiently skip over
uninteresting data. (Robin Sommer, Corelight)
From the documentation:
For cases where your parser just needs to skip over some data, without
needing access to its content, Spicy provides a ``skip`` keyword to
prefix corresponding fields with:
.. spicy-code:: skip.spicy
module Test;
public type Foo = unit {
x: int8;
: skip bytes &size=5;
y: int8;
on %done { print self; }
};
``skip`` works for all kinds of fields but is particularly efficient
with ``bytes`` fields, for which it will generate optimized code
avoiding the overhead of storing any data.
``skip`` fields may have conditions and hooks attached, like
any other fields. However, they do not support ``$$`` in
expressions and hooks.
For readability, a ``skip`` field may be named (e.g., ``padding: skip
bytes &size=3;``), but even with a name, its value cannot be accessed.
* Deprecate parsing `void` fields with attributes. (Robin Sommer, Corelight)
`skip` supersedes the existing void-with-attributes parsing. For
now we continue to support that by internally mapping it over to
the new mechanism, but will eventually remove it. `void` without
attributes (for not parsing anything) remains supported.
* GH-1051: Support `&requires` with `void` fields. (Robin Sommer, Corelight)
1.8.0-dev.133 | 2023-05-10 13:11:54 +0200
* Skip failure notifications for unsuccessful non-error runs. (Benjamin Bannier, Corelight)
1.8.0-dev.131 | 2023-05-09 11:15:01 +0200
* Fix readthedocs build. (Benjamin Bannier, Corelight)
1.8.0-dev.129 | 2023-05-08 15:20:11 +0200
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Slightly tweak existing feature requirements test. (Benjamin Bannier, Corelight)
* GH-1089: Allow to use `offset()` without enabling full random-access support. (Benjamin Bannier, Corelight)
In particular this allows to use `offset()` while still keeping trimming
enabled.
* GH-1421: Store numerical offset in units instead of iterator for position. (Benjamin Bannier, Corelight)
We will use this in a subsequent patch to make `offset()` available
without having to enable full support for random-access.
Closes #1421.
1.8.0-dev.124 | 2023-04-28 11:41:21 +0200
* GH-1436: Make sure `Bytes::sub` only throws HILTI exceptions. (Benjamin Bannier, Corelight)
We previously would directly invoke `Base::substr` which can throw
and leak `std::out_of_range`. We now make sure to only throw HILTI
exceptions from that function.
Closes #1436.
* Implement all `Bytes::sub` in terms of bytes offsets. (Benjamin Bannier, Corelight)
We will use this in a subsequent patch which cleans up exception
handling from `std::string::substr`.
1.8.0-dev.121 | 2023-04-27 15:19:08 +0200
* Suppress uninteresting compiler diagnostic in test stderr diff. (Benjamin Bannier, Corelight)
* Fix gcc-13 warnings about possibly dangling references. (Benjamin Bannier, Corelight)
* Roll back 3rdparty/reproc upgrade. (Benjamin Bannier, Corelight)
* Silence gcc-13 warning for reproc. (Benjamin Bannier, Corelight)
* Add missing include for `uint64_t`. (Benjamin Bannier, Corelight)
1.8.0-dev.115 | 2023-04-24 10:59:27 +0200
* Make sure Spicy runtime config is initialized after `spicy::rt::init`. (Benjamin Bannier, Corelight)
We previously wouldn't explicitly initialize the Spicy runtime
configuration but instead rely on `configuration::get` (which implicitly
initializes it) to be called somewhere in the driver. This makes it hard
to use parsers outside of our drivers, we e.g., saw nullptr derefs in
our fuzzer binaries.
With this patch we force initialization in `spicy::rt::init` which is
the documented way to ensure all runtime dependencies are set up
correctly.
* Roll back 3rdparty/reproc downgrade. (Benjamin Bannier, Corelight)
This appears to have been downgraded to a two year old version by
accident.
1.8.0-dev.112 | 2023-04-24 10:10:57 +0200
* Move exception creation out of reference getters. (Benjamin Bannier, Corelight)
This helps reduce the size of the changed functions which might allow
more inlining.
* Make it cheaper to access a `ValueReference`. (Benjamin Bannier, Corelight)
Since we know that if a `ValueReference` holds a raw pointer it must
always be valid we can inline some paths from `_get` directly into
`_safeGet`. We also reduce the size of `_get` from the knowledge that we
must hold exactly two alternatives, and replace uses of `variant`
`std::get` (which sets up exception handling) with `std::get_if` which
is slightly cheaper at runtime.
* Disallow constructing ValueReferences from nullptrs. (Benjamin Bannier, Corelight)
We will use this in a subsequent patch which makes use of the fact that
`self` references are always valid.
1.8.0-dev.108 | 2023-04-24 10:10:26 +0200
* Add fast pass for iterator difference in `offset()`. (Benjamin Bannier, Corelight)
When generating code for `offset()` we previously would use the
difference of safe iterator which has extra code to make sure the two
iterators belong to the same stream. This check can never fail when
computing `offset()` so use a fast pass in that case.
1.8.0-dev.106 | 2023-04-21 12:03:21 +0200
* Bring new structure of safe/unsafe getters for global/config state to Spicy. (Robin Sommer, Corelight)
* Fix ordering issue in spicy-driver when shutting down runtime. (Robin Sommer, Corelight)
* Switch some more places to use unsafe getters for global/config state. (Robin Sommer, Corelight)
* Speed up access checks for `optional`. (Robin Sommer, Corelight)
* Provide unsafe fastpath to access global state. (Robin Sommer, Corelight)
* Move runtime configuration out of central global state. (Robin Sommer, Corelight)
* Reduce stack checking overhead. (Robin Sommer, Corelight)
* Reduce overhead of profiler operations when not needed. (Robin Sommer, Corelight)
* Add fast path to `View::extract()`. (Robin Sommer, Corelight)
* Inline a couple more stream methods. (Robin Sommer, Corelight)
1.8.0-dev.95 | 2023-04-19 10:13:12 +0200
* Fix gcc-13 warning about pessimizing move. (Benjamin Bannier, Corelight)
1.8.0-dev.93 | 2023-04-14 10:39:02 +0200
* Implement `View::data` with `Bytes::append`. (Benjamin Bannier, Corelight)
* Avoid unneeded temporaries when appending `View` to `Bytes`. (Benjamin Bannier, Corelight)
We used `View::data` to constructs a new `Bytes` to append from. Since
we cannot reuse the temporary in the append but have to copy the data
this introduced unneeded overhead. This patch inlines the low-level
`View` block iteration into `Bytes::append`.
I looked at this coming from zeek/spicy-dns#7, and with that particular
test case where many small views are appended the speedup is only ~1.5%.
I suspect the speedup to be bigger the bigger the view we append.
In order to amortize reallocation costs I also looked into aggressively
reserving capacity in the different `append` overloads, e.g., if the new
size is bigger than the current capacity, reserve double the needed
capacity in anticipation of more append operations to come. This did not
make a noticeable difference, probably due to the underlying
`std::string::append` already doing something similar.
* Bump Zeek docs. (Benjamin Bannier, Corelight)
1.8.0-dev.89 | 2023-04-12 19:40:28 +0200
* Pin btest version on ubuntu-16 in CI. (Benjamin Bannier, Corelight)
The recently released btest-1.0 requires python-3.7 while this system
has only python-3.5.
1.8.0-dev.87 | 2023-04-12 17:37:26 +0200
* Bump copyright dates in license headers. (Benjamin Bannier, Corelight)
* Use diffutils instead of BusyBox `diff` in Alpine CI. (Benjamin Bannier, Corelight)
The version of `btest` recently published to pypi uses flags which are
unsupported by the BusyBox `diff` binary which ships with Alpine by
default, see https://github.com/zeek/btest/issues/92. Use `diffutils`
instead.
* Add missing license headers. (Benjamin Bannier, Corelight)
* Use safe helper to get runtime configuration. (Benjamin Bannier, Corelight)
We previously would directly access a value from the configuration in
the global state in functions for `accept_input` and `decline_input`.
This is only safe if the configuration was created previously, but would
read uninitialized memory if e.g., a parser using these functions is
embedded in code not doing that.
With this patch we use a safe accessor for the configuration which
ensures that it always contains a good value.
1.8.0-dev.81 | 2023-04-12 09:11:00 +0200
* GH-1416: Fix build with gcc13 development snapshot. (Benjamin Bannier, Corelight)
This patch bumps 3rdparty/reproc to a version including
DaanDeMeyer/reproc@0b23d88894ccedde04537fa23ea55cb2f8365342
so Spicy can be build with a gcc13 development snapshot. We do not bump
to the latest development snapshot since
DaanDeMeyer/reproc@51540e1d1f7be2a82f9f4ae80c41232205b6b0ab changed the
process ownership semantics which would require changes on our side; we
will tackle that once upstream has committed to an API in a release.
This patch does not add CI for gcc13 since it is not yet released.
1.8.0-dev.79 | 2023-04-07 09:31:18 +0200
* Avoid some redundant location tracking code. (Robin Sommer, Corelight)
* Emit Spicy globals as true C++ globals by default. (Robin Sommer, Corelight)
This improves runtime performance. We retain the original approach
through an option: `--cxx-enable-dynamic-globals` now switches back to
managing globals through dynamically allocated structs, with each
runtime context creating separate instances.
* Fix compiler warnings. (Robin Sommer, Corelight)
These have started appearing with recent clangs:
```
warning: variable 'yynerrs_' set but not used
```
* Inline `stream::View::size()` and `stream::View::~View()`. (Robin Sommer, Corelight)
* Move runtime location tracking to dedicated TLS variable. (Robin Sommer, Corelight)
* Remove custom location tracking that some runtime types were using. (Robin Sommer, Corelight)
* Apply more fiber tuning. (Robin Sommer, Corelight)
1.8.0-dev.70 | 2023-04-05 10:38:03 +0200
* Bump Zeek docs. (Benjamin Bannier, Corelight)
* Preserve indention when extracting comments in doc generation. (Benjamin Bannier, Corelight)
1.8.0-dev.66 | 2023-04-03 14:28:14 +0200
* Fix signed-unsigned comparison. (Benjamin Bannier, Corelight)
1.8.0-dev.64 | 2023-04-03 09:52:25 +0200
* Document how to check whether an `optional` value is set. (Benjamin Bannier, Corelight)
* Fix docs reference typo. (Benjamin Bannier, Corelight)
* Bump Zeek docs. (Benjamin Bannier, Corelight)
1.8.0-dev.59 | 2023-03-23 15:51:45 +0100
* Install missing dependency for running with clang-15. (Benjamin Bannier, Corelight)
Between clang-15.0.6 upstream broke out the runtime libraries needed for
e.g., ASAN into a separate, optional package. Install it as well.
* Remove unused CI setups for packages. (Benjamin Bannier, Corelight)
1.8.0-dev.56 | 2023-03-23 14:15:40 +0100
* Fix potential double accounting for units in profiling output.
(Robin Sommer, Corelight)
1.8.0-dev.54 | 2023-03-23 11:49:32 +0100
* GH-1396: Fix regex performance regression introduced by constant
folding. (Robin Sommer, Corelight)
* Cache regular expressions to compile each only once. (Robin
Sommer, Corelight)
1.8.0-dev.51 | 2023-03-23 11:09:17 +0100
* GH-1399: Guard access to unit `_filters` member with feature flag. (Benjamin Bannier, Corelight)
The unit `_filters` member is accessed by most low-level functions
extracting data from the input, but its value is only interesting if a
unit actually supports filters. We had previously annotated `_filters`
with a requirement for filter support so any use of it with trigger
enabling support. By guarding access with a feature flag we can detect
such dependent uses in a subsequent patch.
* Extend detection of conditionally used features in optimizer. (Benjamin Bannier, Corelight)
We previously were only able to detect conditionally used features if
the feature flag was an `if` condition. In order to support
conditionally used features in expressions (as opposed to statements) we
add support for detecting their use under a ternary operator with this
patch.
* Add support for folding ternary operator uses in optimizer. (Benjamin Bannier, Corelight)
1.8.0-dev.47 | 2023-03-23 10:21:35 +0100
* Add runtime profiling infrastructure. (Robin Sommer, Corelight)
This add an option `-enable-profiling` to the HILTI and Spicy
compilers. Use of the option does two things: (1) it sets a flag
enabling inserting additional profiling instrumentation into
generated C++ code, and (2) it enables using instrumentation for
recording profiling information during execution of the compiled
code, incl. dumping out a profiling report at the end. The
profiling information collected includes time spent in HILTI
functions as well as for parsing Spicy units and unit fields.
* Fix parser generator to not store any parsed items for transient
containers. (Robin Sommer, Corelight)
* Disallow using `$$` with anonymous containers. (Robin Sommer, Corelight)
Transient containers don't store their values, which means `$$`
produces an always-empty container. Disallowing its use avoids simple
mistakes.
* Fiber tuning (Robin Sommer, Corelight)
- Tune buffer memory management during stack switching.
- Tune fibers' stack buffer management for less memory activity.
- Track high-level water mark for stack size needed by fibers.
1.8.0-dev.30 | 2023-03-22 15:58:42 +0100
* Update docs for changes in spicy-plugin. (Benjamin Bannier, Corelight)
1.8.0-dev.28 | 2023-03-10 11:22:48 +0100
* Fix stringification of enum values. (Benjamin Bannier, Corelight)
If an enum type or label used a C++ reserved identifier we would
previously stringify to name normalized for use in C++. Since we are
generating strings this is not needed and also confusing as stringified
names deviate from what the user wrote in the input.
With this patch we emit names which directly correspond to input
Spicy/HILTI names and labels.
* GH-1394: Fix C++ normalization of generated enum values. (Benjamin Bannier, Corelight)
Closes #1394.
1.8.0-dev.25 | 2023-03-08 16:27:27 +0100
* Fix typo in types documentation (tbfhg)
1.8.0-dev.23 | 2023-03-08 11:38:16 +0100
* Only run lint CI task from scheduled builds or PRs. (Benjamin Bannier, Corelight)
1.8.0-dev.21 | 2023-03-07 09:44:38 +0100
* Avoid expensive checked iterator for internal `Bytes` iteration. (Benjamin Bannier, Corelight)
We previously would go through a checked iterator to iterate a `Bytes`
instance. Since none of the iterators were user-provided this was always
safe, but incurred the performance overhead for checking.
With this patch we iterate the underlying bytes directly.
* GH-1390: Initialize `Bytes` internal control block for all constructors. (Benjamin Bannier, Corelight)
`Bytes` inherits constructors from its base class. We previously would
not initialize the internal control block for these constructors so some
operations involving iterators on such `Bytes` instances would have
failed, even thought the controlled object was still valid.
With this patch we always set up the control block, no matter which
constructor is used.
Closes #1390.
1.8.0-dev.18 | 2023-03-07 09:43:59 +0100
* Do not force locale on users of libhilti. (Benjamin Bannier, Corelight)
Setting a locale from a library can interfere negatively with the rest
of the application. We previously would set a locale inside
`hilti::rt::init`, but we probably do not need it at all.
1.8.0-dev.16 | 2023-03-06 16:24:55 +0100
* Cache HILTI C++ compilations with ccache in CI. (Benjamin Bannier, Corelight)
1.8.0-dev.14 | 2023-03-06 16:24:20 +0100
* GH-1386: Prevent internal error when passed invalid context. (Benjamin Bannier, Corelight)
We previously would trigger an internal error when encountering
`self.context()` in a unit which declared a `%context` to something
which was not a type. With this patch we now reject such calls.
We picked a simplified implementation strategy: when querying a `Unit`'s
`contextType` we still return an optional type to handle both units
without context as well as units where %context does not point to a
type; if a context is present but does not refer to a type we return an
empty result. With that we potentially trigger two errors, from
validating that `%context` refers to a type, and from validating that
the unit had a `%context` when using `self.context()`, i.e., if the
`%context` is invalid we continue parsing as if the unit had no
`%context` at all. The alternative would have been to return e.g., a
`Result<optional_ref<Type>>` from `contextType` to capture both the
unset as well as invalid case and move some of the validation into
`contextType` itself; this still would have led to a lot of manual
unwrapping in users.
Closes #1386.
1.8.0-dev.12 | 2023-02-24 17:13:42 +0100
* Fix potential use-after-move bug. (Benjamin Bannier, Corelight)
1.8.0-dev.10 | 2023-02-22 15:42:04 +0100
* Switch `SafeInt` to current upstream. (Robin Sommer, Corelight)
1.8.0-dev.8 | 2023-02-22 11:16:08 +0100
* Set reupload_on_changes for ccache caches. (Arne Welzel, Corelight)
* Build centos stream again for PRs, disable clang-12 debug. (Benjamin Bannier, Corelight)
1.8.0-dev.5 | 2023-02-20 14:31:21 +0100
* Fix sign-unsigned comparison. (Benjamin Bannier, Corelight)
* GH-169: Build debian-11 instead of centos stream 8 in CI. (Benjamin Bannier, Corelight)
* Update docs for removal of fedora-35, addition of fedora-36. [skip CI] (Benjamin Bannier, Corelight)
* Add placeholder section for 1.8 to NEWS. (Benjamin Bannier, Corelight)
1.7.0 | 2023-02-13 17:41:31 +0100
* Release 1.7.0.
1.7.0-dev.124 | 2023-02-13 17:41:08 +0100
* Revert "Make all CI tasks greedy." (Benjamin Bannier, Corelight)
This reverts commit 3176c2cadc71aaf4c0c3dd28d37c1d465323da8c.
* Revert "Decrease excessive RAM on some CI platforms." (Benjamin Bannier, Corelight)
This reverts commit 6ce665b9f3dc7f3630167e1bdf3f752ce3647dcf.
1.7.0-dev.121 | 2023-02-13 14:47:21 +0100
* GH-1310: Fix ASAN false positive with GCC. (Robin Sommer, Corelight)
* Skip clang-specific ASAN flags with other compilers. (Robin Sommer, Corelight)
* Don't instantiate a debug logger if we aren't going to debug log. (Robin Sommer, Corelight)
* Simplify runtime library's extract methods. (Robin Sommer, Corelight)
* Shortcut some stream offset computations. (Robin Sommer, Corelight)
* GH-1345: Apply alternative fix for #1345. (Robin Sommer, Corelight)
1.7.0-dev.112 | 2023-02-09 11:03:09 +0100
* Make `printParserState` cheaper to call if debug logging is disabled. (Benjamin Bannier, Corelight)
1.7.0-dev.110 | 2023-02-03 15:57:39 +0100
* https://github.com/zeek/spicy-plugin/issues/134: Search
`HILTI_CXX_INCLUDE_DIRS` paths before default include paths.
(Robin Sommer, Corelight)
* Support Zeek-style documentation strings in Spicy source code. (Robin Sommer, Corelight)
This adds support for extracting and retaining Zeek-style
documentation strings (`##`, `##!`) inside Spicy source code to any
top-level AST nodes where it's appropriate: if they are prefixed with
a block of such comments, that information is retained along with the
node for later access.
The code printer (`-p`) learns to emit doc strings coming with the
nodes it's printing out.
* Bump Zeek docs. (Benjamin Bannier, Corelight)
1.7.0-dev.105 | 2023-02-01 11:17:25 +0100
* Documentation updates. (Robin Sommer, Corelight)
- Documenting the new type `export` feature for Zeek.
- Update Zeek docs for JIT removal and `accept_input`/`decline_input`.
1.7.0-dev.102 | 2023-01-31 14:02:45 +0100
* GH-1367: Use unique filename for all object files generated during JIT. (Benjamin Bannier, Corelight)
* Bump Zeek docs. (Benjamin Bannier, Corelight)
* Point doc repo links to latest release instead of `main`. (Benjamin Bannier, Corelight)
* Fix installation links for macos platforms. (Benjamin Bannier, Corelight)
* Bump latest release in docs. (Benjamin Bannier, Corelight)
1.7.0-dev.95 | 2023-01-27 12:00:41 +0100
* Streamline runtime exception hierarchy. (Robin Sommer, Corelight)
We were inconsistent in the structure of exceptions that could
happen during runtime, making it hard for host applications for
consistently handle trouble. We tweak this, as follows:
- We now only have two general subcategories of exceptions under the
top-level `hilti::rt::Exception`:
- `RuntimeError`: non-fatal errors that mean a host
applications cannot complete the current line of processing,
but can otherwise continue. This is the main exception that
host applications should catch, and then proceed
appropriately.
- `UsageError`: fatal errors that indicate something about
using the runtime environment was fundamentally wrong, and a
host application should abort.
- Almost all other exceptions are now derived from `RuntimeError`,
including Spicy's `ParseError` and `RecoverableFailure`, so that
they can treated consistently.
1.7.0-dev.93 | 2023-01-27 11:57:58 +0100
* Search user module paths before system paths. (Robin Sommer, Corelight)
With, e.g., `spicyz -L /x/y/z`, we used to append the given module
path to the internal system paths, so that it was searched last. This
change turns around the priority to search `/x/y/x` first, before the
internal ones. That seems both more useful and more intuitive.
* Provide ability for host applications to initiate runtime's
module-pre-init phase manually. (Robin Sommer, Corelight)
1.7.0-dev.89 | 2023-01-27 11:44:31 +0100
* Add DPD-style `spicy::accept_input()` and `spicy::decline_input()`. (Robin Sommer, Corelight)
This generalizes the accepting/decline input from being a Zeek thing to
being part of Spicy. Any Spicy grammar can call these functions as to
tell its host application if it recognizes the input format. The host
application can (but doesn't have to) install callbacks to receive
that information and react appropriately. For example, the Zeek
plugin will just forward it to Zeek's standard DPD hooks.
For testing, `spicy-driver` gets a new option `--require-accept`
that lets it exit with success only if `spicy::accept()` has
been called, and `spicy::decline_input()` hasn't.
Note: The naming of the two functions is avoid conflicts with the
existing notion of synchronization's confirm/reject.
* Add a global configuration object to the Spicy runtime. (Robin Sommer, Corelight)
We didn't have a mechanism yet for a host application to configure the
runtime library's behavior. This follows what we do on the HILTI-side
to provide that capability.
1.7.0-dev.86 | 2023-01-27 11:35:35 +0100
* Add driver option to output full set of generated C++ files. (Robin Sommer, Corelight)
`-x <prefix>` now writes out all generated C++ code as
`<prefix>_*.cc`, also setting the internal C++ namespace prefix to
`<prefix>`.
* Add test for sink's paying attention to linker scopes with static C++ code. (Robin Sommer, Corelight)
* Make test helper scripts `cxx-*` usable with Spicy code as well. (Robin Sommer, Corelight)
* Provide new HILTI keyword `$scope` to access linker scope. (Robin Sommer, Corelight)
* Prefix the linker scope with the internal CXX namespace. (Robin Sommer, Corelight)
* Fix a couple of places hardcoding the internal C++ namespace prefix. (Robin Sommer, Corelight)
* Prefix the linker symbols for versioning with C++ namespace. (Robin Sommer, Corelight)
1.7.0-dev.78 | 2023-01-27 11:16:01 +0100
* GH-1123: Support arbitrary expression as argument to type
constructors, such as `interval(...)`. (Robin Sommer, Corelight)
So far we used constructor expressions like `interval(...)` to create
*constants* of the given type, and for that we required that the
argument was a single atomic value. The result was that these
constructor expressions were really more like literals for the
corresponding types than "normal" expressions. While that's useful
internally, it's confusing to the user. This commits changes that
to support arbitrary expressions as arguments.
To make this change without loosing a way to create actual constants
(which we need at some places), we implement this in two stages: the
parser initially turns such `interval(...)` expressions into call
operators[1] that the types now define for each desired constructor. We
then extend the normalizer with a new constants folding pass that
turns the call expressions into constants if possible.
The new constant folder remains limited to the cases we need for
this use case, but we'll be able to expand that later to more
general folding.
* Fix bug in cast from `real` to `interval`. (Robin Sommer, Corelight)
* Add anchors to FAQ items in docs [skip CI]. (Benjamin Bannier, Corelight)
1.7.0-dev.53 | 2023-01-25 09:39:02 +0100
* Add stringification to UnitContext. (Benjamin Bannier, Corelight)
1.7.0-dev.51 | 2023-01-25 09:36:56 +0100
* Remove potential race during JIT when using `HILTI_CXX_COMPILER_LAUNCHER`. (Benjamin Bannier, Corelight)
1.7.0-dev.49 | 2023-01-20 10:41:46 +0100
* GH-1349: Fix incremental regexp matching for potentially empty results. (Robin Sommer, Corelight)
When feeding data incrementally into the regexp matcher, we could run
into an assert trying to ensure that the end of a match wasn't in the
previous chunk already. Turns out, however, that *is* possible with
empty matches that are determined only later (and maybe it's possible
in other cases, too). To fix this, we need to account for potentially
backtracking into data from a previous round. This could be a bit
surprising to callers of the `advance()` method, but should
work as long as they aren't trimming the stream in between (which in
our use cases shouldn't be the case).
Closes #1349.
* Install full nlohmann JSON header as well. (Benjamin Bannier, Corelight)
We previously would only install the nlohmann header with forward
declaration while in `hilti/rt/json.h` we include the full header. With
that it was impossible to include this header in user code.
This patch makes sure we install the full header as well.
1.7.0-dev.45 | 2023-01-19 09:46:14 +0100
* Remove dead code. (Benjamin Bannier, Corelight)
* Remove performance pessimization introduced in `84fb4f21f494`. (Benjamin Bannier, Corelight)
When exposing this function directly we made the argument always
evaluate. This now causes us to compute strings for logging even if they
are never emitted.
Undo this change as it is actually not needed for the cleanup it was
initially intended for.
* GH-1345: Fix pathological performance for `_haveEod` on highly chunked streams. (Benjamin Bannier, Corelight)
This function was identified as a performance bottleneck when parsing
`bytes` with a huge size from a stream with many chunks. The reason it
performed so poorly was due to its use of `unsafeEnd`: since `unsafeEnd`
creates an unsafe iterator from `end()` it always needs to compute the
chunk the safe iterator pointed to; if the stream contains many chunks
this causes the seen poor performance.
Since we only access `offset` use safe iterators instead. This makes it
much cheaper to call this function so we can safely use it during
parsing.
Since this change only affects performance we do not add a test case.
For a benchmark see #1345.
Closes #1345.
* Remove EOL fedora-35, add fedora-37. (Benjamin Bannier, Corelight)
* Bump Zeek version in zeek-plugin CI job. (Benjamin Bannier, Corelight)
* Bump Zeek docs. (Benjamin Bannier, Corelight)
1.7.0-dev.37 | 2023-01-09 16:10:40 +0100
* Decrease excessive RAM on some CI platforms. (Benjamin Bannier, Corelight)
* Make all CI tasks greedy. (Benjamin Bannier, Corelight)
* Rerun BTest up to three times in CI. (Benjamin Bannier, Corelight)
1.7.0-dev.33 | 2023-01-05 17:33:00 +0100
* Update repo to not publish Docker images anymore. (Benjamin Bannier, Corelight)
1.7.0-dev.31 | 2023-01-05 15:38:13 +0100
* GH-1326: Generate proper runtime types for enums. (Benjamin Bannier, Corelight)
* Allow default for `vector::Allocator` with some non-class types. (Benjamin Bannier, Corelight)
* Remove unused define. (Benjamin Bannier, Corelight)
1.7.0-dev.25 | 2023-01-04 10:26:33 +0100
* GH-1330: Reject uses of imported module IDs as expression. (Benjamin Bannier, Corelight)
* Fix test `Backtrace.comparison` for aggressive constant folding. (Benjamin Bannier, Corelight)
1.7.0-dev.22 | 2022-12-14 13:25:57 +0100
* Push container images to Docker Hub zeek org, too. (Arne Welzel, Corelight)
1.7.0-dev.20 | 2022-12-14 13:24:47 +0100
* Bump Cirrus macos CI. (Benjamin Bannier, Corelight)
* Fix test `Backtrace.comparison` for aggressive constant folding. (Benjamin Bannier, Corelight)
* Remove workaround for outdated Homebrew db on Cirrus CI. (Benjamin Bannier, Corelight)
1.7.0-dev.16 | 2022-12-14 10:29:59 +0100
* Audit previously disabled clang-tidy-15 checks. (Benjamin Bannier, Corelight)
1.7.0-dev.4 | 2022-12-13 10:56:35 +0100
* Fix docs for enum using semicolon instead of comma. (Arne Welzel, Corelight)
1.7.0-dev.2 | 2022-12-13 10:43:41 +0100
* GH-1322: Bring implementation of `Sink::write` in line with declaration. (Benjamin Bannier, Corelight)
* Add placeholder section for 1.7 to NEWS. (Benjamin Bannier, Corelight)
1.6.0 | 2022-12-12 12:38:01 +0100
* Release 1.6.0. (Benjamin Bannier, Corelight)
1.6.0-dev.147 | 2022-12-12 09:40:49 +0100
* GH-1319: Add validation rejecting unsupported attributes on unit variables. (Benjamin Bannier, Corelight)
1.6.0-dev.145 | 2022-12-12 09:14:22 +0100
* doc/zeek: zkg template changed and mention packet analyzers (Arne
Welzel, Corelight)
1.6.0-dev.143 | 2022-12-12 09:02:59 +0100
* GH-1298: Fix/clarify bitfield docs. (Robin Sommer, Corelight)
1.6.0-dev.141 | 2022-12-12 08:56:24 +0100
* Expose new `pack`/`unpack` operators in Spicy. These provide
low-level primitives for transforming a value into, or out of, a
binary representations. See
https://docs.zeek.org/projects/spicy/en/latest/programming/language/packing.html
for more. Internally, `unpack` already existed; `pack` is new.
(Robin Sommer, Corelight)
* Update clang-tidy config for clang-15. (Benjamin Bannier, Corelight)
* Use clang-15 for lint CI job. (Benjamin Bannier, Corelight)
1.6.0-dev.133 | 2022-12-07 08:44:38 +0100
* Improve efficiency of `startsWith` for long inputs. (Benjamin Bannier, Corelight)
1.6.0-dev.131 | 2022-12-06 10:37:30 +0100
* Drop redundant leading `::` for C++ entities in script land. (Benjamin Bannier, Corelight)
* GH-1285: Remove implemented TODO in docs. (Benjamin Bannier, Corelight)
* Normalize names used in `__library_type`. (Benjamin Bannier, Corelight)
* Normalize any type `cxxID` set through C++ API. (Benjamin Bannier, Corelight)
* Normalize `&cxxname` values to be fully qualified. (Benjamin Bannier, Corelight)
* Expose function instead of macro for `hilti::debug`. (Benjamin Bannier, Corelight)
* Fix duplicate prefixing of `::` for prototypes generated for `&cxxname`. (Benjamin Bannier, Corelight)
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
1.6.0-dev.122 | 2022-12-05 09:31:26 +0100
* GH-1311: Validate that unit using `self.context()` declares `%context`. (Benjamin Bannier, Corelight)
1.6.0-dev.120 | 2022-12-05 09:30:26 +0100
* Explicitly type integer to boolean cast. (Benjamin Bannier, Corelight)
1.6.0-dev.118 | 2022-12-02 12:20:30 +0100
* cmake/FindGoldLinker: Do not use gold by default (Arne Welzel, Corelight)
* cmake/FindGoldLinker: Put -fuse-ld=gold before existing flags (Arne Welzel, Corelight)
1.6.0-dev.115 | 2022-12-01 13:59:34 +0100
* GH-1303: Fix incorrect offset computation in `advanceToNextData`. (Benjamin Bannier, Corelight)
1.6.0-dev.113 | 2022-11-18 11:48:35 +0100
* GH-1294: Add library function to parse an address from string or bytes. (Robin Sommer, Corelight)
## Parses an address from a string. The address can be in standard IPv4 or IPv6
## ASCII representation. The function raises ``InvalidArgument`` if the string
## could not be parsed.
public function parse_address(s: string) : addr;
## Parses an address from a bytes instance. The address can be in standard IPv4
## or IPv6 ASCII representation. The function raises ``InvalidArgument`` if the
## string could not be parsed.
public function parse_address(b: bytes) : addr;
* Support overloaded functions in documentation. (Robin Sommer, Corelight)
1.6.0-dev.110 | 2022-11-16 09:43:26 +0100
* GH-1299: Add validator for bitfield field ranges. (Benjamin Bannier, Corelight)
* Bump doctest to 2.4.9. (Benjamin Bannier, Corelight)
1.6.0-dev.107 | 2022-11-14 09:19:03 +0100
* Post synchronization-related debug messages in `spicy-verbose` stream. (Benjamin Bannier, Corelight)
* Log successful synchronization. (Benjamin Bannier, Corelight)
1.6.0-dev.104 | 2022-11-07 09:54:12 +0100
* Quote CMake string to ensure correct argument count. (Nic Boet)
* Fix Zeek docs build for case-sensitive filesystems. (Benjamin Bannier, Corelight)
1.6.0-dev.101 | 2022-10-27 12:59:41 +0200
* Fix fuzzer build for recently renamed spicy-tftp files. (Benjamin Bannier, Corelight)
* Make it possible to parameterize location of fuzzer lib. (Benjamin Bannier, Corelight)
* Update docs for changes in spicy-plugin and spicy-tftp. (Benjamin Bannier, Corelight)
1.6.0-dev.97 | 2022-10-27 08:47:59 +0200
* Downgrade required Flex version. (Johanna Amann, Corelight)
1.6.0-dev.95 | 2022-10-27 08:47:30 +0200
* GH-1273: Replace `Lambda` class. (Benjamin Bannier, Corelight)
1.6.0-dev.93 | 2022-10-27 08:46:22 +0200
* Show btest progress in CI. (Benjamin Bannier, Corelight)
* Remove configuration from JIT hash. (Benjamin Bannier, Corelight)
1.6.0-dev.90 | 2022-10-26 14:57:02 +0200
* Remove `hilti::rt::isDebugVersion()`. (Robin Sommer, Corelight)
* Remove `--optimize` flag. This was already a no-op, and just
confusing/misleading. (Robin Sommer, Corelight)
* Tie HLTOs files to the Spicy version that generated them. They
will now fail at load time if the current Spicy version doesn't
match what they were built with. (Robin Sommer, Corelight)
* Embed a C function into `hilti-rt` that encodes our version into
its name. If a host applications links against this, it will fail
to load if the runtime version changes without a rebuild. (Robin
Sommer, Corelight)
1.6.0-dev.81 | 2022-10-26 12:32:26 +0200
* Documentation updates. (Robin Sommer, Corelight)
- Document how to pull custom C++ code into Spicy.
- Adapt Zeek instructions for zkg template updates.
- Documentation updates for recent Zeek integration changes.
1.6.0-dev.71 | 2022-10-20 18:40:36 +0200
* Bump justrx. (Robin Sommer, Corelight)
1.6.0-dev.68 | 2022-10-05 08:50:05 +0200
* GH-1266: Fix wrong type for Spicy-side `self` expression. (Robin
Sommer, Corelight)
* GH-1261: Fix inability to access unit fields through `self` in
`&convert` expressions. (Robin Sommer, Corelight)
* GH-1267: Install only needed SafeInt header. (Benjamin Bannier,
Corelight)
* Disallow `self` as ID in declarations. (Robin Sommer, Corelight)
* Fix accidental lookup of empty IDs. (Robin Sommer, Corelight)
* Add helper script to diff two AST. (Robin Sommer, Corelight)
* Documentation updates. (Benjamin Bannier, Corelight)
- Update doc link to commits mailing list.
- Update version referenced in docs.
- Add doc link to releases packages for centos8-stream.
1.6.0-dev.55 | 2022-09-27 13:22:08 +0200
* GH-1267: Install only needed SafeInt header. (Benjamin Bannier, Corelight)
1.6.0-dev.53 | 2022-09-07 16:21:02 +0200
* Clarify that `%context` can only be used in top-level units. (Robin Sommer, Corelight)
1.6.0-dev.50 | 2022-08-22 16:33:10 +0200
* GH-1249: Allow combining `&eod` with `&until` or `&until-including`. (Benjamin Bannier, Corelight)
1.6.0-dev.48 | 2022-08-18 16:30:14 +0200
* GH-1251: When decoding bytes into a string using a given character
set, allow caller to control error handling. (Robin Sommer,
Corelight)
All methods taking a charset parameters now take an additional
enum selecting 1 of 3 possible error handling strategies in case a
character can't be decoded/represented: `STRICT` throws an error,
`IGNORE` skips the problematic character and proceeds with the
next, and `REPLACE` replaces the problematic character with a safe
substitute. `REPLACE` is the default everywhere now, so that by
default no errors are triggered.
This comes with an additional functional change for the ASCII
encoding: we now consistently sanitize characters that ASCII can't
represent when in `REPLACE`/`IGNORE` modes (and, hence, by
default), and trigger errors in `STRICT` mode. Previously, we'd
sometimes let them through, and never triggered any errors. This
also fixes a bug with the ASCII encoding sometimes turning a
non-printable character into multiple repeated substitutes.
* GH-1170: Fix contexts not allowing being passed `inout`. (Robin
Sommer, Corelight)
* GH-1143, GH-1220: Add coercion on assignment for optionals that
only differ in constness of their inner types. (Robin Sommer,
Corelight)
* GH-1115, GH-1196: Explicitly type temporary value used by
`&max_size` logic. (Robin Sommer, Corelight)
* GH-1250: Fix internal errors when seeing unsupported character
classes in regular expression. (Robin Sommer, Corelight)
* GH-1234, GH-1238: Fix assertions with anonymous struct
constructor. (Robin Sommer, Corelight)
* GH-1233: Reject key types for maps that can't be sorted. (Robin
Sommer, Corelight)
* GH-1230: Add coercion to default argument of `map::get`. (Robin
Sommer, Corelight)
* GH-1248: Fix `stop` for unbounded loop. (Robin Sommer, Corelight)
1.6.0-dev.28 | 2022-07-27 08:43:04 +0200
* Fix build with empty CMAKE_CXX_FLAGS_{DEBUG,RELEASE}. (Fabrice Fontaine)
1.6.0-dev.26 | 2022-07-26 11:15:20 +0200
* Fix validator for field `&default` expression types for constness. (Benjamin Bannier, Corelight)
1.6.0-dev.24 | 2022-07-26 11:14:06 +0200
* Merge branch 'topic/bbannier/doc-fixes' (Benjamin Bannier, Corelight)
* Clarify that `&until` consumes the delimiter. (Benjamin Bannier, Corelight)
* Add FAQ item on source locations. (Benjamin Bannier, Corelight)
* Add example for use of `?.`. (Benjamin Bannier, Corelight)
* Bump Zeek docs. (Benjamin Bannier, Corelight)
1.6.0-dev.19 | 2022-07-12 10:46:48 +0200
* GH-1236: Add support for adding link dependencies via `--cxx-link`. (Benjamin Bannier, Corelight)
1.6.0-dev.17 | 2022-07-08 12:12:02 +0200
* GH-1227: Fix code generation when a module's file could be imported through different means. (Robin Sommer, Corelight)
1.6.0-dev.15 | 2022-07-07 12:11:55 +0200
* GH-1231: Add special handling for potential `advance` failure in trial mode. (Benjamin Bannier, Corelight)
* Expose type `MissingData` in HILTI. (Benjamin Bannier, Corelight)
* GH-1231: Add failing test for #1231. (Benjamin Bannier, Corelight)
* Use default compiler on FreeBSD. (Benjamin Bannier, Corelight)
* Update CI to freebsd-13.1. (Benjamin Bannier, Corelight)
1.6.0-dev.8 | 2022-07-06 11:40:25 +0200
* GH-1240: Clarify docs on `SPICY_VERSION`. (Benjamin Bannier, Corelight)
1.6.0-dev.6 | 2022-07-05 14:22:35 +0200
* Add C keywords up to C23 to identifier sanitizer list. (Benjamin Bannier, Corelight)
* Add C++ keywords up to C++20 to identifier sanitizer list. (Benjamin Bannier, Corelight)
* Sort list of C++ keywords to sanitize. (Benjamin Bannier, Corelight)
* Fix doc check for renames in spicy-tftp. (Benjamin Bannier, Corelight)
* Update doc links for latest releases. (Benjamin Bannier, Corelight)
* Add placeholder section for 1.6 to NEWS. (Benjamin Bannier, Corelight)
1.5.0 | 2022-06-30 12:15:14 +0200
* Release 1.5.0 (Benjamin Bannier, Corelight)
* Explicitly request content write permission for create-release action. (Benjamin Bannier, Corelight)
* Update NEWS file. (Benjamin Bannier, Corelight)
1.5.0-dev.131 | 2022-06-30 10:15:01 +0200
* Extending content on zkg template. (Robin Sommer, Corelight)
1.5.0-dev.129 | 2022-06-28 12:08:25 +0200
* GH-1224: Fix import segfault. (Robin Sommer, Corelight)
1.5.0-dev.127 | 2022-06-27 14:35:05 +0200
* Bump doc links to 1.4.1. (Benjamin Bannier, Corelight)
* Fix links to Fedora artifacts for releases. (Benjamin Bannier, Corelight)
* GH-1205: Update Spicy docs for now being built into Zeek. (Benjamin Bannier, Corelight)
1.5.0-dev.123 | 2022-06-21 13:32:59 +0200
* GH-1217: Produce `hilti::rt::Bool` when casting to boolean. (Benjamin Bannier, Corelight)
1.5.0-dev.121 | 2022-06-20 14:39:05 +0200
* GH-1209: Provide error message to `%error` handler. (Robin Sommer, Corelight)
We now allow to optionally provide a string parameter with
`%error` that will receive the associated error message:
on %error(msg: string) { print msg; }
* GH-1206: Fix detection of recursive dependencies. (Robin Sommer, Corelight)
* Fix internal error with not fully resolved IDs. (Robin Sommer, Corelight)
1.5.0-dev.116 | 2022-06-17 11:01:21 +0200
* Avoid recomputing signatures. (Benjamin Bannier, Corelight)
* Avoid computing format string if debug stream not enabled. (Benjamin Bannier, Corelight)
* Use `unordered_set` to cache resolved types. (Benjamin Bannier, Corelight)
* Reorder instructions to avoid performing unneeded work. (Benjamin Bannier, Corelight)
* Avoid recomputing same information when resolving expressions. (Benjamin Bannier, Corelight)
* Prevent excessive copies. (Benjamin Bannier, Corelight)
* Small optimizations in `matchOverloads`. (Benjamin Bannier, Corelight)
* Remove excessive copying in `Operator::operands`. (Benjamin Bannier, Corelight)
1.5.0-dev.107 | 2022-06-17 08:43:12 +0200
* GH-1210: Prevent unnecessarily executable stack with GNU toolchain. (Benjamin Bannier, Corelight)
1.5.0-dev.105 | 2022-06-15 12:11:01 +0200
* Drop support for end-of-life Fedora 33/34, add support for Fedora 35/36. (Benjamin Bannier, Corelight)
* GH-1204: Remove potential use-after-move. (Benjamin Bannier, Corelight)
1.5.0-dev.102 | 2022-06-09 15:31:22 +0200
* Allow changing `DESTDIR` between configure and install time. (Benjamin Bannier, Corelight)
* GH-1183: Update docs for Discourse migration. (Benjamin Bannier, Corelight)
1.5.0-dev.99 | 2022-06-08 16:37:08 +0200
* GH-1201: Adjust removal of symlinks on install for `DESTDIR`. (Benjamin Bannier, Corelight)
1.5.0-dev.97 | 2022-06-08 16:33:24 +0200
* Disable building of benchmark target by default. (Benjamin Bannier, Corelight)
1.5.0-dev.95 | 2022-06-08 12:22:01 +0200
* Prevent too early integer overflow in `pow`. (Benjamin Bannier, Corelight)
* Make handling of sanitizer workarounds more granular. (Benjamin Bannier, Corelight)
1.5.0-dev.92 | 2022-06-07 11:53:18 +0200
* GH-44: Update docs for spicy-plugin rename `_Zeek::Spicy` -> `Zeek::Spicy`. (Benjamin Bannier, Corelight)
1.5.0-dev.90 | 2022-06-03 13:11:14 +0200
* GH-1187: Fix support for having multiple source modules of the
same name. (Robin Sommer, Corelight)
There was an underlying assumption in much of the code that a module
name would be globally unique, meaning it could be used as index into
maps and as seeds for C++ identifiers. However, while that's often the
case, the `import ... from ...` syntax allow pulling in multiple
modules all having the same name.
This change fixes that to track a module's "scope" (i.e., the part
after `from ...`) more thoroughly across various pieces..
* Fix test that could break because of file name clash. (Robin
Sommer, Corelight)
* Fix concatenation of IDs. (Robin Sommer, Corelight)
* Fix error message. (Robin Sommer, Corelight)
* Fix for HILTI printing of `import` statements. (Robin Sommer, Corelight)
* Dump ASTs with ``--dump-code`` as well. (Robin Sommer, Corelight)
1.5.0-dev.83 | 2022-06-02 09:43:07 +0200
* Bump 3rdparty/pathfind to latest snapshot. (Benjamin Bannier, Corelight)
* Bump 3rdparty/filesystem to 1.5.12. (Benjamin Bannier, Corelight)
* Switch to vendored nlohmann/json. (Benjamin Bannier, Corelight)
* Bump Zeek docs. (Benjamin Bannier, Corelight)
1.5.0-dev.78 | 2022-05-31 15:08:20 +0200
* Stop interning Locations. (Benjamin Bannier, Corelight)
* Avoid recomputing already known canonical ID. (Benjamin Bannier, Corelight)
* Simplify creating of refs when building scope. (Benjamin Bannier, Corelight)
* Speed up building of scopes. (Benjamin Bannier, Corelight)
1.5.0-dev.73 | 2022-05-31 15:07:35 +0200
* GH-1073: Switch to ECR images in documentation. (Benjamin Bannier, Corelight)
1.5.0-dev.71 | 2022-05-30 10:24:58 +0200
* GH-1050: Update location when entering most parser methods. (Benjamin Bannier, Corelight)
* Add builder method to set a new location. (Benjamin Bannier, Corelight)
1.5.0-dev.68 | 2022-05-30 10:03:28 +0200
* Suppress GCC warning about maybe uninitialized variable. (Benjamin Bannier, Corelight)
1.5.0-dev.66 | 2022-05-25 14:31:13 +0200
* GH-1174: Do not artificially suppress uses of `(u)int8` in some cases. (Benjamin Bannier, Corelight)
1.5.0-dev.64 | 2022-05-25 14:30:37 +0200
* GH-1190: Replace uses of deprecated `result_of` with `invoke_result`. (Benjamin Bannier, Corelight)
1.5.0-dev.62 | 2022-05-25 09:21:15 +0200
* Bump justrx. (Benjamin Bannier, Corelight)
* Do not build fiber benchmark target by default. (Benjamin Bannier, Corelight)
* Suppress GCC FP warnings around freeing of non-heap objects. (Benjamin Bannier, Corelight)
* Work around GCC incorrect state tracking involving std::optional<T*>` (Benjamin Bannier, Corelight)
1.5.0-dev.57 | 2022-05-18 14:16:12 +0200
* Use `rt::hashCombine` instead of handwritten hash combine. (Benjamin Bannier, Corelight)
* Drop contents of included codes from JIT state hash. (Benjamin Bannier, Corelight)
1.5.0-dev.54 | 2022-05-17 13:05:48 +0200
* GH-1179: Cap parallelism use for JIT background jobs. (Benjamin Bannier, Corelight)
* Move JIT background jobs to its own class. (Benjamin Bannier, Corelight)
1.5.0-dev.51 | 2022-05-16 16:37:15 +0200
* Avoid truncation when performing int64 range check. (Benjamin Bannier, Corelight)
1.5.0-dev.49 | 2022-05-16 16:27:50 +0200
* Fix signed/unsigned comparison warnings flagged by GCC. (Benjamin Bannier, Corelight)
* Remove unused variables. (Benjamin Bannier, Corelight)
* Fix linking of runtime test executables. (Benjamin Bannier, Corelight)
* Only check whether Zeek docs are up-to-date from cron builds. (Benjamin Bannier, Corelight)
1.5.0-dev.44 | 2022-05-10 09:01:16 +0200
* Minor edits to the Zeek Integration section (Christian Kreibich, Corelight)
1.5.0-dev.42 | 2022-05-06 11:43:58 +0200
* GH-1171: Remove freebsd11, add freebsd13. (Benjamin Bannier, Corelight)
1.5.0-dev.40 | 2022-05-04 14:25:52 +0200
* Enable building on armv7. (Benjamin Bannier, Corelight)
* Remove test assumption on whether `char` is signed. (Benjamin Bannier, Corelight)
1.5.0-dev.37 | 2022-05-03 15:54:47 +0200
* Add binary packaging mode. (Benjamin Bannier, Corelight)
1.5.0-dev.35 | 2022-04-26 11:28:26 +0200
* GH-1114: Update test comments to reflect actual behavior. (Benjamin Bannier, Corelight)
1.5.0-dev.33 | 2022-04-21 16:35:04 +0200
* GH-1164: Make compiler plugin initialization explicit. (Benjamin Bannier, Corelight)
* Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)
1.5.0-dev.30 | 2022-04-21 09:10:11 +0200
* Bump pre-commit hooks. (Benjamin Bannier, Corelight)
* Add fuzzing related files. (Benjamin Bannier, Corelight)
* Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)
1.5.0-dev.26 | 2022-04-14 07:53:47 +0200
* Fix warnings from LGTM. (Benjamin Bannier, Corelight)
* Bump LLVM in LGTM config. (Benjamin Bannier, Corelight)
* Allow building with gcc-8.3. (Benjamin Bannier, Corelight)
1.5.0-dev.22 | 2022-04-13 12:18:43 +0200
* Disable tests of lossy (u)int64_t-double conversion on 32bit. (honk)
* Adjust signature in test. (Benjamin Bannier, Corelight)
* Fix `chars_to_uint64` for 32bit platforms. (honk)
* Simplify implementation of `ast::ctor::SignedInteger`. (Benjamin Bannier, Corelight)
* Allow building on 32bit platforms. (Benjamin Bannier, Corelight)
1.5.0-dev.16 | 2022-04-12 11:03:38 +0200
* GH-1142: Fix clang-tidy `HeaderFilterRegex`. (Benjamin Bannier, Corelight)
1.5.0-dev.14 | 2022-04-11 11:34:39 +0200
* Fix incorrect exception for parse errors. (Benjamin Bannier, Corelight)
1.5.0-dev.12 | 2022-04-07 11:44:43 +0200
* GH-1134: Add support for synchronize-at and synchronize-after properties. (Benjamin Bannier, Corelight)
1.5.0-dev.10 | 2022-04-07 11:44:04 +0200
* GH-1080: Reject constant declarations at non-global scope. (Benjamin Bannier, Corelight)
* Fix test for `Map::value_type` stringification. (Benjamin Bannier, Corelight)
1.5.0-dev.7 | 2022-03-30 15:28:35 +0200
* GH-1154: Add stringificaton of `Map::value_type`. (Benjamin Bannier, Corelight)
1.5.0-dev.5 | 2022-03-22 14:21:43 +0100
* GH-1150: Preserve additional permissions from umask when generating HLTO files. (Benjamin Bannier, Corelight)
1.5.0-dev.3 | 2022-03-22 10:44:59 +0100
* Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)
* Fix doc gen script to allow for tuples with named fields in return type. (Benjamin Bannier, Corelight)
* Add placeholder section for 1.5 to NEWS. (Benjamin Bannier, Corelight)
1.4.0 | 2022-03-11 11:16:37 +0100
* Update NEWS. (Benjamin Bannier, Corelight)
* Bump docs for spicy-plugin. (Benjamin Bannier, Corelight)
1.4.0-dev.264 | 2022-03-10 11:22:08 +0100
* Add missing include. (Benjamin Bannier, Corelight)
* Allow teardown of resumables even if runtime is already gone. (Benjamin Bannier, Corelight)
* Fix clang-tidy warning about trivially copyable types in `stream.h`. (Benjamin Bannier, Corelight)
* Remove unused method `Chunk::isCompact`. (Benjamin Bannier, Corelight)
* Take gaps explicitly into account when recovering. (Benjamin Bannier, Corelight)
* Add a test for error recovery on gap. (Benjamin Bannier, Corelight)
* Use more general exception type for recoverable errors in Spicy. (Benjamin Bannier, Corelight)
* Add a chunk type for gaps. (Benjamin Bannier, Corelight)
1.4.0-dev.255 | 2022-03-04 09:01:36 +0100
* GH-1036: Support unit initialization through a struct constructor
expression. (Robin Sommer, Corelight)
This works now:
```
type X = unit {
a: uint8;
b: uint8;
};
...
local x: X = [$a = 1, $b = 2];
```
* GH-1112: Fix constness for dereferencing references. (Robin
Sommer, Corelight)
* Fix code markup in NEWS file. (Benjamin Bannier, Corelight)
1.4.0-dev.247 | 2022-02-23 09:43:39 +0100
* GH-1072: Disallow enum declarations with non-unique values. (Benjamin Bannier, Corelight)
It is unclear what code should be generated when requested to convert an
integer value to the following enum:
type E = enum {
A = 1,
B = 2,
C = 1,
};
For `1` we could produce either `E::A` or `E::C` here.
Instead of allowing that ambiguity we disallow enums with non-unique
values with this patch.
Closes #1072.
1.4.0-dev.245 | 2022-02-23 09:37:14 +0100
* GH-1043: Add validation rejecting untyped variable decl from struct initializer. (Benjamin Bannier, Corelight)
1.4.0-dev.243 | 2022-02-23 09:04:22 +0100
* GH-1096: Bump submodules and pre-commit hooks. (Benjamin Bannier, Corelight)
1.4.0-dev.236 | 2022-02-23 09:03:43 +0100
* GH-1131: Optimize out unused hooks related to synchronization. (Benjamin Bannier, Corelight)
1.4.0-dev.234 | 2022-02-23 09:03:08 +0100
* Fix styling of lists on RTD. (Benjamin Bannier, Corelight)
1.4.0-dev.232 | 2022-02-21 15:28:31 +0100
* Move clang-tidy run to its own CI job. (Benjamin Bannier, Corelight)
1.4.0-dev.230 | 2022-02-21 15:27:55 +0100
* GH-1120: Do not change a LHS to RHS when codegen'ing BuiltinFunctions. (Benjamin Bannier, Corelight)
1.4.0-dev.228 | 2022-02-18 19:12:51 +0100
* Add support for error recovery. (Benjamin Bannier, Corelight)
* Format unsigned integers with uppercase suffix. (Benjamin Bannier, Corelight)
* Do not prefix rendered parse errors with `parse error:`. (Benjamin Bannier, Corelight)
* Provide grammar function to return look-ahead literals for a given (Robin Sommer, Corelight)
production.
* Print grammar debug output only once tables have been computed. (Robin Sommer, Corelight)
* Remove duplicated and incorrect `default` case rendering to HILTI. (Benjamin Bannier, Corelight)
1.4.0-dev.180 | 2022-02-18 12:25:18 +0100
* Add step in `ci/run-ci` checking whether Zeek docs are up-to-date. (Benjamin Bannier, Corelight)
* Add doc build target to check whether docs are up-to-date. (Benjamin Bannier, Corelight)
* Remove unimplemented `deploy_docs` function in `ci/run-ci`. (Benjamin Bannier, Corelight)
1.4.0-dev.176 | 2022-02-17 14:14:06 +0100
* GH-1122: Make sure user-visible enums are represented by `int64_t`. (Benjamin Bannier, Corelight)
1.4.0-dev.174 | 2022-02-02 20:31:53 +0100
* Add a FAQ entry explicitly calling out Layer 2 analyzers. (Benjamin Bannier, Corelight)
1.4.0-dev.172 | 2022-01-26 12:05:12 +0100
* Redefine noop macro case so it always requires a semicolon. (Benjamin Bannier, Corelight)
* Be explicit about bugprone narrowing conversions. (Benjamin Bannier, Corelight)
* Fix diagnostics from clang-tidy's bugprone category. (Benjamin Bannier, Corelight)
* Fix diagnostics from clang-tidy's readability category. (Benjamin Bannier, Corelight)
* Fix diagnostics from clang-tidy's performance category. (Benjamin Bannier, Corelight)
* Fix diagnostics from clang-tidy's modernize category. (Benjamin Bannier, Corelight)
* Fix diagnostics from clang-tidy's cert category. (Benjamin Bannier, Corelight)
* Fix clang-tidy config. (Benjamin Bannier, Corelight)
We previously enabled subcategories incorrectly so that much fewer tests
than expected were run. We will fix the newly triggered diagnostics in
follow-up patches.
* Fix HeaderFilterRegex in clang-tidy config. (Benjamin Bannier, Corelight)
1.4.0-dev.162 | 2022-01-25 12:22:48 +0100
* Fix documentation link to TFTP sample capture files. (Benjamin Bannier, Corelight)
* Update documentation for removed centos releases. (Benjamin Bannier, Corelight)
This is a documentation fixup commit for 55d7775de25c8 where we removed
supported for EOL centos releases.
1.4.0-dev.159 | 2022-01-20 09:38:49 +0100
* GH-1084: Fix typing of map/vector index operators. (Robin Sommer,
Corelight)
* GH-1079: Fix import of global constants. (Robin Sommer, Corelight)
1.4.0-dev.153 | 2022-01-13 17:12:31 +0100
* Remove support for EOL centos releases, add centos-stream-8. (Benjamin Bannier, Corelight)
1.4.0-dev.151 | 2022-01-13 08:18:29 +0100
* GH-1092: Strip installation artifacts. (Benjamin Bannier, Corelight)
This removes debug symbols from installation artifacts. This leads to a
marked reduction in their size.
* Switch configure to use Release mode by default. (Benjamin Bannier, Corelight)
1.4.0-dev.148 | 2022-01-13 08:17:53 +0100
* Use libc++ instead of libstdc++ on debian-9. (Benjamin Bannier, Corelight)
In libstdc++-10-dev-10.3.0-13 Debian includes the upstream GCC patch
2b2d97fc545 without also including at least 4f032929ac9. This leads to
errors e.g., triggered from nlohmann/json were `&noexcept` attributes
are used inconsistently.
Since debian-9 should only receive security updates at this point it is
unclear to me how such a feature change ended up being backported. At
this point it is not clear to me whether it will see another patch
release. For these reasons we switch the build over to use libc++
instead which comes from the LLVM repo and should still receive reliable
updates.
1.4.0-dev.146 | 2022-01-08 10:02:10 +0100
* GH-1076: Do not pass errors inside sink units back up to parent.
(Robin Sommer, Corelight)
* GH-1077: Fix self-recursive units. (Robin Sommer, Corelight)
* GH-1069: Allow implicit C++ conversion from value reference to
strong reference. (Robin Sommer, Corelight)
* Fix typos in the source and docs. (Benjamin Bannier, Corelight)
1.4.0-dev.137 | 2022-01-07 10:41:29 +0100
* Fix typos in the source and docs. (Benjamin Bannier, Corelight)
1.4.0-dev.135 | 2021-12-22 11:04:51 +0100
* Update docs for Zeek plugin functions with recent changes. (Robin
Sommer, Corelight)
1.4.0-dev.133 | 2021-12-21 11:45:51 +0100
* GH-1088: Ensure that field hooks for lists see correct offset. (Benjamin Bannier, Corelight)
* Remove stray baselines. (Benjamin Bannier, Corelight)
* Lint all sh scripts with checkbashisms. (Benjamin Bannier, Corelight)
1.4.0-dev.129 | 2021-12-17 12:30:33 +0100
* GH-1087: Make `offset` return correct value even before parsing of field. (Benjamin Bannier, Corelight)
1.4.0-dev.127 | 2021-12-16 11:46:32 +0100
* GH-1045: Decouple whether Spicy was build with ccache from ccache use during JIT. (Benjamin Bannier, Corelight)
1.4.0-dev.124 | 2021-12-15 16:54:10 +0100
* Remove unneeded `%random-access` in tests and docs. (Benjamin Bannier, Corelight)
* Add validation catching use of deprecated `%random-access`. (Benjamin Bannier, Corelight)
* Enable random-access by default. (Benjamin Bannier, Corelight)
* Generate branch for `%random-access`-dependent trimming. (Benjamin Bannier, Corelight)
1.4.0-dev.117 | 2021-12-15 16:13:53 +0100
* GH-1068: Fix syntax for Spicy `struct` declaration. (Benjamin Bannier, Corelight)
1.4.0-dev.115 | 2021-12-15 15:51:26 +0100
* Fix typos in variable documentation. (Robin Sommer, Corelight)
* Document visibility for globals. (Benjamin Bannier, Corelight)
1.4.0-dev.112 | 2021-12-13 10:46:37 +0100
* Fix zeek_plugin CI task for broken apart zeek/spicy-analyzers. (Benjamin Bannier, Corelight)
1.4.0-dev.110 | 2021-12-13 10:46:04 +0100
* Show log for configure failures. (Benjamin Bannier)
1.4.0-dev.108 | 2021-12-13 10:43:52 +0100
* Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)
* Fix docs build for split-apart zeek/spicy-analyzers. (Benjamin Bannier, Corelight)
That analyzer repo has been broken up into specific analyzer repos. Fix
scripts and docs on our side for that.
1.4.0-dev.105 | 2021-12-10 09:50:54 +0100
* Fix lints from cmake-lint. (Benjamin Bannier, Corelight)
* Reformat CMake files with cmake-format. (Benjamin Bannier, Corelight)
1.4.0-dev.102 | 2021-12-09 11:24:39 +0100
* Avoid calculating AST dump if we do not output it. (Benjamin Bannier, Corelight)
* Fix accidental calculation of immediately discarded debug output. (Benjamin Bannier, Corelight)
1.4.0-dev.99 | 2021-12-09 09:30:35 +0100
* GH-1056: Fix constness check for struct parameters. (Robin Sommer,
Corelight)
* GH-1057: Fix code generator to emit types correctly for transitive
dependencies. (Robin Sommer, Corelight)
1.4.0-dev.95 | 2021-12-07 11:23:17 +0100
* Fix dependency tracking of spicy-build CMake target. (Benjamin Bannier, Corelight)
* Fix outputs of CMake command to generate precompiled headers. (Benjamin Bannier, Corelight)
1.4.0-dev.92 | 2021-12-03 11:34:17 +0100
* Fix a typo in container Github action filter. (Benjamin Bannier, Corelight)
1.4.0-dev.90 | 2021-12-03 08:43:55 +0100
* GH-1059: Allow to use unit contexts in units with parameters. (Benjamin Bannier, Corelight)
* Bump 3rdparty/filesystem to v1.5.10. (Benjamin Bannier, Corelight)
* Bump gitlint pre-commit hooks to v0.17.0. (Benjamin Bannier, Corelight)
* Bump 3rdparty/benchmark to v1.6.0. (Benjamin Bannier, Corelight)
1.4.0-dev.85 | 2021-12-02 12:22:48 +0100
* Do not by default run Docker workflow for PRs. (Benjamin Bannier, Corelight)
* Bump published container to zeek-4.0.4. (Benjamin Bannier, Corelight)
* Use releases instead of HEAD for packages in published container image. (Benjamin Bannier, Corelight)
* Also push container image to ECR. (Benjamin Bannier, Corelight)
* Unify Docker build jobs for tags and for latest. (Benjamin Bannier, Corelight)
* Use variable to store Docker GH action build args. (Benjamin Bannier, Corelight)
* Also build Docker image in PR, but do not publish it. (Benjamin Bannier, Corelight)
* Split build and push step for Docker GH action. (Benjamin Bannier, Corelight)
* Simplify GH action Docker build pipeline. (Benjamin Bannier, Corelight)
* Checkout submodules via normal checkout action. (Benjamin Bannier, Corelight)
1.4.0-dev.74 | 2021-11-19 17:31:51 +0100
* Update documentation for the Zeek plugin's new `$packet` feature.
(Robin Sommer, Corelight)
1.4.0-dev.72 | 2021-11-16 16:56:18 +0100
* Fix potentially use-after-free in stringification of `stream::View`. (Benjamin Bannier, Corelight)
This is a fixup commit for dbc4c6e0d8e where we missed a couple
potential use-after-frees.
1.4.0-dev.70 | 2021-11-12 09:36:24 +0100
* GH-106: Bump zkg in published Docker images. (Benjamin Bannier, Corelight)
1.4.0-dev.68 | 2021-11-12 09:35:59 +0100
* Refresh package docs with `doc/scripts/autogen-zeek-docs`. (Benjamin Bannier, Corelight)
* Fix lints in `doc/scripts/autogen-zeek-docs.` (Benjamin Bannier, Corelight)
* Fix `doc/scripts/autogen-zeek-docs` for changed layout of spicy-analyzers repo. (Benjamin Bannier, Corelight)
1.4.0-dev.64 | 2021-11-12 09:35:38 +0100
* Use clang-format pre-commit hook which installs its own `clang-format` binary. (Benjamin Bannier, Corelight)
1.4.0-dev.62 | 2021-11-09 10:21:28 +0100
* Fix potentially use-after-free in stringification of `stream::View`. (Benjamin Bannier, Corelight)
1.4.0-dev.60 | 2021-11-09 10:19:53 +0100
* Document structs. (Benjamin Bannier, Corelight)
* GH-842: Add Spicy support for struct initialization. (Benjamin Bannier, Corelight)
* Add basic tests for Spicy `struct` types. (Benjamin Bannier, Corelight)
1.4.0-dev.56 | 2021-11-05 14:55:09 +0100
* Use compiler launcher from ccache if one was used there. (Benjamin Bannier, Corelight)
* Use atomic operations to create temporary files during JIT. (Benjamin Bannier, Corelight)
* Use explicit guard controlling deletion of temporary files in JIT. (Benjamin Bannier, Corelight)
* Explicitly enforce that a module is only registered once. (Benjamin Bannier, Corelight)
* Do not delete temporary files if requested. (Benjamin Bannier, Corelight)
* Fix extension of precompiled headers. (Benjamin Bannier, Corelight)
* Cache JIT results in CI. (Benjamin Bannier, Corelight)
* Allow use of ccache during JIT. (Benjamin Bannier, Corelight)
* Include HILTI config in JIT hash. (Benjamin Bannier, Corelight)
* Extend hashCombine so it can combine more than two hashes. (Benjamin Bannier, Corelight)
* Use deterministic locations for files created during JIT. (Benjamin Bannier, Corelight)
* Avoid unneeded parameter copy. (Benjamin Bannier, Corelight)
* Ensure cleanup of temporary driver HLTO files on errors. (Benjamin Bannier, Corelight)
* Compute module linker scope variable from module contents. (Benjamin Bannier, Corelight)
* Remove unused `created` field in linker version. (Benjamin Bannier, Corelight)
1.4.0-dev.40 | 2021-11-04 13:03:09 +0100
* Fix build for RTD. (Benjamin Bannier, Corelight)
1.4.0-dev.38 | 2021-11-04 12:35:22 +0100
* Make Spicy prefix visible for CI jobs testing spicy-[plugin|analyzers]. (Benjamin Bannier, Corelight)
1.4.0-dev.36 | 2021-11-02 13:07:00 +0100
* Show diff if pre-commit fails in CI. (Benjamin Bannier, Corelight)
1.4.0-dev.34 | 2021-11-01 16:32:40 +0100
* GH-1028: Add TODO to docs on unit initialization via list coercion. (Benjamin Bannier, Corelight)
1.4.0-dev.32 | 2021-11-01 13:19:49 +0100
* GH-1030: Make sure types required for globals are declared before being used. (Benjamin Bannier, Corelight)
* Fix bug in emission of prioritized types. (Benjamin Bannier, Corelight)
* Fully qualify some identifiers when emitting C++. (Benjamin Bannier, Corelight)
* Do not emit `__init_globals` if there are no globals. (Benjamin Bannier, Corelight)
1.4.0-dev.27 | 2021-10-26 15:43:54 +0200
* GH-994: Model types in `set` and `map` `in` operators. (Benjamin Bannier, Corelight)
1.4.0-dev.25 | 2021-10-25 16:30:35 +0200
* Prevent exception if cache directory is not readable. (Benjamin Bannier, Corelight)
1.4.0-dev.23 | 2021-10-25 16:29:53 +0200
* Fix issues flagged by shellcheck in `./configure`. (Benjamin Bannier, Corelight)
* Propagate failure from `cmake` up to `./configure`. (Benjamin Bannier, Corelight)
1.4.0-dev.20 | 2021-10-22 11:40:46 +0200
* Make `-D ast-codegen` work even without `-j`. (Benjamin Bannier, Corelight)
1.4.0-dev.18 | 2021-10-22 11:39:45 +0200
* Set `HILTI_DEBUG` to known value for tests. (Benjamin Bannier, Corelight)
* GH-981: Emit comments for enabled features. (Benjamin Bannier, Corelight)
1.4.0-dev.15 | 2021-10-20 11:37:36 +0200
* Fix computation of dependent features if multiple features are involved. (Benjamin Bannier, Corelight)
* Remove workaround always emitting parser functions. (Benjamin Bannier, Corelight)
* Register parsers depending on feature. (Benjamin Bannier, Corelight)
* Do not remove unused functions if they are required by type features. (Benjamin Bannier, Corelight)
* Allow to guard on a list of features. (Benjamin Bannier, Corelight)
* Constant fold logical operations on booleans. (Benjamin Bannier, Corelight)
* Factor out getting of bool literals in optimizer. (Benjamin Bannier, Corelight)
* Always emit filter functionality. (Benjamin Bannier, Corelight)
* Change `connect_mime_type` to consider public units and same linker scope units. (Benjamin Bannier, Corelight)
* Allow registration of non-public parsers. (Benjamin Bannier, Corelight)
* Enable sink support for non-public units as well. (Benjamin Bannier, Corelight)
1.4.0-dev.3 | 2021-10-20 08:10:39 +0200
* Fix internal link in docs. (Benjamin Bannier, Corelight)
* Remove fedora32 from docs, add fedora34. (Benjamin Bannier, Corelight)
* Add placeholder section for 1.4 to NEWS. (Benjamin Bannier, Corelight)
1.3.0 | 2021-10-18 16:23:34 +0200
* Release 1.3.0.
1.3.0-dev.133 | 2021-10-18 16:16:23 +0200
* GH-934: Allow `$$` in place of `self` in unit convert attributes. (Benjamin Bannier, Corelight)
* Add stringification for Keywords. (Benjamin Bannier, Corelight)
1.3.0-dev.130 | 2021-10-12 18:45:17 +0200
* Run pip3 as superuser in macos CI. (Benjamin Bannier, Corelight)
1.3.0-dev.128 | 2021-10-12 12:12:02 +0200
* Fix GCC 11 build error for set iterators (Christian Kreibich, Corelight)
* Modernize CI: drop Fedora 32 (EOL since May), add Fedora 34 (Christian Kreibich, Corelight)
1.3.0-dev.125 | 2021-10-02 08:55:47 +0200
* GH-46: Switch Spicy and HILTI parsers to use variants. (Benjamin Bannier, Corelight)
* GH-1009: Fix overly generous generation of type information. (Robin Sommer, Corelight)
* GH-1013: Split validation into two passes. (Robin Sommer, Corelight)
1.3.0-dev.113 | 2021-09-24 13:02:22 +0200
* Document release artifaces for debian-11. (Benjamin Bannier, Corelight)
* Switch release version referenced in docs to v1.2.1. (Benjamin Bannier, Corelight)
1.3.0-dev.110 | 2021-09-23 18:22:44 +0200
* Remove duplicate rule diagnosed by Bison. (Benjamin Bannier, Corelight)
* Avoid rendering modules if the debug stream is disabled. (Benjamin Bannier, Corelight)
1.3.0-dev.107 | 2021-09-23 15:39:31 +0200
* Remove unneeded copies. (Benjamin Bannier, Corelight)
* Remove unneeded unwrapping of type. (Benjamin Bannier, Corelight)
* Cleanup optimizer after AST rework. (Benjamin Bannier, Corelight)
* Fix execution order of hooks in test. (Benjamin Bannier, Corelight)
* Also collect type uses from function declarations. (Benjamin Bannier, Corelight)
1.3.0-dev.101 | 2021-09-21 12:10:25 +0200
* Major internal overhaul of the AST infrastructure. (Robin Sommer,
Corelight)
1.3.0-dev.100 | 2021-09-17 13:18:50 +0200
* Fix lints in rst files. (Benjamin Bannier, Corelight)
* Bump pre-commit-hooks. (Benjamin Bannier, Corelight)
1.3.0-dev.97 | 2021-09-16 10:48:42 +0200
* Use safe integers for runtime size functions. (Benjamin Bannier, Corelight)
In fd5c53cb8ffd1a18aadb7d21b927bda801e264c8 we changed the size
functions of runtime containers to use safe integers. This patch adjusts
the remaining size functions to use safe integers as well.
1.3.0-dev.95 | 2021-09-15 11:46:16 +0200
* More fixes for building on ARM64. (Benjamin Bannier, Corelight)
In 89d3295f1363d8d07eda5c711740b18f31483d88 we adjusted some
preprocessor code to enable building on arm64. That patch missed an
update of another instance to the same conditional which we fix with
this patch.
Some ARM64 platforms do not seem to reliably set the `__arm64__` define,
but instead `__aarch64__`. We now allow that as an alternative.
1.3.0-dev.93 | 2021-09-15 11:45:50 +0200
* Fix example links for recent reorg in zeek/spicy-analyzers. (Benjamin Bannier, Corelight)
1.3.0-dev.91 | 2021-09-14 14:10:28 +0200
* Fix parsers for bison-3.8.1. (Benjamin Bannier, Corelight)
1.3.0-dev.89 | 2021-09-13 15:13:00 +0200
* GH-999: Unify `transform` and `filter` functions. (Benjamin Bannier, Corelight)
* Add `insert` with hint to `Vector` and `Set`. (Benjamin Bannier, Corelight)
1.3.0-dev.86 | 2021-09-09 10:38:22 +0200
* Precompile `libspicy.h`. (Benjamin Bannier, Corelight)
1.3.0-dev.84 | 2021-09-09 10:36:23 +0200
* GH-941: Allow use of units with all defaulted parameters as entry points. (Benjamin Bannier, Corelight)
1.3.0-dev.82 | 2021-09-09 10:33:14 +0200
* Disable leak tests if `ASAN_OPTIONS` are already set. (Benjamin Bannier, Corelight)
* Correctly handle lookups for NULL library symbols. (Benjamin Bannier, Corelight)
1.3.0-dev.79 | 2021-09-09 10:32:29 +0200
* GH-939: Document initializers for unit variables. (Benjamin Bannier, Corelight)
1.3.0-dev.77 | 2021-09-08 10:46:32 +0200
* Make it possible to build on arm64. (Benjamin Bannier, Corelight)
Since we currently do not test on arm64 we do not advertize it as a
supported platform.
1.3.0-dev.75 | 2021-09-07 08:39:58 +0200
* GH-982: Model feature constants with actual constants. (Benjamin Bannier, Corelight)
1.3.0-dev.73 | 2021-09-06 09:19:16 +0200
* GH-988: Use safe integer for runtime container sizes. (Benjamin Bannier, Corelight)
1.3.0-dev.71 | 2021-09-03 14:36:35 +0200
* Add collection logging to the optimizer. (Benjamin Bannier, Corelight)
* Rename global optimizer to optimizer everywhere. (Benjamin Bannier, Corelight)
1.3.0-dev.68 | 2021-08-31 10:11:16 +0200
* Disable constant folding for anything but feature variables. (Benjamin Bannier, Corelight)
1.3.0-dev.66 | 2021-08-31 10:10:42 +0200
* GH-979: Properly track use of uninitialized enum values in units. (Benjamin Bannier, Corelight)
1.3.0-dev.64 | 2021-08-30 16:23:13 +0200
* Add optimizer pass removing unused members. (Benjamin Bannier, Corelight)
* Always emit `__sink` field for units with `%mime-type`. (Benjamin Bannier, Corelight)
* Mark unit field `__offsets` as `&always-emit`. (Benjamin Bannier, Corelight)
* Optimize away unused sink functionality. (Benjamin Bannier, Corelight)
* Add annotations for sink functionality. (Benjamin Bannier, Corelight)
* Optimize away unused filter functionality. (Benjamin Bannier, Corelight)
* Add annotations for filter functionality. (Benjamin Bannier, Corelight)
* Collect feature requirements from function call parameters. (Benjamin Bannier, Corelight)
* Add support for HILTI function parameter attributes. (Benjamin Bannier, Corelight)
* Use `&needed-by-feature` to annotate feature-dependent fields. (Benjamin Bannier, Corelight)
* Add an optimizer helper function to get the innermost type. (Benjamin Bannier, Corelight)
* Fix generated code to always invoke parsing of subunits. (Benjamin Bannier, Corelight)
1.3.0-dev.51 | 2021-08-26 15:34:35 +0200
* Wrap long line in NEWS file. (Benjamin Bannier, Corelight)
1.3.0-dev.50 | 2021-08-26 12:38:15 +0200
* Fix C++ snippets in "Custom Host Application" docs. (Benjamin Bannier, Corelight)
1.3.0-dev.48 | 2021-08-24 15:22:43 +0200
* Add optimizer pass removing unused `%random-access` features. (Benjamin Bannier, Corelight)
* Annotate parser fields related to `%random-access`. (Benjamin Bannier, Corelight)
* Annotate generated parser code with feature flags. (Benjamin Bannier, Corelight)
* Add function optimizer pass removing orphaned method implementations. (Benjamin Bannier, Corelight)
* Add an optimizer pass performing basic constant folding. (Benjamin Bannier, Corelight)
* Add mutator for `statement::If` removing `else` branch. (Benjamin Bannier, Corelight)
* Prepare `statement::Block` for optimizer edits. (Benjamin Bannier, Corelight)
* Add accessor for resolved IDs of expressions. (Benjamin Bannier, Corelight)
* Fix type visitor to properly handle wrapped types. (Benjamin Bannier, Corelight)
* Check environment variable `HILTI_OPTIMIZER_PASSES` for optimizer passes to enable. (Benjamin Bannier, Corelight)
* Change optimizer tests to compare HILTI code instead of ASTs. (Benjamin Bannier, Corelight)
* Change `-p` to print source after optimizations. (Benjamin Bannier, Corelight)
* Clear module declaration cache after optimizations. (Benjamin Bannier, Corelight)
* Add a visitor base class for the global optimizer. (Benjamin Bannier, Corelight)
* Change return type of `AttributeSet::add` so it always returns some set. (Benjamin Bannier, Corelight)
* Remove outdated TODO. (Benjamin Bannier, Corelight)
* Prevent unneeded copy of return value. (Benjamin Bannier, Corelight)
1.3.0-dev.30 | 2021-08-23 09:36:24 +0200
* Hardcode DockerHub org we publish images for in GH workflow. (Benjamin Bannier, Corelight)
1.3.0-dev.28 | 2021-08-19 10:38:45 +0200
* Add FAQ item for parsing TCP conversations. (Benjamin Bannier, Corelight)
1.3.0-dev.26 | 2021-08-06 14:52:54 +0200
* Make test type public to prevent it being optimized out. (Benjamin Bannier, Corelight)
* Add optimizer pass to remove unused types. (Benjamin Bannier, Corelight)
* Clear preserved nodes of modules after optimizing. (Benjamin Bannier, Corelight)
* Factor out function to remove an AST node. (Benjamin Bannier, Corelight)
* Remove unused state in global optimizer. (Benjamin Bannier, Corelight)
* Rename visitor for functions in global optimizer. (Benjamin Bannier, Corelight)
* Move optimizer collection state into visitor. (Benjamin Bannier, Corelight)
1.3.0-dev.18 | 2021-08-05 17:26:57 +0200
* GH-970: Use publicly accessible links to Docker images. (Benjamin Bannier, Corelight)
1.3.0-dev.16 | 2021-08-05 11:24:08 +0200
* Implement removal of unused member functions in global optimizer. (Benjamin Bannier, Corelight)
* Explicitly mark parse functions of filters as `&always-emit`. (Benjamin Bannier, Corelight)
* Factor out functions to compute IDs for global optimizer. (Benjamin Bannier, Corelight)
* Remove unused function `hilti::type::Struct::ids`. (Benjamin Bannier, Corelight)
1.3.0-dev.11 | 2021-08-05 11:23:24 +0200
* Fix flaky test `util.memory_statistics` in hilti-rt-tests. (Benjamin Bannier, Corelight)
* Execute correct test target in validate_release_tarball Cirrus CI task. (Benjamin Bannier, Corelight)
* Fix validate_release_tarball Cirrus CI task. (Benjamin Bannier, Corelight)
* Only consider typical release tags when checking whether we are on a release tag. (Benjamin Bannier, Corelight)
* GH-960: Swallow Git error messages during CMake if run from tarball. (Benjamin Bannier, Corelight)
1.3.0-dev.5 | 2021-08-03 11:04:26 +0200
* GH-966: Emit linker glue type definitions for vector fields. (Benjamin Bannier, Corelight)
1.3.0-dev.3 | 2021-08-02 16:40:30 +0200
* Directly maintain VERSION file in repo. (Benjamin Bannier, Corelight)
1.2.0 | 2021-07-29 16:05:44 +0200
* Release 1.2.0. (Benjamin Bannier, Corelight)
1.2.0-dev.110 | 2021-07-29 15:01:46 +0200
* GH-956: Fix issues around generating version information. (Benjamin Bannier, Corelight)
1.2.0-dev.105 | 2021-07-29 09:49:24 +0200
* GH-961: Make detection of current executable more robust. (Benjamin Bannier, Corelight)
1.2.0-dev.103 | 2021-07-29 09:46:05 +0200
* Make it possible to build Spicy as a CMake subproject. (Benjamin Bannier, Corelight)
1.2.0-dev.101 | 2021-07-23 16:47:13 +0200
* Add Debian Bullseye (11) to CI. (Matthias Vallentin)
* Update plugin docs for build system changes. (Robin Sommer,
Corelight)
1.2.0-dev.94 | 2021-07-19 18:59:32 +0200
* CI tweaks. (Robin Sommer, Corelight)
1.2.0-dev.92 | 2021-07-19 10:05:36 +0200
* Remove spicy-plugin and spicy-analyzers submodules, along with the
`--build-zeek-plugin` configure option. Spicy no longer support
building them in-tree. This comes along with removing almost all
Zeek-related pieces from Spicy proper (except for CI testing
integration with the external Zeek plugin).
* Add environment variable `HILTI_CXX_INCLUDE_DIRS` to specify
additional C++ include directories when compiling generated code.
(Robin Sommer, Corelight)
1.2.0-dev.75 | 2021-07-14 11:07:14 +0200
* GH-940: Add runtime check for parsing progress during loops.
(Robin Sommer, Corelight)
1.2.0-dev.71 | 2021-07-14 11:02:28 +0200
* Add unique "linker scope" identifier to HLTO files. In Spicy, we
use now store this linker scope along with the parsers when they
get registered with the runtime. (Robin Sommer, Corelight)
* GH-952: Switch the Cirrus macOS images to `*-xcode` and (still)
update Brew for Catalina. (Robin Sommer, Corelight)
* Make Spicy version available to subprojects. (Robin Sommer, Corelight)
* Bump Zeek submodules. (Robin Sommer, Corelight)
* Fix missing documentation update. (Robin Sommer, Corelight)
* Perform optimizations in a loop. This patch changes the optimizer
to now run a collect/prune loop until no further AST changes are
made. (Benjamin Bannier, Corelight)
* Disable clang-tidy `NewDeleteLeaks` check. (Benjamin Bannier, Corelight)
* Reduce memory load for clang-tidy CI step. (Benjamin Bannier, Corelight)
* Use clang-tidy-12 in CI. (Benjamin Bannier, Corelight)
* Fix accidental truncating conversion in integer code. (Benjamin
Bannier, Corelight)
1.2.0-dev.55 | 2021-07-09 08:34:26 +0200
* Perform optimizations in a loop. (Benjamin Bannier, Corelight)
1.2.0-dev.53 | 2021-07-08 12:36:43 +0200
* Disable clang-tidy `NewDeleteLeaks` check. (Benjamin Bannier, Corelight)
* Reduce memory load for clang-tidy CI step. (Benjamin Bannier, Corelight)
* Use clang-tidy-12 in CI. (Benjamin Bannier, Corelight)
* Remove accidental truncating conversion. (Benjamin Bannier, Corelight)
* Document for clang-tidy that field can never be unset. (Benjamin Bannier, Corelight)
* Silence warning on unused variable in generated code. (Benjamin Bannier, Corelight)
* Do not drop errors when move-constructing Nodes. (Benjamin Bannier, Corelight)
* Fix range of check. (Benjamin Bannier, Corelight)
1.2.0-dev.43 | 2021-07-05 09:18:12 +0200
* GH-918: Remove workaround for #918. (Benjamin Bannier, Corelight)
* GH-918: GH-949: Use forward declarations in linker metadata. (Benjamin Bannier, Corelight)
* Add registered cxx_includes to linker outputs. (Benjamin Bannier, Corelight)
1.2.0-dev.38 | 2021-06-29 09:33:19 +0200
* Avoid name collision when codegen'ing struct element initializers. (Benjamin Bannier, Corelight)
1.2.0-dev.36 | 2021-06-29 09:31:59 +0200
* Prefer VERSION file over git information. (Benjamin Bannier, Corelight)
* Introduce top-level `spicy-*/` directory in release tarball. (Benjamin Bannier, Corelight)
1.2.0-dev.33 | 2021-06-28 09:23:54 +0200
* Add optimizer code removing unused function declarations. (Benjamin Bannier, Corelight)
1.2.0-dev.30 | 2021-06-17 10:52:32 +0200
* Remove unused variable lookup. (Benjamin Bannier, Corelight)
* Use prefiltered candidate list when resolving operators. (Benjamin Bannier, Corelight)
* Make `Module::declarations` more robust. (Benjamin Bannier, Corelight)
* Fix Scope rendering to not access expired references. (Benjamin Bannier, Corelight)
* Whitelist stringification functions in optimizer collection phase. (Benjamin Bannier, Corelight)
* Implement removal of unused global hooks. (Benjamin Bannier, Corelight)
* Bump spicy-plugin and spicy-analyzer submodules. (Benjamin Bannier, Corelight)
1.2.0-dev.17 | 2021-05-31 12:36:27 +0200
* Add tests for global optimizations. (Benjamin Bannier, Corelight)
* Add optimizer pass removing unimplemented member functions.
This patch introduces a global pass triggered after all individual input
ASTs have been finalized, but before we generate any C++ code. We then
strip out any unimplemented member functions (typically Spicy hooks),
both their definitions as well as their uses.
In order to correctly handle previously generated C++ files which might
have been generated with different optimization settings, we disallow
optimizations if we detect that a C++ input file was generated by us. (Benjamin Bannier, Corelight)
* Fix computation of unset locations. (Benjamin Bannier, Corelight)
* Detect presence of sink hooks at compile time.
Sink hooks are by default generated for any `public` unit. With the
introduction of an optimizer pass to remove unused hooks we might now
remove them and need to detect their presence when e.g., registering a
parser.
This patch moves from these instances from branching on whether a unit
is `public` to directly detecting the presence of the hooks. (Benjamin Bannier, Corelight)
* Introduce dedicated codegen step in Driver. (Benjamin Bannier, Corelight)
* Fix test to correctly log output.
We previously logged the output of this test to a file we never
examined. (Benjamin Bannier, Corelight)
* Autoupdate pre-commit hooks. (Benjamin Bannier, Corelight)
1.2.0-dev.9 | 2021-05-31 12:11:30 +0200
* GH-913: Add support for unit switch &parse-at and &parse-from attributes. (Benjamin Bannier, Corelight)
* Add validation of unit switch attributes. (Benjamin Bannier, Corelight)
1.2.0-dev.5 | 2021-05-31 12:10:44 +0200
* Make only draft releases from automation. (Benjamin Bannier, Corelight)
1.2.0-dev.3 | 2021-05-31 12:08:12 +0200
* Adjust processes used to build Docker image. (Benjamin Bannier, Corelight)
* Update NEWS with 1.2.0 placeholder sections. (Benjamin Bannier, Corelight)
* Bump spicy-plugin and spicy-analyzer submodules. (Benjamin Bannier, Corelight)
* Update NEWS. (Benjamin Bannier, Corelight)
1.1.0-dev.104 | 2021-05-20 10:32:54 +0200
* GH-920: Track patching of Bison files through proper dependencies.
(Benjamin Bannier, Corelight)
1.1.0-dev.102 | 2021-05-20 10:31:25 +0200
* GH-917: Default-initialize forwarding fields without type arguments.
(Benjamin Bannier, Corelight)
1.1.0-dev.99 | 2021-05-20 09:22:52 +0200
* Fix missing operator<< for enums when generating debug code.
(Robin Sommer, Corelight)
1.1.0-dev.95 | 2021-05-18 13:26:32 +0200
* Bump version. (Benjamin Bannier, Corelight)
* Draft 1.1 release notes. (Robin Sommer, Corelight)
1.1.0-dev.91 | 2021-05-17 14:29:24 +0200
* Remove redundant version computation. (Benjamin Bannier, Corelight)
* Do not update unchanged version files. (Benjamin Bannier, Corelight)
* Remove redundant indention in generated VERSION file. (Benjamin Bannier, Corelight)
1.1.0-dev.87 | 2021-05-12 18:38:40 +0200
* Make sure container sizes are runtime integers. (Benjamin Bannier, Corelight)
1.1.0-dev.85 | 2021-05-11 16:19:12 +0200
* Add functions for CRC32 computation to Spicy library. (Robin
Sommer, Corelight)
* Add a find() method to units to search within their block of
input. (Robin Sommer, Corelight)
* Add support for searching a stream view backward for bytes. (Robin Sommer, Corelight)
* Optimize implementation of Spicy's library filters a little bit. (Robin Sommer, Corelight)
* Fix missing update to input position before running `%done` hook. (Robin Sommer, Corelight)
* Add support for `&eod` to `void` fields. This skips all data until
the end of the current input is encountered. (Robin Sommer,
Corelight)
* Add support for `&until` to `void` fields.
Example:
```
public type Foo = unit {
: void &until=b"XYZ";
}
```
This skips all data until `XYZ` is encountered. Similar to the
`&until` attribute that `bytes` provides, the deliminator is
extracted from the stream before continuing. (Robin Sommer,
Corelight)
* Add support for `&chunked` when parsing bytes data with `&until`
or `&until_including`. (Robin Sommer, Corelight)
1.1.0-dev.74 | 2021-05-06 19:11:00 +0200
* GH-911: Add `encode()` method to strings for conversion to bytes.
(Robin Sommer, Corelight)
* Let CI show any diff from clang-tidy. (Robin Sommer, Corelight)
* Fix to run-clang-tidy. (Robin Sommer, Corelight)
1.1.0-dev.70 | 2021-05-05 11:40:32 +0200
* Lower minimum required Python version to 3.2. (Benjamin Bannier, Corelight)
1.1.0-dev.68 | 2021-05-05 11:40:05 +0200
* Add rough scaffolding for fuzzing parsers. (Benjamin Bannier, Corelight)
1.1.0-dev.66 | 2021-05-04 16:20:50 +0200
* Specify python3 on cmake invocations for autogen-version. (Eli M)
* Bump minimum python version to 3.6. (Eli M)
* Port autogen-version to python3. (Eli M)
1.1.0-dev.62 | 2021-05-03 12:29:38 +0000
* GH-901: Fix type resolution bug in &convert. (Robin Sommer, Corelight)
* Extend HILTI printer to include IDs into tuple type output. (Robin
Sommer, Corelight)
1.1.0-dev.59 | 2021-04-30 14:34:20 +0200
* GH-882: Add Cirrus CI task for opensuse-15.2. (Benjamin Bannier, Corelight)
* Add Dockerfile for opensuse-15.2. (Benjamin Bannier, Corelight)
* Silence a signed-unsigned comparison warning. (Benjamin Bannier, Corelight)
1.1.0-dev.53 | 2021-04-29 13:00:26 +0000
* Updating submodule(s).
* Do not run clang-tidy/clang-format on Zeek submodules. (Robin
Sommer, Corelight)
1.1.0-dev.48 | 2021-04-27 15:52:40 +0200
* GH-882: Make spicy compile with bison 3.0. (Johanna)
1.1.0-dev.45 | 2021-04-27 08:51:29 +0000
* Provide mktime() library function. This allows creating a `time`
value from individual components, similar to mktime(3). (Robin
Sommer, Corelight)
1.1.0-dev.43 | 2021-04-26 15:32:25 +0000
* Allow to pass zlib's window bits parameter into Zlib unit type.
(Robin Sommer, Corelight)
* Force type information to come last in generated C++ code.
* Fix crash on macOS ARM when writing over an existing HLTO file.
(Robin Sommer, Corelight)
1.1.0-dev.37 | 2021-04-24 09:28:04 +0200
* Add new Spicy runtime function `spicy::zlib_init(window_bits)`
that initializes a `ZlibStream` for decompression with the given
window size argument. See the zlib documentation at
https://www.zlib.net/manual.html for more on the argument. (Robin
Sommer, Corelight)
1.1.0-dev.35 | 2021-04-23 11:27:07 +0200
* GH-37: Add documentation on how to skip data with `void` fields.
1.1.0-dev.33 | 2021-04-23 09:41:18 +0200
* Reject `void` fields with names. (Benjamin Bannier, Corelight)
* Validate supported attributes for `void` fields. (Benjamin Bannier, Corelight)
* Add validation rejecting `$$` in hooks not supporting it. (Benjamin Bannier, Corelight)
* GH-37: Fix handling of `&size` attribute for anonymous void fields. (Benjamin Bannier, Corelight)
1.1.0-dev.26 | 2021-04-21 13:13:11 +0200
* GH-889: Fix hook handling for anonymous void fields.
1.1.0-dev.24 | 2021-04-19 15:30:16 +0200
* Build Zeek plugin for benchmarking job. (Benjamin Bannier, Corelight)
1.1.0-dev.22 | 2021-04-19 09:24:14 +0000
* GH-766: Port Spicy to Apple silicon. (Robin Sommer, Corelight)
1.1.0-dev.20 | 2021-04-19 08:58:24 +0000
* GH-872: Fix missing normalization of enum label IDs. (Robin
Sommer, Corelight)
* GH-878: Fix casting integers to enums. (Robin Sommer, Corelight)
1.1.0-dev.15 | 2021-04-19 08:57:24 +0000
* Fix versioning for development. (Robin Sommer, Corelight)
1.1.0-dev.13 | 2021-04-15 14:27:47 +0200
* GH-844: Add support for &size attribute for unit switch statement. (Benjamin Bannier, Corelight)
1.1.0-dev.12 | 2021-04-15 14:26:45 +0200
* GH-26: Add skip, skip-pre and skip-post properties. (Benjamin Bannier, Corelight)
* Perform explicit validation of Spicy module properties. (Benjamin Bannier, Corelight)
1.1.0-dev.9 | 2021-04-15 13:25:20 +0200
* Add assertion to ensure that code cannot be reached. (Benjamin Bannier, Corelight)
1.1.0-dev.7 | 2021-04-13 11:49:36 +0200
* Disable 'create-release' and 'docker-tags' workflows for dev tags. (Benjamin Bannier, Corelight)
* Include build status for 1.0 release branch in README. (Benjamin Bannier, Corelight)
1.1.0-dev.3 | 2021-04-12 18:23:41 +0200
* Fix leftovers in GH actions from master -> main rename. (Benjamin Bannier, Corelight)
* Remove build tag for release build from README for now. (Benjamin Bannier, Corelight)
1.0.0 | 2021-04-12 14:22:09 +0200
* Fix metadata in Docker publishing workflow. (Benjamin Bannier, Corelight)
0.4.0-1694 | 2021-04-12 11:15:53 +0200
* Do not take freebsd12 into account for cumulative CI status.
* Documentation updates for 1.0.0 release. (Benjamin Bannier, Corelight)
* Do not automatically create `spicy:latest` Docker tag from `main`.
0.4.0-1687 | 2021-04-08 17:07:45 +0200
* Fix version string checking out release tags.
* Switch versioning scheme for dev versions for 1.0.0 release. (Benjamin Bannier, Corelight)
0.4.0-1684 | 2021-04-06 10:56:32 +0000
* Add a couple of documentation updates. (Robin Sommer, Corelight)
* Update CI release infrastructure. (Benjamin Bannier, Corelight)
* GH-860: Fix regexp matching across chunk boundaries. (Benjamin
Bannier, Corelight)
* Fix code markup in unit and sink docs. (Benjamin Bannier,
Corelight)
* GH-861: Add new combined operation for combined map indexing &
assignment. (Benjamin Bannier, Corelight)
* Add map index operator which can mutate contained returned value.
(Benjamin Bannier, Corelight)
* Bump pre-commit plugins. (Benjamin Bannier, Corelight)
0.4.0-1662 | 2021-03-27 08:49:01 +0000
* Update Zeek submodules.
* Fix doc links to package artifacts. (Benjamin Bannier, Corelight)
* Test plugins in CI on macOS. (Benjamin Bannier, Corelight)
0.4.0-1656 | 2021-03-25 12:19:35 +0000
* Adapt to `master` -> `main` branch renaming. (Robin Sommer, Corelight)
0.4.0-1653 | 2021-03-25 09:23:12 +0100
* Add new (missing) autogenerated docs. (Benjamin Bannier, Corelight)
0.4.0-1650 | 2021-03-25 07:40:06 +0000
* GH-848: Disable default building of Zeek plugin. (Benjamin
Bannier, Corelight)
* Update installation instructions for recent changes. (Robin
Sommer, Corelight)
0.4.0-1640 | 2021-03-24 22:09:16 +0100
* Clean up of CI artifact naming. (Benjamin Bannier, Corelight)
* Drop noexcept in Sink move constructor. (Benjamin Bannier, Corelight)
0.4.0-1632 | 2021-03-23 20:01:10 +0100
* GH-853: Fix extraction of integer bits when all bits requested.
When calculating the extraction mask, we previously would shift an
integer beyond its width which triggers UB. With this patch we now
correctly detect such cases and directly return the input.
* GH-450: Add `!in` to HILTI and Spicy parser.
0.4.0-1614 | 2021-03-18 12:27:45 +0000
* GH-827: Bump Zeek LTS version to 4.0.0 in some CI and Docker
images. (Benjamin Bannier, Corelight)
* Code optimization: Pass some arguments by ref if no value is
needed. (Benjamin Bannier, Corelight)
* Remove duplicate key in clang-format configuration. (Benjamin
Bannier, Corelight)
* Updating submodules.
0.4.0-1609 | 2021-03-18 09:45:01 +0000
* GH-817: Fix vector parsing potentially missing EOD. (Robin Sommer, Corelight)
* GH-835: Fix some issues diagnosed by LGTM. (Benjamin Bannier, Corelight)
* Fix `?.` operator to work with constant structs. (Robin Sommer, Corelight)
0.4.0-1603 | 2021-03-17 10:02:17 +0100
* GH-815: Add creation of DEB and RPM packaging artifacts. (Benjamin Bannier, Corelight)
0.4.0-1575 | 2021-03-13 09:24:03 +0000
* GH-821: Fix &max-size with constant argument. (Robin Sommer,
Corelight)
* GH-819: Fix unification of internal compiler arguments to not
change order. (Robin Sommer, Corelight)
0.4.0-1568 | 2021-03-11 07:04:13 +0000
* Fix some CMake, CI, and documtation issues. (Robin Sommer,
Corelight)
0.4.0-1564 | 2021-03-10 11:21:02 +0000
* Fix broken links in documentation. (Robin Sommer, Corelight)
0.4.0-1561 | 2021-03-10 08:40:38 +0000
* Move Zeek plugin into submodule and remove all dependencies on
Zeek from main Spicy distribution. (Robin Sommer, Corelight)
* Add environment variable HILTI_PRINT_SETTINGS that trigger
printing out compiler settings to stderr. (Robin Sommer,
Corelight)
* GH-812: Fix typos in docs. (Benjamin Bannier, Corelight)
* GH-813: Add missing word in docs. (Benjamin Bannier, Corelight)
* GH-569: Fix error message in validation of `real` fields.
(Benjamin Bannier, Corelight)
* GH-806: Fix benchmark script for move of analyzers. (Benjamin
Bannier, Corelight)
0.4.0-1551 | 2021-03-04 09:52:00 +0100
* GH-570: Explicitly reject comparisons of units with `==` and `!=`.
Comparisons with `==` and `!=` are not explicitly modelled in Spicy or
HILTI which means that while we resolved the types of the LHS and RHS,
during codegen we just emit a `==` or `!=` which needs to be resolved.
While most runtime types should support these operators, units do not,
so that previously users would see a compiler error during JIT when
comparing values of unit type with `==` or `!=`. This patch adds a
validator which explicitly rejects such comparisons.
0.4.0-1547 | 2021-03-03 09:13:25 +0100
* GH-798: Prevent conversions of some integers to doubles on the Zeek interface.
While we previously provided specialized conversion functions to Zeek
`ValPtr` these were not used for plain integer types like e.g.,
`std::size_t` or `unsigned long`. Instead C++ overload resolution
preferred to instead convert the integers to `double` (for which we
provide a specialization) and them being emitted with `TYPE_DOUBLE`.
This patch introduces an additional specialization which converts such
naked integers to safe integers and then converts them to Zeek.
0.4.0-1545 | 2021-03-03 09:12:29 +0100
* GH-733: Validate that `foreach` is only used with container fields.
0.4.0-1543 | 2021-03-02 07:37:21 +0000
* Move protocol analyzers into submodule. This moves all the
protocol analyzers into its own submodule `spicy-analyzers`, which
doubles as a Zeek package installing them. There's a new
`spicy-analyzers` target to pre-compile those analyzers for Zeek.
(Robin Sommer, Corelight)
* Zeek plugin updates (Robin Sommer, Corelight)
- Move registration of Zeek components to load time.
- Work around problem in Zeek's analyzer API.
* Update pathfind to fix problem on Alpine. (Robin Sommer,
Corelight)
0.4.0-1541 | 2021-03-02 07:22:26 +0000
* Add support for `@if/@else/@endif` in Spicy scripts, similar to
what EVT files already provided. This currently supports testing
for three constants at compile time: `SPICY_VERSION`,
`ZEEK_VERSION`, `HAVE_ZEEK`. See the documentation for more.
(Robin Sommer, Corelight)
* Increase CI memory for platforms that are prone to OOMing. (Robin
Sommer, Corelight)
0.4.0-1533 | 2021-03-02 07:21:39 +0000
* Fix strftime unit test: increase portability of localtime_r()
usage (Christian Kreibich, Corelight)
0.4.0-1531 | 2021-03-01 11:40:35 +0100
* GH-768: Add language support to explicitly trigger a parse error.
This patch adds a Spicy `throw` statement of the form
throw EXPR;
which when hit triggers a parse error with the message `EXPR`.
0.4.0-1529 | 2021-03-01 11:06:22 +0100
* GH-767: Add &max-size attribute.
This patch adds a &max-size field and unit attribute. The attribute
allows setting up a maximum number of bytes a parse step should consume.
It is conceptually similar to the existing &size attribute but differs
from it in that an error is raised only if more than the specified bytes
were extracted (&size enforces that exactly the specified number of
bytes is extracted).
&max-size cannot be used together with &size.
0.4.0-1521 | 2021-02-26 08:50:04 +0000
* GH-791: Bring back validation of field parseability. (Robin
Sommer, Corelight)
* GH-762: Add support for Zeek file analyzer replacement (requires
Zeek 4.1+). (Jon Siwek, Corelight)
0.4.0-1513 | 2021-02-25 14:49:43 +0000
* GH-771: Improve docs for vector &until parsing. (Jon Siwek,
Corelight)
0.4.0-1511 | 2021-02-24 19:34:56 +0000
* GH-694: Support unit-level `&convert` attribute. (Robin Sommer,
Corelight)
This attribute transforms a unit instance after parsing by returning the
result of a custom expression to parent units, instead of the instance
itself.
Example:
type Data = unit {
data:
bytes& size = 2;
}
&convert = self.data.to_int();
public
type Foo = unit {
numbers:
Data[];
on % done { print self.numbers; }
};
#printf 12345678 | spicy - driver unit - convert.spicy
[12, 34, 56, 78]
Note how the `Data` sub-elements have been turned into integers. Without
the new `&convert` attribute, the output would have looked like this:
[[$data=b"12"], [$data=b"34"], [$data=b"56"], [$data=b"78"]]
* GH-736: Fix bug that could cause hooks for anonymous fields to not
execute. (Robin Sommer, Corelight)
* Support parsing of nested fields. (Robin Sommer, Corelight)
The new test case shows a number of examples that are now supported:
public type Test = unit {
i1:
(bytes& size = 5);
i1b:
(bytes& size = 5 & convert = 42);
i2:
(int8[3]);
i3:
(int8[2])[3];
i4:
(int8[]& until = ($$ == 'X'))[]& until = ($$ == vector(55, 56));
i5:
(FooImpl);
i6:
(FooImpl[2]);
i7:
(FooImpl[2])[3];
i8:
(FooImpl& convert = 42);
i9:
(FooImpl& convert = 42)& convert = $$ * 2;
i10a:
(bytes& size = 5 & convert = $$.to_int());
i10b:
(Bytes& size = 5 & convert = $$.to_int());
i10c:
(FooImpl& convert = $$.data.to_int());
i11:
(FooImpl& convert = $$.data.to_int())& convert = $$ * 2;
i12:
(FooImpl& convert = $$.data.to_int())[2];
i13: (FooImpl &convert=$$.data.to_int())[2] &convert=[3 * xyz for xyz in $$];
}
While the expectation isn't really that people would write this, this
puts the infrastructure in place to support functionality that needs
more complex field structures. (Robin Sommer, Corelight)
* Force use of clang11 on FreeBSD 11. (Benjamin Bannier, Corelight)
* Support inline methods in HILTI structs. (Benjamin Bannier,
Corelight)
* Prettify error message when ID lookups yield something of
unexpected type. (Robin Sommer, Corelight)
* Add HILTI support for "auto" type that's derived through operator
resolving. (Robin Sommer, Corelight)
0.4.0-1491 | 2021-02-24 09:00:24 +0100
* GH-773: Refine check for incompatible parse attributes.
0.4.0-1489 | 2021-02-24 08:59:36 +0100
* GH-769: Skip evaluating `parse-at` or `parse-from` if field condition is False.
0.4.0-1486 | 2021-02-23 19:02:16 +0000
* Documentation updates. (Jon Siwek, Corelight)
- Add explicit mention of named-tuple elements.
- Fix inline literal markup in "optional" type.
- Add time/interval to Spicy->Zeek type conversion table.
- Remove "missing" tag on README's link to "real analyzer tutorial".
- Fix documentation link. (Benjamin Bannier, Corelight)
0.4.0-1479 | 2021-02-23 14:22:12 +0100
* GH-770: Evaluate the AND of all field `requires` clauses.
We already did this correctly for units by previously only evaluated the
first `&requires` clause for fields.
0.4.0-1476 | 2021-02-23 14:21:25 +0100
* GH-774: Fix method argument type deduction for argument packs.
0.4.0-1474 | 2021-02-22 13:46:31 +0000
* Tweak auto-discover of *.hlto files in Zeek plugin. (Robin Sommer, Corelight)
The plugin now scans for *.hlto files at startup at two locations by
default:
1. A directory "modules/" inside the plugins top-level directory.
2. Zeek's default plugin directory.
Both of these places are searched *recursively*, similar to how Zeek
looks for binary plugins.
Alternatively, one can override both of these by setting an environment
variable `SPICY_MODULE_PATH` to a colon-separated list of directories,
which will then be searched recursively *instead* of the default
locations.
* Extend spicy-config with a couple more Zeek-related options, and
tweak some existing option names. (Robin Sommer, Corelight)
0.4.0-1470 | 2021-02-22 14:31:35 +0100
* GH-772: Allow trailing comma in tuple type list.
Closes #772. (Benjamin Bannier, Corelight)
0.4.0-1468 | 2021-02-22 10:12:33 +0100
* GH-600: Add validation for field names clashing with unit names.
This would previously fail during C++ codegen; now we reject this code.
Closes #600. (Benjamin Bannier, Corelight)
0.4.0-1462 | 2021-02-22 10:10:53 +0100
* GH-719: Reject local variables at module scope.
Closes #719. (Benjamin Bannier, Corelight)
0.4.0-1455 | 2021-02-18 17:05:01 +0100
* Document `&byte-order` unit attribute. (Benjamin Bannier, Corelight)
* Document `&requires` unit attribute. (Benjamin Bannier, Corelight)
* Convert %byte-order unit property to an attribute. (Benjamin Bannier, Corelight)
* Convert %requires property to an attribute. (Benjamin Bannier, Corelight)
* Make tuple index type calculation more resilient. (Benjamin Bannier, Corelight)
* GH-744: Fix out-of-bounds access in tuple index type resolution.
This code was previously indexing into a vector beyond the end when
determining the type of a tuple index operation. We now
return an unknown type here here so that we can reject the operation
during later validation.
0.4.0-1446 | 2021-02-18 09:31:13 +0000
* GH-709: Fix codegen's result type of unit.offset(). (Robin Sommer,
Corelight)
* GH-709: Add description of enum use to Zeek plugin documentation.
(Robin Sommer, Corelight)
* Fix bug with custom enum types not being available in time for
Zeek scripts. Turns out that for precompiled Zeek analyzers it
didn't work to automatically make the Spicy-side enum types
available to Zeek scripts because of a timing issue. (Robin
Sommer, Corelight)
* Add grouping() function to builder API. (Robin Sommer, Corelight)
* Add new function linkage "preinit" to HILTI. (Robin Sommer,
Corelight)
0.4.0-1437 | 2021-02-17 11:08:16 +0100
* GH-682: Remove TODO in docs. (Benjamin Bannier, Corelight)
0.4.0-1435 | 2021-02-17 11:07:03 +0100
* GH-723: Provide constructor for ports.
Closes #723. (Benjamin Bannier, Corelight)
0.4.0-1430 | 2021-02-17 11:06:27 +0100
* GH-731: Add unit `position` method.
This method returns an iterator to the current position in the unit's
input stream.
Closes #731. (Benjamin Bannier, Corelight)
0.4.0-1428 | 2021-02-16 16:04:42 +0000
* GH-690: Add `%finally` hook to Spicy units.
``on %finally {
...
}``
Executes once unit parsing has completed in any way. This hook is
most useful to modify global state that needs to be updated no
matter the success of the parsing process. Once `%init` triggers, this
hook is guaranteed to eventually execute as well. It will run
*after* either ``%done`` or ``%error``, respectively. (If a new
error occurs during execution of ``%finally``, that will not
trigger the unit's ``%error`` hook.)
* Generalize %error to catch all exceptions. So far it was catching
only HILTI-side errors. Now we do general catch-all so have a
reliably hook for error handling no matter what's the cause.
(Robin Sommer, Corelight)
* Support finalizers in HILTI structs. There's a now new magic hook
`~finally` that will run automatically when a struct instance gets
destroyed. (Robin Sommer, Corelight)
* Fix double normalization in ID strings. (Robin Sommer, Corelight)
0.4.0-1422 | 2021-02-16 10:49:34 +0000
* GH-696: Add support for shared unit contexts with lifetime
determined by host application. (Robin Sommer, Corelight)
This introduces a new unit property for public unit types:
%context = <type>;
The unit's new `context()` method will then return a value of type
`<type>&` (i.e, a reference to an instance of the given type). This will
be initialized with a reference to a default-constructed instance of the
type.
What makes this special is that the host application can determine how a
context instance is being shared across different parsers: it can (1)
let multiples units link their context to the same instance, and (2)
retain a context instance beyond any specific unit's lifetime.
We use this new mechanism to provide contexts with connection-level
semantics: the Zeek plugin, and also spicy-driver's batch mode, will
create a single context instance that the two originator-side and
responder-side units will share (they must declare matching contexts
for that to work). The object will live until the connections gets
torn down (also meaning it'll be reused across the per-packet parsers
for UDP).
See the documentation for examples.
* Add `<vector>.pop_back()`. (Robin Sommer, Corelight)
* Support assignment to tuple elements. (Robin Sommer, Corelight)
* Extend the spicy-driver batch format to be able to express
bi-direction connection semantics. This increases the batch format
version to v2. (Robin Sommer, Corelight)
* Add new variant of external function linkage to HILTI that skips
the resumable wrapping. (Robin Sommer, Corelight)
0.4.0-1415 | 2021-02-16 10:29:53 +0000
* GH-726: $$ wasn't accessible from inside an `&until-including`
expression. (Robin Sommer, Corelight)
* GH-740: May address the unstable behavior by rewriting the code
surrrounding the `&while` parsing (but keeping semantics the
same). (Robin Sommer, Corelight)
0.4.0-1412 | 2021-02-16 10:07:45 +0000
* Activate CI for Debian 10 on master. (Robin Sommer, Corelight)
* GH-629: Use linb::any instead of std::any across the code base.
(Robin Sommer, Corelight)
* GH-738: Fix crash in spicy-driver when used with units that
require parameters. (Robin Sommer, Corelight)
* GH-737: Fix non-converging AST in case of ambiguous ID usage.
(Robin Sommer, Corelight)
0.4.0-1403 | 2021-02-09 13:23:55 +0000
* Fix missing f-string in Spicy Pygments lexer for floats/reals (Jon
Siwek, Corelight)
* GH-722: Bump pathfind. (Benjamin Bannier, Corelight)
* GH-732: Add test for setting input beyond current position. (Benjamin Bannier, Corelight)
* Add missing dependency of header precompilation target on script.
(Benjamin Bannier, Corelight)
* Fix documentation of `set_input`. (Benjamin Bannier, Corelight)
0.4.0-1394 | 2021-02-05 07:09:51 +0000
* Fix compatibility with Zeek 4.1 API changes/removals. (Jon Siwek,
Corelight)
0.4.0-1392 | 2021-02-04 15:14:31 +0000
* GH-962: Fix type inference for constructors of
vector/list/set/map, and for locals/globals. (Benjamin Bannier,
Corelight; Robin Sommer, Corelight)
* Remove unused function typeOfExpressions. (Benjamin Bannier,
Corelight)
0.4.0-1385 | 2021-02-04 14:59:06 +0000
* Add Pygments lexers for `.spicy` and `.evt` files. (Jon Siwek,
Corelight)
* Use syntax highlighting for Spicy/EVT/Zeek/C++ in documentation.
(Jon Siwek, Corelight)
* Add 'livehtml' target to `doc/Makefile` that will run
`sphinx-autobuild`, plus some cleanup. (Jon Siwek, Corelight)
0.4.0-1375 | 2021-02-03 11:43:19 +0100
* Enhance parsing of single-element container constructors so that
trailing `,` is optional. (Benjamin Bannier, Corelight)
* GH-711: Fix formatting of a number of predefined enum values.
(Benjamin Bannier, Corelight)
* GH-697: Allow trailing ',' in enum declarations. (Benjamin
Bannier, Corelight)
* Update some links in the documentations. (Benjamin Bannier,
Corelight)
* GH-699: Add script to store and upload release artifacts.
(Benjamin Bannier, Corelight)
0.4.0-1358 | 2021-02-01 10:40:23 +0100
* GH-701: Make offsets usable from unit field attributes.
We previously updated parse positions when entering productions. This is
insufficient as the evaluation of attributes might already access the
position via `self.offset()`.
With this patch we now only update the parse position before we start
parsing of a field.
Closes #701. (Benjamin Bannier, Corelight)
* Use ubuntu-20 as base for published Docker image. (Benjamin Bannier, Corelight)
* Set PATH and ZEEK_PLUGIN_PATH for ubuntu Docker images. (Benjamin Bannier, Corelight)
* Validate Zeek installation for Docker CI tasks. (Benjamin Bannier, Corelight)
* Fix Docker images for possible lib libdir.
This patch fixes the `ZEEK_PLUGIN_PATH` set in Docker images so Zeek can
find plugins even if the Spicy installation was performed with a libdir
named `lib` instead of `lib64`. (Benjamin Bannier, Corelight)
0.4.0-1346 | 2021-01-29 08:26:26 +0000
* GH-685: Allow to define a minimum Spicy version that a module
requires. Adding `%spicy-version = "1.0";` to a module now lets it
require to compile with a Spicy of at least that version. The
version can be specified as either `x.y` or `x.y.z`. (Robin
Sommer, Corelight)
* GH-688: Add functions `zeek::uid()` and `zeek::fuid()` to access
UIDs and FUIDs, respectively. (Robin Sommer, Corelight)
* GH-689: Allow to pass a MIME type to `file_new()` when beginning
file analysis. (Johanna Amann, Corelight)
* GH-700: Support `&size` attribute at the unit level.
This limits input to the given number of bytes for the unit:
type Bar = unit(n: uint64) {
x:
bytes& eod;
} &size=n;
Parsing less than `n` bytes is an error. (Robin Sommer, Corelight)
* Fix --git-root in autogen-version call. (Robin Sommer, Corelight)
* Fix using `Null` as a parameter default value for `optional<T>`.
(Robin Sommer, Corelight)
* Add timestamp removal to the Zeek log canonification in tests.
(Johanna Amann, Corelight)
0.4.0-1330 | 2021-01-28 10:22:55 +0100
* GH-701: Add strptime Spicy library function.
0.4.0-1324 | 2021-01-26 17:59:30 +0000
* Update reproc to release version. (Robin Sommer, Corelight)
0.4.0-1321 | 2021-01-26 13:23:11 +0100
* Lots of CI / clang-format / Docker cleanup. (Benjamin Bannier, Corelight)
0.4.0-1305 | 2021-01-21 15:34:39 +0000
* Fix failing Zeek tests. (Robin Sommer, Corelight)
0.4.0-1304 | 2021-01-21 13:37:38 +0000
* Fix a couple of issues with Zeek 4.0. (Robin Sommer, Corelight)
* GH-640: Extend spicy-doc to document begin/end operators. (Robin
Sommer, Corelight)
* Bulk update of all test baselines. (Robin Sommer, Corelight)
* Update copyright headers. (Robin Sommer, Corelight)
* Do not use #pragma once in libhilti.h. (Robin Sommer, Corelight)
* Disable output from header compilation. (Robin Sommer, Corelight)
* Update README. (Robin Sommer, Corelight)
* Remove unused TLS baselines. (Robin Sommer, Corelight)
* Prettify configure output. (Robin Sommer, Corelight)
0.4.0-1286 | 2021-01-21 12:04:58 +0100
* Add CI job to validate release tarball. (Benjamin Bannier,
Corelight)
* Document more platforms in the installation docs. (Benjamin
Bannier, Corelight)
* Use "default" compiler if possible in Docker images. (Benjamin
Bannier, Corelight)
* Make master-only CI tasks triggerable. (Benjamin Bannier,
Corelight)
0.4.0-1277 | 2021-01-20 19:55:47 +0000
* Add final newline to ASCII output of spicy-dump. (Robin Sommer,
Corelight)
* GH-671: Fix parsing of empty look-ahead vectors. (Robin Sommer,
Corelight)
* Fix Zeek plugin build when toolchain is disabled. (Robin Sommer,
Corelight)
* Let the autogen-version script execute out of the top-level
directory. (Robin Sommer, Corelight)
0.4.0-1270 | 2021-01-20 14:04:16 +0100
* Allow to enforce sequential execution of jobs spawned during JIT.
If the environment variable `HILTI_JIT_SEQUENTIAL` is set jobs are
executed sequentially. (Benjamin Bannier, Corelight)
* Use sequential JIT mode in CI to save memory. (Benjamin Bannier,
Corelight)
* GH-582: Add master CI task for macOS Big Sure. (Benjamin Bannier,
Corelight)
* GH-670: Remove pre-commit hooks which require external tool
installation. (Benjamin Bannier, Corelight)
* Move `spicy::rt::getenv` to `hilti::rt`. (Benjamin Bannier,
Corelight)
0.4.0-1262 | 2021-01-19 08:28:41 +0000
* macOS fixes (Robin Sommer, Corelight)
- GH-664: Increase minimum fiber stack size.
- Pass CMAKE_OSX_SYSROOT through to JIT linker.
* Add a couple cron scripts to repository. (Benjamin Bannier, Corelight)
- `ci/cron/spicy-benchmark.sh:` run Spicy benchmarks
- `ci/cron/update_spicy_dockerhub.sh: update Spicy Docker images
0.4.0-1255 | 2021-01-18 08:45:26 +0000
* Port JIT to use host C++ compiler instead of libclang/libllvm.
(Robin Sommer, Benjamin Bannier; Corelight).
This completely removes the dependency on an existing clang
installation. Instead, JIT now requires only a C++17 host
compiler. Internally, this comes with a substantial refactor of
the JIT components.
This changes also includes the following additional pieces:
- We pull in `https://github.com/DaanDeMeyer/reproc` as a
submodule for spawning the host compiler.
- The environment variable `HILTI_CXX` overrides the
executable to use as the C++ compiler during JIT. It won't
work to switch in a completely different compiler that way
(because we'd continue to make some assumptions based on the
default compiler), but it allows to provide an additional
wrapper around the default compiler.
- We remove the distinction between building the toolchain
with/without JIT. We now always assume we have the JIT
capbility available, but will fail gracefully when unable to
use the host compiler.
- JIT compilation captures & reports C++ error output.
* Extend `{hilti,spicy}-config` with options `--cxx-flags-hlto`/`--cxx-flags-hlto`
printing compiler flags for building HLTO files.
* CMake: Redoing object library linking. We weren't consistent in
how we linked them, which led to some off effects. (Robin Sommer,
Corelight)
* Codegen: Construct initializer lists from literals. Benjamin
Bannier, Corelight)
* Cleanup left-over ASAN code that was no longer in use. (Robin
Sommer, Corelight)
* Increase CI memory limit for Ubuntu debug task. (Robin Sommer, Corelight)
* Bump CMake in CI Docker image. (Benjamin Bannier, Corelight)
0.4.0-1227 | 2021-01-13 07:41:35 +0000
* Re-implement the runtime's fiber (co-routine) code. (Benjamin
Bannier, Robin Sommer; Corelight).
This should be mostly transparent to users, but internally the
main pieces are:
- GH-561: Switch from libtask to new 3rd-party fiber library
(https://github.com/simonfxr/fiber.git).
- GH-561: Support "shared stack" mode for fibers where the runtime
only allocates a single stack that's used by all fibers, vs.
individual per-fiber stacks. This comes with a slight
performance penalty for context switching, but substantially
improves memory usage (and, in turn, performance with lots of
concurrent fibers). The shared stack is used by default now.
There's also new benchmark hilti-rt-fiber-benchmark to profile
performance.
- GH-460: Add runtime checks for risk of stack overflow. When the
HILTI codegen creates a function, it now always inserts a stack
check at the beginning that ensures that a minimal amount of
stack space remains available. If it is not, we abort with a
runtime exception. This gives us well-defined behavior if call
stacks go too deep. The minimum amount of stack space is
configurable through a global option.
Along with this comes large refactor of the runtime's fiber code.
* Fix Linux' --ldflags for pthread. (Robin Sommer, Corelight)
* Make hilti::rt::isInitialized() save to be used when no global
state is available. (Robin Sommer, Corelight)
* When running the CI container through the Makefile, use local
user's UID/GID. (Robin Sommer, Corelight)
* Adding missing call to rt::done() to default main(). (Robin
Sommer, Corelight)
* Fix CMake logic for hilti-rt. (Robin Sommer, Corelight)
* Enable leak checking by default in ASAN builds. ASAN itself
enables it only on Linux by default. (Robin Sommer, Corelight)
* Let autogen-version find VERSION independent of current directory.
(Robin Sommer, Corelight)
0.4.0-1201 | 2021-01-11 21:03:10 +0100
* Activate building Spicy in CentOS 7 Docker image. (Benjamin Bannier,
Corelight)
0.4.0-1199 | 2021-01-10 14:03:48 +0000
* Remove manual installation of Zeek CMake files on FreeBSD CI. (Benjamin
Bannier, Corelight)
0.4.0-1196 | 2021-01-09 19:13:50 +0100
* GH-650: Add infrastructure to upload a release tarball for tags. (Benjamin
Bannier, Corelight)
* GH-633: Fixes for FreeBSD, plus CI support. (Benjamin Bannier, Corelight)
* Bump minimal version of BTest in CI to 0.66. (Benjamin Bannier,
Corelight)
0.4.0-1175 | 2021-01-06 07:28:23 +0000
* Fix for macOS 10.16. (Robin Sommer, Corelight)
0.4.0-1173 | 2021-01-05 11:01:34 +0100
* GH-644: Remove Spicy support for `else` on `while`.
0.4.0-1148 | 2021-01-04 08:10:25 +0000
* Overhauling regexp matching semantics.
We were inconsistent in how regular expressions (1) handled being anchored
vs not, and (2) allowed to retrieve the matching data. We fix this as
follows:
- We rename the regexp methods for more consistency: `match` now means
"matching is anchored at the beginning of the data", and `find` means
we find matches anywhere in the data. This leads to the following
renames:
* <regexp>.find -> <regexp>.match
* <regexp>.find_groups -> <regexp>match_groups
* <regexp>.find_span -> <regexp>.find
`<bytes>.match` adapts to the new semantics as well.
- We remove the `&anchor` attribute because that's the default now.
- `&no_sub` remains available to override the default choice of the
matcher type: it'll use the minimal matcher and disable capture group
support.
0.4.0-1145 | 2021-01-02 20:43:29 +0100
* GH-635: Collection of cleanups from consuming Spicy in external project.
(Benjamin Bannier)
* GH-620: Add paths to Spicy and Zeek to Ubuntu Docker image PATHs. (Benjamin Bannier, Corelight)
* Extend Docker images. (Benjamin Bannier, Corelight)
- Add Debian-10 Docker image.
- Add Debian-9 Docker image.
- Add Fedora-32 Docker image.
- Add Fedora-33 Docker image.
- Add Ubuntu-16 Docker image.
- Add Ubuntu-18 Docker image.
- Add Ubuntu-20 Docker image.
- Remove unsupported Ubuntu-19.10.
- Fix yum repository name in CentOS 8 Docker images. (GH-619)
- Use correct download location for Zeek packages.
* GH-575: Add support for FreeBSD. (Benjamin Bannier, Corelight)
* Fix test 'spicy.rt.time'. (Benjamin Bannier, Corelight)
* C++ fix to be explicit on which variant we construct in `Result`. (Benjamin
Bannier, Corelight)
0.4.0-1118 | 2020-12-11 07:18:52 +0000
* GH-602: Install Zeek from packages. (Benjamin Bannier, Corelight)
0.4.0-1112 | 2020-12-10 10:33:31 +0000
* GH-25: Add support for manual backtracking to Spicy. If a field is
marked with ``&try``, a later call to ``self.backtrack()``
anywhere down in the parse tree will return to that position and
continue there. See the documentation for more, including an
example. (Robin Sommer, Corelight).
* Support &eod for vector parsing. (Robin Sommer, Corelight)
* Fix Sphinx helper script that emits command lines. (Robin Sommer,
Corelight)
0.4.0-1106 | 2020-12-10 10:10:16 +0000
* Make capture groups available inside hooks of regular expression fields.
(Robin Sommer, Corelight)
Example:
x: /(a.c)(de*f)(h.j)/ {
print $1, $2, $3;
}
This will print the pieces corresponding to the three groups inside the
expression.
Couple notes:
- We don't make $0 available as that would be the same as `$$`.
- Matching with capture groups is more expensive that without. If
you don't use any groups in the regexp, the overhead is
automatically removed. If you use groups, but don't need them to
capture, add `&nosub` to the field.
* Make new captures() method available on regexp matching state in
HILTI. (Robin Sommer, Corelight)
* Fix argument mismatches between Spicy generated C++ prototypes and
Zeek plugin runtime code. (Robin Sommer, Corelight)
* Fix compile error on non-JIT version of plugin, and fix unsafe use
of string_view. (Robin Sommer, Corelight)
* Fix warnings reported by Apple's compiler. (Robin Sommer,
Corelight)
0.4.0-1093 | 2020-12-10 08:39:26 +0100
* Switch runtime library to use "fiber" library instead of libtask.
We now leverage https://github.com/simonfxr/fiber. (Benjamin
Bannier, Corelight)
* GH-599: Fix construction of `Bytes`. (Benjamin Bannier, Corelight)
* GH-603: Disable precompiled headers if we are building with GCC.
(Benjamin Bannier, Corelight)
* GH-94: Explicitly enable leak checking on tests relying on it being
active. (Benjamin Bannier, Corelight)
* GH-492: Add unit tests for IntrusivePtr. (Benjamin Bannier,
Corelight)
* Make run-zeek script also work with ASAN on macOS. (Benjamin
Bannier, Corelight)
* Remove unused Zeek compiler driver variable. (Benjamin Bannier,
Corelight)
* Add missing runtime cleanups in a few places. (Benjamin Bannier,
Corelight)
* Fix some documentation links and link checking. (Benjamin Bannier)
* Fix GitHub link. (Robin Sommer, Corelight)
0.4.0-1071 | 2020-11-30 18:16:35 +0000
* Series of speed-ups to the Spicy compiler. (Robin Sommer, Corelight)
- Add caching of derived information to some AST nodes.
- Use optional_ref<T> for returning references of AST node attributes instead of copies. (Robin Sommer, Corelight)
- Cache location instances.
- Returning const references for lots of accessor methods. (Robin Sommer, Corelight)
- Mark two AST node classes as singletons.
- Switch AST infrastructure from shared_ptr to IntrusivePtr.
- Use forward declaration for nlohmann:json where we can.
* Relabel profiling collector names in JIT code. (Robin Sommer, Corelight)
* Fix for type erasure profiling. (Robin Sommer, Corelight)
0.4.0-1057 | 2020-11-23 08:35:04 +0000
* GH-584: Add new vector methods resize/sub/at. (Robin Sommer, Corelight)
* Revert accidental unbumping of tinyformat. (Benjamin Bannier, Corelight)
0.4.0-1053 | 2020-11-18 08:33:32 +0000
* GH-568: Change parsing of look-ahead vectors to use a loop instead
of recursion. (Robin Sommer, Corelight)
* GH-583: Fix `foreach` inside a unit hook. The `foreach` attribute
wasn't passed on, and the '$$' lookup didn't work inside a
non-inline hook. (Robin Sommer, Corelight)
* GH-574: Implement and document hook priorities. Robin Sommer,
Corelight)
* GH-389: Add additional tuple constructor expression (`tuple(...)`).
(Robin Sommer, Corelight)
* GH-544: Let Zeek plugin search for HLTO files at startup. By
default, the plugin now loads all *.hlto files found in
`$prefix/lib/spicy/Zeek_Spicy/modules/`. This can be changed by
setting the environment variable SPICY_MODULE_PATH to a
colon-separated list of directories to search. (Robin Sommer,
Corelight)
* Force upgrade to well-defined BTest version in our Docker
containers. (Robin Sommer, Corelight)
* Forward Zeek plugin's debug messages to Zeek's debug logger. That
means that 'zeek -B plugin-_Zeek-Spicy' will send them into Zeek's
debug.log. (Robin Sommer, Corelight)
0.4.0-1031 | 2020-11-17 14:05:32 +0100
* GH-571: Fix padding handling when formatting nan and inf values.
This patch bumps tinyformat to a version containing the upstream fix for
https://github.com/c42f/tinyformat/issues/76 (and the follow up fix
https://github.com/c42f/tinyformat/pull/80).
We also add a regression test.
Closes #571. (Benjamin Bannier, Corelight)
* GH-589: Fix "'_debug’ was not declared in this scope" build error (Robin Sommer, Corelight)
* Mention header precompilation in installation docs. (Benjamin Bannier, Corelight)
* Disable precompiled headers for gcc CI job.
While mixing GCC and Clang with precompiled headers is an issue for JIT
(JIT always uses Clang to compile modules while precompiled headers are
created with the compiler used to compile Spicy itself), we disable
precompiled headers for the non-JIT, GCC job since GCC emits a
non-suppressible warning, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89808,
build/hilti/runtime/include/hilti/rt/libhilti.h:3:9: error: #pragma once in main file [-Werror]
3 | #pragma once
| ^~~~
cc1plus: all warnings being treated as errors
Instead of not treating warnings as errors anymore we disable precompiled
headers in this build since it does not use them anyway. (Benjamin Bannier, Corelight)
* Add missing include. (Benjamin Bannier, Corelight)
* Optionally precompile libhilti.h.
This patch adds precompilation of libhilti.h for developer setups. This
header is included in all files during JIT, and precompiling it can for
individual files reduce JIT time by up to 30%. This behavior can be
disabled by setting the CMake option `HILTI_DEV_PRECOMPILE_HEADERS=OFF`.
Specifically, we add a cmake target which creates debug and release
versions of a precompiled libhilti.h. We modify HILTI's built-in
CXXFLAGS to check for the presence of the needed precompiled header in
the expected location. We expect the header either in a specific
location in the build directory, or optionally in a version directory in
`$HOME/.cache/spicy`.
In this patch we enable use of the precompiled header only during JIT,
but e.g., not when building Spicy targets. This allows us to profit from
faster JIT while still making efficient use of e.g., ccache.
Additionally, this approach allows us to punt on fixing a number of
issues in header files which get exposed during JIT (JIT artifacts are
not affected at the moment).
The original goal was to precompile lihilti.h before installation which
is not reached. In general, it seems that precompiled headers are not
intended to be moved after creation. This poses a challenge when the
header gets installed: while we could in principle generate the
precompiled header at the final location during install time, this seems
not reliably possible when generating packages which can be expanded in
arbitrary locations.
Additionally, we also install a script for users to take advantage of
precompiled headers once Spicy has been installed. (Benjamin Bannier, Corelight)
0.4.0-1020 | 2020-11-12 11:04:57 +0000
* Fix performance regression by removing wasteful formatting of debug message
in release mode in Spicy runtime driver. (Robin Sommer, Corelight)
* Fix fiber result handling on abort. (Robin Sommer, Corelight)
* GH-578: Fix glib decompression. Robin Sommer, Corelight)
* Add helper tool 'spicy-batch-extract' that extracts a single session out
of a Spicy batch file. (Robin Sommer, Corelight)
* Add preliminary benchmark script `scripts/run-benchmarks`. See developer
docs for more. (Robin Sommer, Corelight)
0.4.0-1011 | 2020-11-05 17:09:53 +0100
* GH-451: Support building on centos7 (Benjamin Bannier, Corelight)
0.4.0-1006 | 2020-11-05 12:56:56 +0000
* GH-497: Add support for packet analyzers to Zeek plugin. (Robin Sommer,
Corelight)
Zeek's master branch received support for adding low-level "packet
analyzers" through plugins. This extends the Spicy plugin to support that.
The documentation gained a new section explaining how to sue that.
Note: This feature will not "officially" be supported until Zeek 4.0
comes out (because we don't have Zeek master in CI).
* Add --zeek-version argument to spicy-config. (Robin Sommer, Corelight)
* Compile fix for GCC on Ubuntu. (Robin Sommer, Corelight)
0.4.0-1001 | 2020-11-05 08:47:23 +0000
* Use stable names for hook implementations, plus a text fix.
0.4.0-999 | 2020-11-04 15:18:45 +0100
* Use a lightweight lambda class to callbacks executing on fibers. (Benjamin Bannier, Corelight)
* Require all callbacks executing on fibers to return something. (Benjamin Bannier, Corelight)
* Remove unused type_info::overload. (Benjamin Bannier, Corelight)
* GH-546: Model `TypeInfo` with a union instead of a `std::variant`. (Benjamin Bannier, Corelight)
0.4.0-994 | 2020-11-03 09:22:40 +0000
* Update test baseline. (Robin Sommer, Corelight)
0.4.0-993 | 2020-11-02 16:09:13 +0000
* GH-159: Add some missing operator<< to runtime library. This now lets
"-X trace" work for Spicy parsers. (Robin Sommer, Corelight)
* GH-531: Make generated grammar symbols globally unique during Spicy
codegen. (Robin Sommer, Corelight)
* GH-176: Avoid infinite recursion when printing ASTs with self-recursive
types. (Robin Sommer, Corelight)
* Fix premature library shutdown in spicy-dump, which could lead to crashes
during cleanup of the parsed unit. (Robin Sommer, Corelight)
* Fix error message in validator. (Robin Sommer, Corelight)
* Aggressively compress package artifact for static build. (Benjamin Bannier,
Corelight)
* Remove packaging tasks for static CI build. Static builds tend to produce
very large binaries which cannot reliably be compressed to be inside Cirrus
CI's 1GB artifact size limit. (Benjamin Bannier, Corelight)
* Fix documentation source link. (Benjamin Bannier, Corelight)
* Consistently use `hilti::rt::filesystem` instead of `std::filesystem`.
(Benjamin Bannier, Corelight)
0.4.0-979 | 2020-10-23 14:11:27 +0000
* GH-95: Zeek plugin: Respect %port and %mime-type properties in
units. (Robin Sommer, Corelight)
* GH-513: Catch if an interval's number of seconds is out of range.
(Robin Sommer, Corelight)
* Clean up parsers' bounds checking of signed integer values. (Robin
Sommer, Corelight)
* GH-515: Change {
time, interval
}::nanoseconds() to return integers
instead of real. (Robin Sommer, Corelight)
* Fix bashism in test helper script. (Robin Sommer, Corelight)
* GH-490, GH-507: Provide error message when attempting to have
spicy-driver compile inputs when JIT is not available. (Robin
Sommer, Corelight)
* Reject regexp unit fields that aren't constant. (Robin Sommer, Corelight)
* GH-524: Fix spicy-driver usage message. (Robin Sommer, Corelight)
* GH-526: Show executable's name in usage/error messages of binaries
replying on the spicy-driver host code. (Robin Sommer, Corelight)
* GH-527: Updating usage output in docs. (Robin Sommer, Corelight)
* GH-528: Clarify spicyc's error message on needing HLTO filenames
with -o. (Robin Sommer, Corelight)
* GH-529: Clarifying libhilti's usage() fallback message. (Robin
Sommer, Corelight)
* GH-540: Fix bug in regular expression parsing. (Robin Sommer,
Corelight)
* GH-270: Fix for unit filters. (Robin Sommer, Corelight)
* GH-500: Remove --include-linker option from driver. The better way
to do all this is using "-l" to generate *just* the linker code,
then compile the two together as two compilation units. (Robin
Sommer, Corelight)
* GH-499: Provide has_label() method for enum values. Returns true
if the enum maps to a known type value other than Undef. (Robin
Sommer, Corelight)
* GH-556: Fix coercion of optional. (Robin Sommer, Corelight)
* GH-481: Correctly set locale in centos8 image. (Benjamin Bannier,
Corelight)
* Use working and stable link in host application documentation.
(Benjamin Bannier, Corelight)
* Fix documentation to reference latest support alpine version.
(Benjamin Bannier, Corelight)
* GH-546: Use plain function pointers for type info accessors.
(Benjamin Bannier, Corelight)
0.4.0-946 | 2020-10-23 09:35:04 +0200
* GH-522: Switch filesystem impl over to ghc::filesystem.
Closes #522. (Benjamin Bannier, Corelight)
* Fix use of uninitialized value when numbering unit fields.
When were using a non-const member function to call a delegated base
class constructor. Since at the point where the member function was
called the derived class was not fully constructed, the members it
accessed and mutated were uninitialized, we were reading from an unset
variable, attempting to set it, and then overwrote its values when the
member initializer was triggered (after the base class was constructed).
This lead to unit field indices set to large value which were not usable
down the line.
We now delegate maintaining the field counter to a mixin class which can
be constructed before the base class needing its state. (Benjamin Bannier, Corelight)
0.4.0-939 | 2020-10-21 14:35:30 +0200
* GH-521: Do not create copies when loading HLTO libraries. (Benjamin Bannier, Corelight)
0.4.0-935 | 2020-10-12 07:13:46 +0000
* Add support for new %print hook to Spicy units. (Robin Sommer,
Corelight)
This allows a unit to customize its appearance when printed (or
more generally: rendered into a string).
Example:
#cat test.spicy
public type Test = unit {
on % init { print self; }
f1:
uint32;
f2:
uint8;
on % done { print self; }
on % print {
local f1 = "-";
local f2 = "-";
if ( self?.f1 )
f1 = "%s" % self.f1;
if ( self?.f2 )
f2 = "%s" % self.f2;
return "%s|%s" % (f1, f2);
}
};
#printf '\x00\x00\x00\x01\x02' | spicy - driver test.spicy
-|-
1|2
* Allow customizing the string rendering of HILTI structs. If a
struct provide a hook __str__(), that will be used instead of the
built-in rendering of all its fields. (Robin Sommer, Corelight)
* Make ID getter method constant. (Robin Sommer, Corelight)
0.4.0-930 | 2020-10-09 18:10:13 +0200
* Change `Stream::data` and `stream::View::data` to return a `Bytes`
instead of a `string`. (Benjamin Bannier, Corelight)
* Test & CI updates. (Benjamin Bannier, Corelight)
- Add a CI master step to validate Homebrew formula works.
- Add missing Cirrus CI instance to Homebrew task.
- Build compiler toolchain by default when configuring directly from CMake.
- Bump default Zeek package in Docker images to more recent version.
- Fix removal of build directory in CentOS 8 Docker container.
- Increase sleep amount in test to make time measurable in CI.
- Speed up test `spicy.types.unit.trimming`.
* Help clang-tidy understand control flow in a couple of places.
(Benjamin Bannier, Corelight)
0.4.0-918 | 2020-10-06 12:23:08 +0200
* Introduce the notion of Spicy-specific compiler options.
So far, we only had codegen options for the HILTI compiler. This
change allows Spicy to track options as well that control specifics of
how the Spicy-to-HILTI code generations operates. Includes a couple of
things:
- As there's no appropriate global state on the Spicy side
where we could story options easily, we extend the
HILTI-side options struct with the notion of "auxiliary
options", described by key/value pairs. Spicy can then store
its options in there.
- We derived a separate spicy::Driver from hilti::Driver to
customize behaviour.
- We introduce hooks into the hilti::Driver to add support for
additional command line options. This remains a bit ad-hoc
for now. At some point, we'll clean up of command-line
handling generally works across the various tools, we have
quite a bit of code duplication there right now.
- This introduces a Spicy-side `-Q` option to record stream
offsets during parsing. Right now, the option isn't having
any effect yet, the implementation is going to come in a
subsequent commit. spicy-dump's previous `-o` is renamed to
`-Q` as well. (Robin Sommer, Corelight)
* Add an option to emit data offsets from spicy-dump.
This patch adds an option `-o` to `spicy-dump` which can be used to
trigger outputting of the stream offsets where unit fields were parsed
from. (Benjamin Bannier, Corelight)
* Make parser struct `__offset` member accessible from Values. (Benjamin Bannier, Corelight)
* Store field offsets during parsing. (Benjamin Bannier, Corelight)
* Make unit fields store their unit-wide field counter.
Initially the parser will create unit fields as `UnresolvedField`s. This
patch adds an `_index` field and accessor for this field which will
be used to number all unit fields in the final, fully constructed and
resolved unit so offsets can be computed. (Benjamin Bannier, Corelight)
* Store whether a struct field is internal on type level. (Benjamin Bannier, Corelight)
* Fix `type::struct_::Field::is*` methods to return booleans.
These functions were previously returning optional field values which
carry no information beyond being set or not. This patch changes these
methods to return booleans instead. (Benjamin Bannier, Corelight)
* Fully type constructions of empty optional values. (Benjamin Bannier, Corelight)
* Make `rt::Map::contains` `const`. (Benjamin Bannier, Corelight)
0.4.0-906 | 2020-10-06 08:11:53 +0200
* GH-511: Fix order of flags in spicyc usage message.
* GH-512: Fix typo in error message.
0.4.0-903 | 2020-09-28 18:42:38 +0000
* Reorganize source code to allow for building the runtime library
standalone. This adds a corresponding configure option:
--build-toolchain=yes/no (default: yes)
* Add configure option --disable-zeek-plugin-install to disable
installing the Zeek plugin into Zeek's plugin path. (Robin Sommer,
Corelight)
* Add CMake module that enables Zeek plugins to build Spicy parsers.
* Rename Zeek plugin to "_Zeek::Spicy" to work-around a loading
order issue in Zeek. (Robin Sommer, Corelight)
* Turn all 3rdparty code into proper git submodules. (Robin Sommer,
Corelight)
* Make using Gold linker default in CMake config. This is no change
from when using "configure", but it changes the default when
running "cmake" directly. (Robin Sommer, Corelight)
* Fix configuration of Gold linker to apply to cmake modules as
well. (Robin Sommer, Corelight)
* Extend hilti-config and spicy-config with options to print the
runtime's C++ include directories. (Robin Sommer, Corelight)
* Avoid needing "which" in configure. (Robin Sommer, Corelight)
* Disable a couple unit tests when running as root. (Robin Sommer,
Corelight)
* Fix compiler warning. (Robin Sommer, Corelight)
0.4.0-890 | 2020-09-28 18:41:08 +0000
* Port Zeek plugin to current Zeek master. (Robin Sommer, Corelight)
* GH-508: Prevent loss of precision when codgening Time
constructors. (Benjamin Bannier, Corelight)
* GH-503: Fix constructor `interval_ns` to accept unsigneds like
documented. (Benjamin Bannier, Corelight)
* Include `netinet/in.h` for `in[6]_addr` in `rt/address.h`.
(Benjamin Bannier, Corelight)
* Fix whitespace in spicy-driver batch usage string. (Benjamin
Bannier, Corelight)
0.4.0-884 | 2020-09-08 07:39:00 +0000
* Reimplement HILTI's stream type to substantially improve
performance. (Robin Sommer, Corelight)
* Move resolving of exceptions' backtrace symbols to point-of-use
for improving performance. (Robin Sommer, Corelight)
* Fix problem with fiber reuse. (Robin Sommer, Corelight)
* Fix a unit test that's using multiple filters. (Robin Sommer,
Corelight)
* Add macOS work-around for problem with doctests' platform
detection. (Robin Sommer, Corelight)
* When configured for debug mode, build tests without -O and link
them against {hilti,spicy}-rt-debug. (Robin Sommer, Corelight)
0.4.0-874 | 2020-08-27 14:21:57 +0200
* Update parser lookup tables exclusively from `spicy::rt::init`.
We previously were updating `globalState()->parsers_by_mime_type` from
`registerParser` while `globalState()->parsers_by_name` was updated by
`init`. This patch makes `init` the only function updating the parser
lookup tables `parsers_by_name` and `parsers_by_mime_type`. (Benjamin Bannier, Corelight)
* Remove unintentional `return` in `rt::haveEod`. (Benjamin Bannier, Corelight)
* Avoid creating vain global state in `spicy::rt::{
done, isInitialized
}`. (Benjamin Bannier, Corelight)
* Reinitialize default parser when reinitializing spicy::rt library. (Benjamin Bannier, Corelight)
0.4.0-861 | 2020-08-24 13:04:33 +0200
* Diagnose invalid `zlib::Stream`s.
When a `zlib::Stream` enters a failed state it cannot be reused
anymore. Previously we would return empty data whenever such a stream
was used again nevertheless. Since empty data is a valid result of
decompressing non-empty input this leaves users with no good way to
catch such scenarios after the fact.
This patch modifies `zlib::Stream::decompress` to now throw an exception
instead of returning valid but empty data. (Benjamin Bannier, Corelight)
0.4.0-855 | 2020-08-24 10:56:17 +0000
* Fix Cirrus CI file for Alpine 3.12. (Robin Sommer, Corelight)
0.4.0-854 | 2020-08-24 08:41:23 +0000
* Switch Dockerfile for Alpine from 3.11 to 3.12.
* Compile code with debug information also when in release mode.
(Robin Sommer, Corelight)
* Compile the toolchain with -O3 instead of -O2. (Robin Sommer,
Corelight)
* Always link against thread library. The performance of some of our
runtime code depends on whether the executable is linked against
pthread or not. (Robin Sommer, Corelight)
0.4.0-849 | 2020-08-24 08:39:14 +0000
* Fiber runtime library updates. (Robin Sommer, Corelight)
- Fix state tracking in fibers. When taking a fiber from the
cache, we would reset its state on INIT, which then led to
restarting it from scratch instead of reusing the already
running loop.
- Add a function to fill the fiber cache with preinitialized
instances.
- Track number of fiber initialization in statistics.
- Unify fiber debug logging.
0.4.0-843 | 2020-08-21 13:10:42 +0200
* Consistently deal with `std::filesystem` still being experimental in hilti-rt. (Benjamin Bannier, Corelight)
0.4.0-840 | 2020-08-21 10:04:02 +0200
* Prevent use after free in `rt::base64::finish`. (Benjamin Bannier, Corelight)
0.4.0-833 | 2020-08-19 11:06:44 +0200
* Add assertion catching out-of-bounds HILTI module access. (Benjamin Bannier, Corelight)
* Add check to prevent nullptr dereference in `hilti::rt::done`. (Benjamin Bannier, Corelight)
* Add missing context cleanup in `hilti::rt::done`.
We configure a `Context` owned by `GlobalState` in `hilti::rt::init`.
Since we destruct that context in `hilti::rt::done` we also need to
reset the context as we would otherwise leave a dangling refernence
behind. (Benjamin Bannier, Corelight)
* Call out gitlint in commit message style guidelines. (Benjamin Bannier, Corelight)
* Remove guidance on not rebasing development branches. (Benjamin Bannier, Corelight)
* Add developer manual section on pre-commit linters. (Benjamin Bannier, Corelight)
* Add pre-commit linter for commit messages. (Benjamin Bannier, Corelight)
0.4.0-819 | 2020-08-19 11:05:55 +0200
* Explicitly set locale in centos Docker container.
`hilti::rt::init` needs to be able to set the locale on the host system
and will abort with a fatal error if it cannot do so. This patch makes
sure this is possible in the centos-8 Docker container provided by us.
Note: This already worked in BTests since its config file enforces the
needed environment variables. This patch fixes it also for standalone
usage of Spicy (and now e.g., the `locale` command emits no more
errors). (Benjamin Bannier, Corelight)
0.4.0-817 | 2020-08-18 11:24:52 +0200
* Add `hilti::rt::Library::symbol`.
This patch adds a new member function `Library::symbol` which can be
used to obtained a symbol from a specific library.
While currently `Library` users just call `::dlsym` globally to obtain a
symbol, this becomes ambiguous if multiple libraries in the process image
provide similarly named symbols. Instead they could hold handles to
all interesting libraries and obtain symbols with the added function.
With that we could also `::dclose` libraries in the `Library`
destructor; we'll leave that change for a later point. (Benjamin Bannier, Corelight)
* Fix warning potentially emitted from `rt::Library` destructor. (Benjamin Bannier, Corelight)
* Fix format string in `rt::createTemporaryFile`. (Benjamin Bannier, Corelight)
* Extend error message when `rt::Library::save` fails. (Benjamin Bannier, Corelight)
* Change CI job to use non-privileged user.
At least on ubuntu `root` seems to be able to modify even read-only
directories which makes it hard to write tests checking handling of
permission errors.
Change the image used in CI to use a non-privileged user instead. (Benjamin Bannier, Corelight)
* Change directory used for ccache store in CI.
We were originally using `/var/spool/ccache` which could require
extended privileges to create. Switch over to use `/tmp/ccache` which
should be easier to create even for unprivileged users. (Benjamin Bannier, Corelight)
* Prevent `rt::Resumable::get` from leaking non-HILTI exceptions. (Benjamin Bannier, Corelight)
* Make `hilti::rt::Resumable`'s operator bool explicit. (Benjamin Bannier, Corelight)
* Consistently provide stringification for all exceptions. (Benjamin Bannier, Corelight)
* Add comparison operators to `rt::Backtrace`. (Benjamin Bannier, Corelight)
* Fix `to_string_for_print` for `DeferredExpression`.
Since `DeferredExpression` is a template itself we cannot specialize the
master template of the `detail` namespace; instead we need to provide an
overload. (Benjamin Bannier, Corelight)
0.4.0-798 | 2020-08-14 07:25:00 +0000
* CI: Move Zeek package specs back into tasks to work around Cirrus
issue, per https://github.com/cirruslabs/cirrus-ci-docs/issues/692.
(Robin Sommer, Corelight)
0.4.0-796 | 2020-08-14 07:20:31 +0000
* Embed version information into HLTO files that encodes information
about the HILTI version/config they were built with. (Robin
Sommer, Corelight)
We use this for detecting (1) mismatches in the HITLI version used
for building vs running, and (2) running an optimized HLTO file
with a debug runtime. In both cases, the runtime will print a
warning to stderr, but still proceed processing the library.
Note that this is breaking change that will prevent HLTOs built
with earlier versions from loading now.
* Add option -U to tools that summarizes runtime resource
consumption at termination. The measurement excludes time spent
for startup/termination. (Robin Sommer, Corelight)
* Use cout/cerr in debug logger instead of reopening them
separately. Output can get mixed up otherwise if other places
write to cout/cerr directly. (Robin Sommer, Corelight)
* Update nlohmann-json and move to top-level 3rdparty directory.
(Robin Sommer, Corelight)
0.4.0-789 | 2020-08-11 07:47:30 +0000
* Lots of new test coverage for HILTI runtime library: references,
debug logger, configuration, safe-int, iterator, backtrace; along
with related fixes and cleanups. (Benjamin Bannier, Corelight)
0.4.0-737 | 2020-08-11 07:39:31 +0000
* Fix exception handling across library boundaries. (Robin Sommer,
Corelight)
* Allow limiting the "%port" of a unit to a specific direction.
(Robin Sommer, Corelight)
We now allow to add attributes `&originator`/`&responder` to a `%port`
unit property to limit it to the corresponding direction of a session.
When selecting a parser through `--parser <port>` in spicy-driver, one
can now add `%orig` or `%resp` to the port to indicate the desired
directionality. If one leaves that off, only parsers are selected that
have no directionlity defined (i.e., just like it used to be).
* Add batch input mode to spicy-dump. spicy-dump gains an option -F
to process multiple interleaved input streams pre-recorded in a
batch input file. The format of that file is described in the
manual, and there's a Zeek script generating batch files from PCAP
input traces. (Robin Sommer, Corelight)
* When naming a parser to use through "spicy-driver -p XXX", one can
now also specify a port ("80/tcp") or a MIME type
("application/text") if the desired Spicy unit defines a
corresponding property. See the documentation of spicy-driver for
more. (Robin Sommer, Corelight)
* Port Zeek plugin to new driver stateful matching API. (Robin
Sommer, Corelight)
0.4.0-730 | 2020-08-11 07:38:44 +0000
* Switch to Zeek 3.2.0 release packages for CI. (Robin Sommer,
Corelight)
* Add helper Makefile to build and run the CI docker image locallly.
(Robin Sommer, Corelight)
0.4.0-725 | 2020-08-05 12:44:26 +0200
* Remove conversion of `rt::Time` to `bool`. (Benjamin Bannier, Corelight)
* Fix assignment operator for `rt::Union` of non-copyable types. (Benjamin Bannier, Corelight)
* Fix format string error possibly triggered when adding Times and Intervals. (Benjamin Bannier, Corelight)
0.4.0-716 | 2020-08-03 22:24:56 +0200
* Make default-constructed `rt::Address` instances invalid. (Benjamin Bannier, Corelight)
* Make `rt::Address` remember its address family.
This class was previously unable to distinguish IPv4 and IPv6 addresses
with all bits sets to zero from each other. By explicitly storing its
address family we are able to distinguish them, and can also preserve
the address family across operations like masking. (Benjamin Bannier, Corelight)
* Bump doctest to 2.4.0. (Benjamin Bannier, Corelight)
* Validate prefix length inputs when constructing `rt::Network` values. (Benjamin Bannier, Corelight)
0.4.0-703 | 2020-08-03 07:41:51 +0000
* GH-444: Port Zeek plugin to Zeek 3.2. (Robin Sommer, Corelight)
* GH-323: Suppress bogus warning by Zeek when installing our plugin.
(Robin Sommer, Corelight)
* Extend EVT file syntax with @if/@else/@endif preprocessor directives.
For now, we support one type of expression for "@if": a Zeek version check. Example:
@if ZEEK_VERSION < 30200
....
@else
....
@endif (Robin Sommer, Corelight)
Also collecting all the Zeek includes into zeek-compat.h. (Robin Sommer, Corelight)
* Fix frequent off-by-one errors in EVT error message locations.
(Robin Sommer, Corelight)
0.4.0-691 | 2020-07-31 15:19:57 +0200
* Emitted fully typed initializer lists in codegen.
* Correctly handle flipping with zero valid bytes in `rt::integer::flip`. (Benjamin Bannier, Corelight)
* Add additional precondition checks to `rt::integer::bits`. (Benjamin Bannier, Corelight)
* Consistently use `ntohX` type functions for `rt::integer::ntohX`. (Benjamin Bannier, Corelight)
0.4.0-681 | 2020-07-23 14:55:25 +0000
* Fix potential invalid iterator deref in debug output. (Robin Sommer, Corelight)
* Add operator "bytes += uint8" for adding a single byte. (Robin Sommer, Corelight)
* Fix missing filter end-of-input signal. (Robin Sommer, Corelight)
* GH-423: Remove C{,XX}_SYSTEM_INCLUDE_DIRS cmake flags. (Robin
Sommer, Corelight)
If this functionality is needed, set CXX_FLAGS instead during
configure, e.g.:
CXXFLAGS="-isystem /opt/rh/devtoolset-9/root/usr/include/c++/9" ./configure ...
* GH-441: Fix generation of lists of reserved words. (Robin Sommer, Corelight)
* GH-449: Fix documentation example. (Robin Sommer, Corelight)
0.4.0-670 | 2020-07-20 15:25:04 +0000
* GH-415: Add missing type coercion for parameterized types. (Robin
Sommer, Corelight)
* GH-395: Fix tuple constructor expressions to not return a type
until all elements are fully resolved. (Robin Sommer, Corelight)
* GH-402: Allow switch statements without any "case". (Robin Sommer,
Corelight)
* GH-387: Extend a test's life time. (Robin Sommer, Corelight)
0.4.0-661 | 2020-07-16 07:06:14 +0000
* Fix build failures from building against Zeek 3.2 dev versions.
(Tim Wojtulewicz, Corelight)
0.4.0-659 | 2020-07-16 06:34:30 +0000
* GH-352: Add chapter on custom host applications to manual. (Robin
Sommer, Corelight)
* Series of smaller documentation updates. (Robin Sommer, Corelight)
* Add documentation for types time/interval/optional.
* Add list of reserved keywords.
* Update debugging sections.
* Add note about commutativity to operator documentation where applicable.
* Add note about checking if optional unit variables remains unset still.
* Fix typo in documentation for 'print'.
* Fix an operator's description.
* GH-408: Remove "time + time" operator. (Robin Sommer, Corelight)
* GH-404: Add `time_ns(X)` and `interval_ns(X)` expressions to
instantiate values with nanosecond resolution. (Robin Sommer,
Corelight)
* Extend `{hilti,spicy}-config` with a `--dynamic-loading` option.
This adapts `--ldflags` for applications that want to dynamically
load precompiled parsers. (Robin Sommer, Corelight)
* Make `&cxxname` usable for user code: The code generator now
automatically adds prototypes for the corresponding C++ functions.
To optionally disable that, a new attribute `&have_prototype` has
been added. We also no longer require explicit global scoping for
`&cxxname` IDs. (Robin Sommer, Corelight)
* Fix joint usage of options `spicyc` options `-cKo`. This would end
up saving only the linker code, not the combined parsing + linker
code, in the specific output file. (Robin Sommer, Corelight)
0.4.0-645 | 2020-07-16 06:33:23 +0000
* GH-423: Update configure's usage message. (Robin Sommer,
Corelight)
* GH-433: Add support for declaring constants with explicit types.
(Robin Sommer, Corelight)
0.4.0-641 | 2020-07-14 06:13:11 +0000
* GH-429: Fix generation of C++ prototypes for host applications.
(Robin Sommer, Corelight)
0.4.0-639 | 2020-07-13 10:01:31 +0000
* GH-400: Fix build with static libraries. (Robin Sommer, Corelight)
* Fix ill-defined order of internal HILTI plugin execution. (Robin
Sommer, Corelight)
* Remove JIT's {init,finish}Runtime() methods. They were no-ops
already. (Robin Sommer, Corelight)
0.4.0-634 | 2020-07-09 12:32:17 +0200
* Lots of new & updates unit tests for the HILTI runtime. (Benjamin Bannier, Corelight)
* Remove `AddressFamily::` prefix when stringifying values. (Benjamin Bannier, Corelight)
0.4.0-612 | 2020-07-08 15:21:20 +0000
* GH-39: Add `spicy-dump`, a new standalone command line application
that compiles and executes Spicy parsers on the fly and prints out
the parsed information at the end in either a readable, custom
ASCII format, or as JSON. (Robin Sommer, Corelight)
#echo "GET /index.html HTTP/1.0" | spicy - dump my - http.hlto
MyHTTP::RequestLine {
method:
GET uri : / index.html version : MyHTTP::Version {
number:
1.0
}
}
#echo "GET /index.html HTTP/1.0" | spicy - dump - J my - http.hlto
{
"method" : "GET", "uri" : "/index.html", "version" : { "number" : "1.0" }
}
* GH-56: Add reflection API to libhilti for dynamic, runtime
inspection of parsed units. This includes:
* A "typeinfo(X)" expression for HILTI that, at runtime, wwill
yield a pointer to a new `hilti::rt::TypeInfo`` struct
describing the type of `X`, which can be a type or an
expression.
* `include/rt/type-info.h` builds out a comprehensive set of
classes to describe HILTI types, and to access their values
generically, at runtime.
* A 3rd kind of host application-side parse function for public
units. Spicy's driver's processInput() now uses that.
* Change string rendering of ports to skip "Protocol::" prefix.
(Robin Sommer, Corelight)
* Fix bug in move assignment operator of ValueReference. (Robin
Sommer, Corelight)
* Add stringification for exception type. (Robin Sommer, Corelight)
* Add stringification for "any" type. (Robin Sommer, Corelight)
* Add missing &internal attributes to some codegenerated struct
fields. This hides them from user visible output. (Robin Sommer,
Corelight)
* Add equality operator to regexp type. (Robin Sommer, Corelight)
* Add operator<< for regexp. (Robin Sommer, Corelight)
* Support anonymous namespaces in C++ output. (Robin Sommer,
Corelight)
* Add missing printer code for iterator<set> and iterator<map>.
(Robin Sommer, Corelight)
* Add const version of vector iterator's dereference operator.
(Robin Sommer, Corelight)
* Add -O0 to libhilti/libspicy for debug builds. (Robin Sommer,
Corelight)
0.4.0-572 | 2020-07-08 11:36:53 +0200
* Fix parsing of `rt::Port` for out-of-range port values. (Benjamin Bannier, Corelight)
0.4.0-552 | 2020-07-03 15:47:06 +0200
* Switch `real::unpack` error reporting to use `rt::Result`. (Benjamin Bannier, Corelight)
0.4.0-549 | 2020-07-01 15:03:27 +0200
* Remove unintended conversion of `rt::Result` to `bool`.
0.4.0-547 | 2020-07-01 11:41:06 +0200
* GH-392: Fix handling of vector of bools. (Benjamin Bannier, Corelight)
0.4.0-544 | 2020-07-01 09:35:52 +0200
* Remove `hilti::rt::List` in favor of `hilti::rt::Vector`. (Benjamin Bannier, Corelight)
0.4.0-538 | 2020-06-30 08:08:16 +0000
* Fixed semantics of `hilti::rt::Interval`-bool conversions. They
now return true for non-zero intervals. (Benjamin Bannier,
Corelight)
* Add tests of `rt::Interval` stringification. (Benjamin Bannier, Corelight)
* Add tests of `result::Error` stringification. (Benjamin Bannier, Corelight)
0.4.0-532 | 2020-06-30 08:03:40 +0000
* Add a DHCP analyzer. (Benjamin Bannier, Corelight)
* Fix filename in getting started docs. (Dirk Loss)
* Move reporting logic for --report-times so that it always executes
when the HILTI-side driver destructs. (Robin Sommer, Corelight)
* Fix header inclusion in Spicy plugin. (Benjamin Bannier, Corelight)
* Remove unused `is_optional` type trait. (Benjamin Bannier,
Corelight)
* Enable translation of optional types on the Spicy-Zeek interface.
(Benjamin Bannier, Corelight)
0.4.0-525 | 2020-06-26 14:55:08 +0200
* Add dedicated method to resize and assign a vector. (Benjamin Bannier, Corelight)
0.4.0-522 | 2020-06-24 22:24:34 +0200
* Fix `hilti::rt::is_optional` so it applies to const and ref of optionals also. (Benjamin Bannier, Corelight)
0.4.0-520 | 2020-06-24 06:59:04 +0000
* GH-398: Remove duplicate spicy-driver CMake target. (Benjamin
Bannier, Corelight)
0.4.0-518 | 2020-06-24 06:50:48 +0000
* GH-140: Add "&requires"/"%requires" unit attribute/property.
(Robin Sommer, Corelight)
Fields may now use an attribute ``&requires=EXPR`` to enforce
additional constraints on their values. ``EXPR`` must a be boolean
expression that will be evaluated after the parsing for the field
has finished, but before any hooks execute. If ``EXPR`` returns
``False``, the parsing process will abort with an error, just as
if the field had been unparsable in the first place.
One can similarly enforce conditions globally at the unit level
through a property ``%requires = EXPR``, which will be evaluated
once the unit has been full parsed, but before any ``%done`` hook
executes. If ``EXPR`` returns ``False``, the unit's parsing
process will abort with an error.
* Add Doxygen output to Sphinx documentation. It's integrated into
the "Read the Docs build". Locally, "make doxygen" will run it.
(Benjamin Bannier, Corelight)
* Clean up Sphinx custom extension. (Benjamin Bannier, Corelight)
- Remove use of `sphinx.locale.l_`
- Format with autopep8.
0.4.0-510 | 2020-06-23 16:16:43 +0200
* GH-386: Add codegen for Spicy time difference. (Benjamin Bannier, Corelight)
0.4.0-508 | 2020-06-22 06:03:34 +0000
* Series of improvements for utility infrastructure (Benjamin Bannier, Corelight)
- Add tests for `util::escapeBytes`.
- Add tests for some functions in `hilti::rt::util`.
- Extend list of supported escape characters in hilti::rt escaping functions.
- Fix `rt::split` for trailing delimiters.
- Fix edge cases rt::atoi_n.
- Fix typos in `hilti/rt/util.h`.
- Make `rt::util::tuple_for_each` also work on rvalues.
- Make documentation for `hilti::rt::rtrim` more specific.
- Remove unused `escape_control` argument to `util::escapeBytes`.
- Use raw character literals in some `hilti::rt::utils` tests.
* Fix extraction of event priority in EVT files. (Benjamin Bannier, Corelight)
0.4.0-496 | 2020-06-18 15:46:44 +0000
* GH-381: Have the code generator fully declare enum types directly
as part of the forward-declarations block. (Robin Sommer, Corelight)
* GH-380: Corrections to tutorial. (Duffy Ocraven)
* Adding short "Next Steps" section to tutorial. (Robin Sommer, Corelight)
0.4.0-492 | 2020-06-17 10:41:13 +0200
* Move hilti's `util` namespace into the `hilti` namespace. (Benjamin Bannier, Corelight)
0.4.0-490 | 2020-06-17 10:36:36 +0200
* GH-228: Prevent more overflows when working with hilti::rt Time and Interval (Benjamin Bannier,
Corelight)
- Change `hilti::rt::Interval` to use safe integers.
- Change `hilti::rt::Time` to use safe integers.
- Fix some typos in `hilti/base/util.h`.
- Remove hilti-rt-tests dependency on hilti library.
- Reorganize test cases for `hilti::util`.
0.4.0-484 | 2020-06-16 07:15:25 +0000
* Add new tutorial chapter. (Robin Sommer, Corelight)
* Add Zeek-side option "Spicy::enable_print" to control whether the
output of Spicy-side "print" output is shown. Default is off
(which is a change in behaviour compared to previous versions).
* Clean up representation of type arguments and function calls in
auto-generated operator reference. (Robin Sommer, Corelight)
* Make the target stream of hilti::print() configurable. Default
remains standard output. Leaving it unset will silence
hilti::print(). (Robin Sommer, Corelight)
* Do not use use "using X === ..." for forward-declaring struct
types in C++ output, as that could accidentally pull in IDs from
outer namespaces. (Robin Sommer, Corelight)
* Fix scanner for accepting comment lines at end of file without
terminating newline. (Robin Sommer, Corelight)
0.4.0-477 | 2020-06-15 07:36:43 +0000
* GH-293: Fix precedence of "in" operator. (Robin Sommer, Corelight)
* GH-319: When using &convert with a list expression, transparently
turn the result into a vector. (Robin Sommer, Corelight)
* GH-324: Fix error message that was too specific about source type.
(Robin Sommer, Corelight)
* GH-325: Fix a couple of segfaults in scope builder when hitting
invalid code. (Robin Sommer, Corelight)
* Update spicy-driver usage message. (Robin Sommer, Corelight)
* Handle unknown options in getopt() consistently in driver code.
(Robin Sommer, Corelight)
* GH-344: Type check assignments and fix constness propagation.
(Robin Sommer, Corelight)
* GH-345: Validate type of &default expressions. (Robin Sommer,
Corelight)
* Add more debug information about constness to AST. (Robin Sommer,
Corelight)
0.4.0-467 | 2020-06-15 07:18:03 +0000
* Redo testing of `hilti::rt::Stream` and clean up implementation (Benjamin Bannier, Corelight)
- Add Doxygen documentation to `rt::Stream`.
- Add `begin` and `end` methods for `rt::list::Empty`.
- Add additional nullptr checks when accessing `rt::Stream` iterator chunks or contents.
- Fix escaping of `rt::Stream` and `rt::View` in stringification
to ensure all non-printable byte sequences are escaped.
- Make `rt::stream::View` iterators be safe by default.
- Prevent nullptr dereference in `rt::Stream` iterator `chunk` and `content`.
- Remove `rt::Stream::safe_[begin|end]`.
- Remove `rt::stream::View::safe_[begin|end]`.
- Remove `safe_begin` and `safe_end` extension points.
- Remove unused `Chunk::begin`.
- Remove unused `rt::Stream::chain` function.
- Remove unused `rt::Stream::compare` functions.
- Remove use of unsafe iterators in a few places.
- Rename `rt::iterator::safe_range` to drop the 'safe' qualifier.
- Add lots of tests new cases, and reorganize exists test cases.
* Add `begin` and `end` methods for `rt::list::Empty`. (Benjamin
Bannier, Corelight)
0.4.0-430 | 2020-06-12 08:07:16 +0200
* Reenable some already passing clang-tidy checks. (Benjamin Bannier, Corelight)
0.4.0-428 | 2020-06-12 08:06:23 +0200
* GH-274: Unbreak build for when other Spicy headers are already installed.
If Spicy headers were already installed into a path somewhere in the
include lookup, we could have found the same header twice, preferring
the installed header over the one of the current checkout. This could
have lead to diagnostics about duplicate definitions since due to use of
`#pragma once` in our headers the compiler would not have detected that
a header was already included (and even if it would have detected this,
it would have included the incorrect header).
This patch adds include paths from the current checkout before any
system paths.
0.4.0-426 | 2020-06-09 17:16:13 +0000
* Increase CI memory limit for Ubuntu Docker further. (Robin Sommer,
Corelight)
* GH-352: Fix Zeek link in README. (Robin Sommer, Corelight)
0.4.0-424 | 2020-06-09 15:22:26 +0000
* GH-355: Allow access to unit parameters inside default expressions for unit
variables. (Robin Sommer, Corelight)
* GH-315: Fix autogen-spicy-lib to extract multi-line comments.
(Robin Sommer, Corelight)
0.4.0-417 | 2020-06-08 10:26:32 +0000
* Increase CI memory limit for GCC further. (Robin Sommer, Corelight)
* Fix GCC warning. (Robin Sommer, Corelight)
0.4.0-415 | 2020-06-08 08:28:19 +0000
* GH-368: Fix &convert for vectors of enums. (Robin Sommer, Corelight)
* Add missing HILTI printer code for "unset" operator. (Robin
Sommer, Corelight)
0.4.0-411 | 2020-06-05 07:28:40 +0000
* GH-361: Fix construction of vectors with element types that
provide default values. (Robin Sommer, Corelight)
* GH-365: Zeek plugin: Catch if more parameters are given in an EVT
spec than Zeek expects. (Robin Sommer, Corelight)
* Remove branch name from Cirrus CI emails due to API limitations.
(Benjamin Bannier, Corelight)
0.4.0-406 | 2020-06-04 09:14:27 +0000
* GH-354: No longer allow dashes inside identifiers. (Robin Sommer, Corelight)
* GH-350: Restrict parsing of attributes and properties to avoid
ambiguities. (Robin Sommer, Corelight)
* Clean up the expression mode in the Spicy Bison parser. (Robin Sommer, Corelight)
This comes with a couple of grammar changes to avoid ambiguitues:
- "new X" no longer allows full expressions for X, just values or
identifiers.
- Switch cases that use look-ahead now also need to be introduced
with an arrow, e.g.:
switch {
->a : A;
->b : B;
->c : C;
};
* Reduce resources requested from Cirrus. (Robin Sommer, Corelight)
* Make iterators into vectors ordered. (Benjamin Bannier, Corelight)
* GH-343: Fix duplicate evaluation of if-condition when used with
&parse-at. (Robin Sommer, Corelight)
* Add validation check for initializations that use the ID that's
currently being declared. (Robin Sommer, Corelight)
* Fix mailing list links in README. (Robin Sommer, Corelight)
0.4.0-393 | 2020-06-04 08:41:28 +0200
* Rework implementation of `hilti::rt::Bytes`.
- Add test cases for `rt::Bytes` member functions.
- Fix construction of `rt::Bytes` from string and charset.
- Fix handling of undefined byte order in ByteOrder conversions to integers.
- Prevent throwing `std::out_of_range` from `rt::Bytes::extract`.
- Add additional safety to `bytes::Iterator`.
- Made some `rt::Bytes` methods `const`.
- Remove rvalue overload of `rt::Bytes::str`.
* Fix ADL for `rt::to_string`. (Benjamin Bannier, Corelight)
* Fix `rt::util::split` for empty delimiter case. (Benjamin Bannier, Corelight)
0.4.0-364 | 2020-06-04 08:40:25 +0200
* Make functions of `rt::string` throw instead of triggering internal errors
for invalid input. (Benjamin Bannier, Corelight)
0.4.0-362 | 2020-06-02 15:20:59 +0000
* GH-282: Set PATH when building documentation so that we find the
Spicy tools. (Robin Sommer, Corelight)
* GH-348: Fix integer overflows in regexp parsing & matching.
(Robin Sommer, Corelight)
0.4.0-356 | 2020-05-28 10:23:04 +0000
* Small documentation updates, including some notes on DPD
integration and standalone plugin builds. (Robin Sommer,
Corelight)
0.4.0-348 | 2020-05-27 13:03:22 +0200
* GH-335: Add http_all_headers() event to HTTP EVT file.
The event isn't used anywhere by default in Zeek, but it's a nice
example of transforming Spicy information on the fly to match a
Zeek-side event signature.
# 335(Robin Sommer, Corelight)
0.4.0-346 | 2020-05-27 13:00:28 +0200
* jrx: Fix leak in `jrx_match_state_done`. (Benjamin Bannier, Corelight)
* jrx: Fix test harness. (Benjamin Bannier, Corelight)
* Fix leaks of jrx state when copying `regexp::MatchState`. (Benjamin Bannier, Corelight)
* jrx: Allow copying from const. (Benjamin Bannier, Corelight)
* Bump doctest to 2.3.8. (Benjamin Bannier, Corelight)
0.4.0-325 | 2020-05-27 06:49:26 +0000
* GH-333: Validate that unit fields have parseable types. (Robin Sommer, Corelight)
* GH-330: Add range check to port parsing in EVT files. (Robin
Sommer, Corelight)
* GH-335: Add support for converting maps from Spicy to Zeek. (Robin
Sommer, Corelight)
* Send emails for successful CI builds. (Benjamin Bannier, Corelight)
0.4.0-318 | 2020-05-26 10:07:38 +0000
* Change Slack links to point to Zeek's Connect page. (Robin Sommer, Corelight)
0.4.0-317 | 2020-05-25 09:54:25 +0200
* GH-286: Fix regexp handling to respect &size. (Benjamin Bannier, Corelight)
0.4.0-311 | 2020-05-21 07:54:43 +0000
* Fix JIT-less build of Zeek plugin. (Robin Sommer, Corelight)
0.4.0-309 | 2020-05-20 12:35:36 +0000
* GH-299: Support compiling the Zeek plugin without JIT support.
(Robin Sommer, Corelight)
If configured with `--disable-jit`, the Zeek plugin will now still
be built and will support loading pre-compiled HLTO files. The
plugin will have dependencies only on system libraries, and hence
can be installed on systems that don't have the Spicy libraries
available.
There's also a new configure option `--disable-jit-for-zeek` that
disables JIT support just for the Zeek plugin, creating the same
dependency-free result.
* GH-83: Add standalone tool spicyz to precompile analyzers for
Zeek. (Robin Sommer, Corelight)
This replaces the previous shell wrapper around Zeek, meaning that
we can now precompile Zeek analyzers without actually relying on
having a Zeek binary itself available. (We continue to require
access to the Zeek development headers.)
As a bonus feature, the new spicyz comes with an option "-c
<prefix>" that, instead of producing a fully compiled HLTO, just
writes out all the generated C++ code into files, each named
"<prefix>-<module ID>.cc". This goes along way towards supporting
compiling Spicy analyzers statically into an application, similar
to how BinPAC is being used (GH-72).
* Fix for Bison versions >= 3.6. (Robin Sommer, Corelight)
0.4.0-295 | 2020-05-20 07:23:14 +0000
* Fix a couple typos in documentation. (Robin Sommer, Corelight)
0.4.0-293 | 2020-05-19 06:57:49 +0000
* Rework implementation of `hilti::rt::List`. (Benjamin Bannier, Corelight)
- Hide that `rt::List` inherits from `std::list` from users.
- Make iterators of `rt::List` safe.
- Add tests for `rt::List`.
0.4.0-286 | 2020-05-18 10:38:44 +0000
* Remove FAQ pointer from README. (Robin Sommer, Corelight)
* Fix bug in EVT reading. (Robin Sommer, Corelight)
0.4.0-284 | 2020-05-18 08:29:49 +0000
* GH-237: Add library function `spicy::strftime`. (Benjamin Bannier,
Corelight)
0.4.0-282 | 2020-05-18 08:28:08 +0000
* GH-312: Fix comment parsing in EVT files. (Robin Sommer,
Corelight)
0.4.0-280 | 2020-05-15 07:11:01 +0000
* Rework implementation of `hilti::rt::Map`. (Benjamin Bannier, Corelight)
- Hide that `rt::Map` inherits from `std::map` from users.
- Make iterators of `rt::Map` safe.
- Slim down `AssignProxy`.
- Fix `rt::Map::get`.
- Add test for `rt::Map` stringification.
* Fix display of compiler flags in JIT debug log. (Benjamin Bannier, Corelight)
* Document using Docker Hub images. (Benjamin Bannier, Corelight)
0.4.0-270 | 2020-05-14 07:00:54 +0000
* Clean up example code. (Robin Sommer, Corelight)
* Change structure of example section, and add more links. (Robin
Sommer, Corelight)
* Make autogen-docs script more robust. (Robin Sommer, Corelight)
* Fix documentation of -= and /= opertators. (Robin Sommer, Corelight)
0.4.0-264 | 2020-05-13 14:18:08 +0000
* Add an end-to-end TFTP analyzer for Zeek, it comes with a Zeek
script producing a typical ``tftp.log`` log file. This analyzer is
a good introductory example because the Spicy side is pretty
straight-forward. (Robin Sommer, Corelight)
* GH-306: Do not fail documentation build for broken links.
(Benjamin Bannier, Corelight)
0.4.0-260 | 2020-05-12 10:37:16 +0000
* Documentation updates. (Robin Sommer, Corelight)
- GH-230: Fix a series of smaller inaccuracies and typos. Reported by Duffy Ocraven.
- Document list type.
- Include CHANGES into the documentation's release notes.
- Update link to macOS LLVM binaries. (Robin Sommer, Corelight)
* Clean up parsing of list constructor. (Robin Sommer, Corelight)
* Clean up cast operators. (Robin Sommer, Corelight)
* Remove a couple of empty source files. (Robin Sommer, Corelight)
0.4.0-251 | 2020-05-12 10:33:24 +0000
* Redo implementation of `hilti::rt::Set`. (Benjamin Bannier, Corelight)
- Hide that `rt::Set` inherits from `std::set` from users.
- Make iterators of `rt::Set` safe.
- Add test of rt::Set stringification.
0.4.0-245 | 2020-05-11 08:39:49 +0000
* Redo implementation of `hilti::rt::Vector`. (Benjamin Bannier, Corelight)
- Hide that `rt::Vector` inherits from `std::vector` from users.
- Replace iterators of `rt::Vector` with safe alternatives.
- Add Doxygen comments to `rt::Vector`.
- Remove support for stringification of `std::vector` in hilti::rt.
- Add test for stringification of `Empty` vector.
- Make `rt::MIMEType` default-constructible.
- Fix includes of `rt/mime.h`.
- Use less ambiguous form for generated `rt::Vector` constructors.
* Require Zeek plugin for Zeek protocol parser tests. (Benjamin
Bannier, Corelight)
0.4.0-234 | 2020-05-11 08:33:55 +0000
* GH-229: Add a Spicy function to render bytes as hex string.
(Benjamin Bannier, Corelight)
## Returns a bytes value rendered as a hex string.
public function bytes_to_hexstring(value: bytes) : string;
*GH - 200 : Add library function `spicy::getenv()` to retrieve environment variables.(Robin Sommer, Corelight)
##Returns the value of an environment variable,
if set.public function getenv(name
: string)
: optional<string>;
* GH-200: Add option -f to `spicy-driver` to read input from file.
(Robin Sommer, Corelight)
* Use fully qualified IDs for internal `&cxxname` in libraries
`hilti.hlt` and `spicy.spicy`. (Benjamin Bannier, Corelight)
* Fix shell commands in `docker/Makefile`. (Benjamin Bannier,
Corelight)
0.4.0-226 | 2020-05-06 15:12:52 +0000
* Apply a set of tweaks to specifics of &size. (Robin Sommer,
Corelight)
- We now enforce that a field consumes all its &size characters of
input, unless explicitly overridden where a types allow for that
(as, e.g., "bytes" does with an additional &eod).
- We slightly change semantics for parsing bytes: A &size of zero
will now set the field an empty bytes value; it used to leave
the field unset.
- Document that &size works with all fields (which it already
did).
- Clean up the implementation of &size.
* GH-289: Do not perform look-ahead parsing for vectors fed from
&parse-from/at. (Robin Sommer, Corelight)
* GH-288: Rename "&until_including" attribute to "&until-including".
(Robin Sommer, Corelight)
* GH-269: Add support for "&until-including" when parsing bytes.
(Robin Sommer, Corelight)
* GH-286: Fix Bison parser's attribute association for regular
expressions. (Robin Sommer, Corelight)
* Add an "unset" operator to HILTI's struct type to clear optional
types. (Robin Sommer, Corelight)
0.4.0-216 | 2020-05-05 12:25:40 +0000
* GH-212: Resize eagerly when accessing out-of-bounds Vector
elements. (Benjamin Bannier, Corelight)
0.4.0-214 | 2020-05-04 10:51:06 +0000
* GH-267: Fix spicy-build to support absolute output paths. (Robin
Sommer, Corelight)
* GH-275, GH-276: Sort options in the usage messages of various
tools. (Robin Sommer, Corelight)
0.4.0-208 | 2020-05-04 10:49:21 +0000
* Use doctest test suites as top-level grouping. (Benjamin Bannier,
Corelight)
* GH-268: Properly escape data when stringifying streams. (Benjamin
Bannier, Corelight)
0.4.0-203 | 2020-05-03 15:37:30 +0000
* GH-265: Require unit properties to end with a semicolon. We were
inconsistent with that. Note that this will break existing code
that wasn't using the semicolons, including some examples from the
documentation (which have been updated). For example, it's now
`%filter;` instead of `%filter`. (Robin Sommer, Corelight)
* GH-266: Fix missing validation of unit properties. (Robin Sommer,
Corelight)
* GH-262: Allow signs in intN/uintN() constructors. (Robin Sommer,
Corelight)
* GH-254: Make UTF8 default for bytes::decode(). (Robin Sommer,
Corelight)
* GH-248: Fix AST inconsistency with how modules were stored in
scopes. (Robin Sommer, Corelight)
* GH-263: Improve error messages for unresolved operators and IDs.
(Robin Sommer, Corelight)
* Improve suppression of repetitive errors. (Robin Sommer,
Corelight)
* Avoid leaking passwords into command lines in `docker/Makefile`.
(Benjamin Bannier, Corelight)
0.4.0-189 | 2020-04-30 10:33:54 +0000
* GH-196: Reject container constructors with inconsistent types, and
coerce elements once the type is known. (Robin Sommer, Corelight)
0.4.0-187 | 2020-04-30 10:26:13 +0000
* Improve `vector` implementation. (Benjamin Bannier, Corelight)
- Fix semantics of safe mutating access to `hilti::rt::Vector` elements.
- Remove broken `rt::Vector::set`.
- Fix includes of `hilti/rt/types/vector.h`.
- Add unit test for a couple of `hilti::rt::Vector` methods.`
* GH-272: Document precompiled MacOS packages. (Benjamin Bannier, Corelight)
0.4.0-179 | 2020-04-29 15:15:14 +0200
* GH-192: Remove internal errors that we ended up running into during AST
validation. (Robin Sommer, Corelight)
* GH-178: Fix spurious semicolons in C++ output and potential problem in C++
block rendering (Robin Sommer, Corelight)
0.4.0-173 | 2020-04-29 08:20:21 +0000
* Report column information in error messages. Locations are emitted
as `<line>:<character>` so that e.g., the second character on the
fourth line would be displayed as `4:2`. This extends to ranges as
well which look e.g., like `1:1-2:30`. (Benjamin Bannier,
Corelight)
0.4.0-171 | 2020-04-28 19:08:50 +0000
* Add a section to the development manual on the structure of the
runtime libraries. (Robin Sommer, Corelight)
* Fix `hilti::rt::to_string` for character literals. (Benjamin
Bannier, Corelight)
* GH-179: Enable `hilti::rt::to_string` for `std::vector`. (Benjamin
Bannier, Corelight)
* Remove return of rvalue when generating parse functions. (Benjamin
Bannier, Corelight)
0.4.0-162 | 2020-04-28 10:14:01 +0000
* GH-219: Clean up string interface exposed by `rt::Bytes` to
remove overloading ambiguities. (Benjamin Bannier, Corelight)
* GH-257: Fix documentation whitespace issues and remove outdated
FIXMEs around usages of `Result`. (Benjamin Bannier, Corelight)
0.4.0-154 | 2020-04-27 11:58:00 +0000
* GH-242: Suppress any output from doc's `make check` by default.
(Benjamin Bannier, Corelight)
* GH-251: Ignore package links in documentation linkcheck.
(Benjamin Bannier, Corelight)
* Edit pass on documentation's reassembly section. (Robin Sommer,
Corelight)
0.4.0-148 | 2020-04-24 13:19:22 +0000
* GH-226: Add CI setup for building published Docker images. This
patch adds an `update_dockerhub` target to `docker/Makefile`,
which builds, tests and uploads images to Dockerhub. It also
changes the Dockerfiles in `docker/` back to include Spicy build
commands, so that they become practically self-documenting
(Benjamin Bannier, Corelight).
* GH-222: Output changes to make rendered ASTs easier to parse
(Benjamin Bannier, Corelight)
0.4.0-140 | 2020-04-24 09:18:51 +0000
* Tune update-changes configuration. (Robin Sommer, Corelight)
* GH-245: Fix typo in documentation. (Dominik Charousset)
0.4.0-136 | 2020-04-23 16:53:22 +0000
* Update link in README. (Yacin Nadji)
* Fix link to FAQ in README.
* Remove left-over debug output from DNS grammar.
0.4.0-130 | 2020-04-23 10:44:12 +0000
* GH-170: Change "+<int>" to create a signed integer value. It used
to create an unsigned integer.
* GH-225: Preserve parentheses inside expressions during code
generation.
* GH-231: Add range check to digits in Spicy's atoi_n().
* GH-210: Catch & report broken escape sequences in HILTI/Spicy code
at parse time.
* GH-217: Fix code generation of escaped C++ string literals.
0.4.0-120 | 2020-04-22 15:40:34 +0000
* Fix package artifact links in docs.
* Fix documentation links to Spicy mailing lists.
* Fix issue base URL in documentation.
* Add documentastion build target to check links.
* Fix package creation in CI.
0.4.0-114 | 2020-04-22 13:17:05 +0000
* Make sure auto-generated documentation items have stable ordering.
* Cache CI ccache even if build failed.
0.4.0-111 | 2020-04-22 10:23:15 +0000
* Build documentation from documentation folder instead of from a
symlink inside the build directory.
0.4.0-109 | 2020-04-21 11:40:41 +0000
* Execute CI jobs automatically.
* Fix potential sign-unsigned mismatch in comparison.
* Fix unused variable warning.
0.4.0-105 | 2020-04-21 10:41:25 +0000
* Update documentation links in README.
0.4.0-104 | 2020-04-21 10:15:14 +0000
* Tweak documentation build for standalone operation.
* GH-172: Re-add missing autogenerated usage for spicy-build.
* Add map & set to the auto-documented types.
0.4.0-96 | 2020-04-21 08:27:51 +0000
* GH-129: Add infrastructure and documentation for providing binary
packages.
0.4.0-91 | 2020-04-21 07:38:18 +0000
* GH-174: Fix a number of smaller documentation issues.
* GH-191: Change bytes iterator's deref operator to return uint8
instead of uint64, and fix implementation to no longer use a
signed integer ingternally.
* GH-153: Prettify error messages for fmt() problems.
* GH-154: Fix crash when accessing an unknown tuple element.
* GH-171: Fix spelling in some diagnostic messages.
* GH-175: Catch unresolved unit fields during validation.
* GH-198: Show functions as "<function>" when rendering them.
* GH-203: Add type check for elements in vector/set/map constructors.
* GH-204: Remove support for legacy syntax for parsing sequences.
"vector<T>" is no longer supported for parsing sequences; one must
now use the new "T[]" syntax instead.
* Remove Spicy's legacy list type (which was just an alias for
"vector").
* GH-181: Validate correct usage of struct/unit parameters.
* Add a number of validation checks for unit hooks.
* GH-195: Add range check for bytes' to_int() base parameter, which
now must be between 2 and 36.
* GH-155: Overhaul AST error reporting for more concise error
messages.
* GH-119: Generally include location information when throwing
runtime exceptions.
* Add packaging to CMake setup and CI jobs.
* Set proper CMake project version.
* Execute installation tests in Docker CI tasks.
0.4.0-48 | 2020-04-15 16:56:28 +0000
* GH-163: Fix initialization of &default struct parameters.
* GH-156: Fix look-ahead parsing for vectors terminated through
end-of-data.
* GH-182: Fix "for" loop's type resolution for the iteration variable.
* GH-180: Fix printing of bytes instances that have non-printable
characters. This also changes how we print "\n/\r/\t" inside
bytes: We now render them as "\xXX", like any other non-printable
character.
* Add missing 'Undef' to byte-order enum.
* Add clang-format to pre-commit config.
* Add minimal CI test runs for CentOS 8, Alpine 3.11, and Ubuntu
19.10.
* Clean up our Docker setup/usage.
0.4.0-29 | 2020-04-15 09:04:28 +0000
* Add pre-commit check that generated docs are up-to-date.
* Clean up spicy-doc-to-rst and make output stable.
* Add auto-generated docs to the tree so that one can build the full
documentation without needing to build Spicy itself.
0.4.0-20 | 2020-04-09 15:11:43 +0000
* GH-159: Properly close quotes and parens in "hilti-flow" debug
statement.
* GH-146: Validate that &byte-order attributes have an argument.
0.4.0-16 | 2020-04-09 15:06:04 +0000
* GH-92: Documentation updates.
0.4.0-13 | 2020-04-09 11:23:15 +0000
* Copy edit "Getting Started".
0.4.0-11 | 2020-04-09 10:56:29 +0000
* GH-145: Fix underlinking of hilti-rt library.
* GH-147: Add a configure time check validating that submodules are
in place.
* Add more CI configurations, including for macOS.
0.4.0 | 2020-04-07 14:50:17 +0000
* Starting CHANGES.
Reference in New Issue View Git Blame Copy Permalink