End to end Bazel + Mayhem example.

NOTE: To use rules_fuzzing, you must have Python 3.9. You can use pyenv to switch:

pyenv install 3.9.0
pyenv global 3.9.0

Mayhem Config Home

It is required to authenticate against the Mayhem server. This is done with a Mayhem configuration file, and specified with the config home directory. To set your Mayhem config home on Linux, pass:

--action_env=XDG_CONFIG_HOME="$HOME/.config"

to your bazel build command or your .bazelrc. On Windows, pass:

--action_env=XDG_CONFIG_HOME="%USERPROFILE%\.config"

Note that using the default user home directory is not required, as long as the mayhem config file is in the specified directory. The remainder of this document assumes a Mayhem config file in the user's home directory on Linux.

Uninstrumented target

To build calculator target

bazel build //main:calculator

To package calculator target

bazel build //mayhem:package_calculator

To run calculator target

bazel build --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //mayhem:run_package_calculator 

The above works with the test target //test:test_calculator as well.

LibFuzzer target

To build libfuzzer target

bazel build --config=libfuzzer //fuzz:fuzz_calculator

To run libfuzzer target locally

bazel run --config=libfuzzer //fuzz:fuzz_calculator_run

To build docker image with libfuzzer target

bazel build --config=libfuzzer //mayhem:fuzz_calculator_image

To push docker image with libfuzzer target

bazel run --config=libfuzzer //mayhem:push_fuzz_calculator_image

To run Mayhem against docker image for libfuzzer target

bazel build --config=libfuzzer --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //mayhem:run_fuzz_calculator_image

Regression Testing

To run Mayhem on regression tests only, wait for the run to finish, and output a SARIF report

bazel build --config=libfuzzer --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //mayhem:run_test_calculator_package

Code Coverage

You can download code coverage generated by Mayhem with the mayhem_download() rule:

mayhem_download(
    name = "download_combined_test_calculator_results",
    owner = "training",
    project = "mayhem-bazel-example",
    target = "test_calculator_package",
    output_dir = "combined_test_calculator-pkg",
    testonly = True,
)

Then, run the following command to download the code coverage:

bazel build --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //test:download_combined_test_calculator_results

The code coverage will be downloaded to bazel-bin/test/combined_test_calculator-pkg/coverage.tgz. If you'd like to generate coverage manually, you can do this with Bazel, but you'll have to run each individual file under the testsuite directory. An easy way to do this is:

for test in $(ls ./bazel-bin/test/combined_test_calculator-pkg/testsuite); do bazel coverage --combined_report=lcov --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //test:combined_test_calculator --test_arg=test/combined_test_calculator-pkg/testsuite/$test; done

Local testing

If you don't want to generate coverage but just want to run the tests locally, simply change bazel coverage to bazel test:

for test in $(ls ./bazel-bin/test/combined_test_calculator-pkg/testsuite); do bazel test --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //test:combined_test_calculator --test_arg=test/combined_test_calculator-pkg/testsuite/$test; done

Gtest Integration

Under the test directory, there are a couple of examples of how to integrate Gtest with Mayhem.

• test_calculator shows basic unit testing without any infrastructure such as Google Test.

• gtest_calculator is a simple example of using Gtest, with test fixtures for each function in Calculator.

• combined_test_calculator extracts the test fixture behavior into a test function, and conditionally either executes Gtest or the test functions with generated inputs.

You can run this on Mayhem with:

bazel build --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //mayhem:run_test_calculator_package

• harness_utils_test_calculator is an example inlining harness functionality via a HARNESS macro. The test functions are designed to take a buffer and size, and the HARNESS macro automatically calls the test function with the generated inputs.

• fuzzing_utils_test_calculator declares a FUZZ_TEST fixture that providers a fuzzed data provider. The FUZZ_TEST fixtures are conditionally called based on the presence of a file on the command line; otherwise, the TEST fixtures run normally.

For example, if you have a test that looks like:

TEST(CalculatorTest, TestAdd) {
  test_add(1, 2);
}

This allows you to create a FUZZ_TEST fixture that looks like:

FUZZ_TEST(CalculatorTest, FuzzTestAdd) {
    INIT_FUZZ_TEST;
    int x = provider.ConsumeIntegral<int>();
    int y = provider.ConsumeIntegral<int>();
    test_add(x, y);
}

Run this on Mayhem with:

bazel build --action_env=MAYHEM_URL=$MAYHEM_URL --action_env=XDG_CONFIG_HOME="$HOME/.config" //mayhem:run_fuzzing_utils_test_calculator_package