Bazel Test Outputs

Introduction

Sometimes, when a unit test fails, developers would like to inspect intermediate test outputs to understand the failure. In Bazel, however, because of sandboxing, it’s often unclear where those intermediate test outputs shall be saved so that developers can access after Bazel test execution. When Bazel test is executed remotely, it’s even more unclear where those intermediate test outputs shall be saved and how developers can access them.

In this blog post, I would like to discuss how to dump the intermediate test outputs from Bazel test.

TEST_TMPDIR VS TEST_UNDECLARED_OUTPUTS_DIR

In Bazel test, there are two private directories that are writable during the test execution: TEST_TMPDIR and TEST_UNDECLARED_OUTPUTS_DIR. From Bazel Test Encyclopedia, we have the following definitions for these two directories:

• TEST_TMPDIR: absolute path to a private writable directory
• TEST_UNDECLARED_OUTPUTS_DIR: absolute path to a private writable directory (used to write undeclared test outputs). Any files written to the TEST_UNDECLARED_OUTPUTS_DIR directory will be zipped up and added to an outputs.zip file under bazel-testlogs.

Because both directories are writable during the test execution, the questions become:

• What are the differences between TEST_TMPDIR and TEST_UNDECLARED_OUTPUTS_DIR?
• When should we use TEST_TMPDIR and when should we use TEST_UNDECLARED_OUTPUTS_DIR?

None of these seem to be documented in the Bazel documentation.

TEST_UNDECLARED_OUTPUTS_DIR is a directory that is not declared in the Bazel build file. Every file or directory that Bazel will read and write have to be declared in the Bazel build file. This means Bazel will not mostly not touch TEST_UNDECLARED_OUTPUTS_DIR except that Bazel will zip up the TEST_UNDECLARED_OUTPUTS_DIR directory and add it to an outputs.zip file under bazel-testlogs. Thus, TEST_UNDECLARED_OUTPUTS_DIR is intended to be used for saving intermediate test outputs.

TEST_TMPDIR, on the other hand, is declared, probably implicitly, in the Bazel build file. This means Bazel will can touch TEST_TMPDIR even if the test code does not have any logic that reads or writes to TEST_TMPDIR. For example, the intended behavior of Bazel is to clean up TEST_TMPDIR before test execution. Additionally, if Bazel test is executed in sandbox, because the sandbox will be deleted after Bazel test is completed, the TEST_TMPDIR will be deleted as well.

Downloading Intermediate Test Outputs

After the Bazel test intermediate test outputs are saved in TEST_UNDECLARED_OUTPUTS_DIR, they will be zipped up and added to an outputs.zip file under bazel-testlogs.

When Bazel test is executed remotely, the user can download the outputs.zip file together with the bazel-testlogs directory using Bazel startup options, such as --remote_download_outputs=all.

Conclusions

Use TEST_UNDECLARED_OUTPUTS_DIR to save intermediate test outputs that developers would like to inspect after Bazel test execution. Use TEST_TMPDIR to save temporary files that are not intended to be inspected after Bazel test execution.

Use Bazel startup options, such as --remote_download_outputs=all, to download the outputs.zip file together with the bazel-testlogs directory when Bazel test is executed remotely.

References

• Bazel Test Encyclopedia
• Bazel Sandboxing
• Bazel Extending Rules: Declaring Outputs
• Bazel Discussion: Extracting Golden Files From Test Results
• Keep TEST_TMPDIR - GitHub Bazel