Testing

Unreal Engine provides tools and features to help test the game.


Run Tests

The session frontend is one way to run the tests. In the editor it is found in the menu Window/Developer Tools/Session Frontend. It can also be run standalone in Engine/Binaries/Win64/ UnrealFrontend.exe. The Automation tab list the tests that are available in the session you are connected to. Select the tests using the checkboxes and the press the start tests button to run them.


Command line tests

Run UE4Editor-Cmd.exe and use the command-line argument -ExecCmds to run the automation console command. Multiple commands can be separated with semicolon.


  • List - Show the available tests. It's the same ones shown in the session frontend.

  • RunFilter - Run tests that belong to one of Engine, Smoke, Stress, Perf or Product.

  • RunTests - Select test on Group:, Filter:XXX (only test that start with XXX) or substring match.

  • RunAll - Run all the tests. Takes time so better to select the specific ones to run.


Besides -ExecCmds there are a few UE4 commands line arguments that are useful for testing.


  • -NullRHI - Don't instantiate the whole editor.

  • -testexit="Automation Test Queue Empty" - When the tests are completed the editor will log 'Automation Test Queue Empty'. This option will shut down the editor when it is printed.

  • -unattended - Disable anything requiring feedback from user.

  • -nopause - Close the log window automatically on exit.

  • -ReportOutputPath - Generates a report of all the tests in JSON format in the destination folder. It can be converted to a jUnit XML without to much work.

  • -log="loadmaps.log" - Set the filename of the log file to use. If running multiple test runs in a row it can be good to give them separate names.


Existing Tests

There are a few tests in the engine that are easy and useful to run on the project.


  • Project.Blueprints.Compile Anims - Compile animation blueprints.

  • Project.Blueprints.Compile Blueprints - Compile blueprints.

  • Project.Maps.Load All In Game - Load all levels.

Create Tests

Automation Testing are declared by macros, overriding virtual functions from the FAutomationTestBase class. Both macros ( IMPLEMENT_SIMPLE_AUTOMATION_TEST and IMPLEMENT_COMPLEX_AUTOMATION_TEST ) takes three parameters, TClass, PrettyName and TFlags. TClass is the C++ class to create, PrettyName is the hierarchical test name that will appear in the UI and TFlags tells unreal how the test will be used. Two functions can be implemented for the class. RunTest() performs the actual test, return true if the test passed or false if it failed. GetTests() is used in Complex Tests only, it creates a array of parameters that will be used to call RunTest() once for each parameter. It can be used to get a list of maps for example and then run the test on each map. Each map will be shown as a seperate test in the UI.


To run code over multiple frames Latent Commands (IAutomationLatentCommand) are used. Use it with the macro ADD_LATENT_AUTOMATION_COMMAND with the constructor of the Latent Command you wish to add. Examples of a command is FWaitLatentCommand to wait a number of seconds. Define your own with the DEFINE_LATENT_AUTOMATION_COMMAND macro. I takes a single parameter, the name of the C++ class to create. Then override it's update(), it will be called every frame until it return true. To define commands with a parameter use the DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER macro.


  • Put tests in the Private\Tests directory. Name the file [ClassFilename]Test.cpp.

  • include Misc/AutomationTest.h

  • Wrap the tests in #if WITH_DEV_AUTOMATION_TESTS



The Functional Test (AFunctionalTest) is used to setup test in levels. The test itself can be built into the functional test (with code or blueprint) or created directly in the level blueprint. Override the PrepareTest, IsReady, OnTestStart and OnTestFinished as needed. When the test is complete call FinishTest with the test result. The name of the actor will be used as the test name in the Sessions Frontend.



References

2020 - Harnessing the Unreal Engine Automation Framework for Performance Measurement

2019 - Automated Testing of Gameplay Features in 'Sea of Thieves'

2019 - Automated Testing at Scale in Sea of Thieves

2018 - UE4 Unit Tests in Jenkins

JUnit XML reporting file format for Jenkins