Step 3: Connect Tests

Start pushing Test Results

Add Script in CI Pipeline to Connect Tests

Need to add an executable script in the CI to call the TestBrain API with the test conditions.


  • Required: Python 3.7+

Installation Command

--pip install appsurifyci --upgrade

Available Arguments



Call to Appsurify TestBrain API


API Key from Appsurify Instance. Found in top right hand corner drop-down menu. Can be stored as a Secret in CI server secret storage capabilities. Reference the Secret via variable Script execution in the CI configuration.


Url of Appsurify Instance i.e.,


Name of the project


Name of the Test Suite


Name of the branch the tests are running against


SHA of the commit (need full SHA)



Options - all tests, no tests, prioritized tests. See below for more details.


Options - mvn, pytest, cypress, mocha, mstest, vstest, opentest...etc. See Integrations for more supported Test Frameworks


Command to execute tests in the target environment, if command is custom to your test suite. Note, --runcommand will override generic --testtemplate at time of final configuration and prioritized test execution.


Location of XML reports created by the test run if this is not the default location for the test type.


Additional logging is printed to the screen from the Test Execution runner to help with Debugging and Logs

Test Execution Parameters

--all tests

Will run all tests. Will fail on any failure. Recommended for nightly or weekly test executions. Recommended for Appsurify to Learn

--no tests

Will run no tests. Will fail on any failure. Recommended when starting with Appsurify to just upload data.

--prioritized tests

Requires the Percentage or Time based Argument. Will fail the build on new / re-opened defects. Predictive Test Selection Technology


--runtestswithappsurify --url "" --apikey "api key" --project "project" --testsuite "testsuite" --runtemplate "all tests" --runcommand "mvn" --commit "commit id" --branch "branch" --report "report" --pipeout "true"

Example in CI Pipeline

--runtestwithappsurify --url "" --apikey "NTk6cD1Ua3doV3uNDMdWzz" --project "TestBrainDemoJunit" --testsuite "Selenium" --runtemplate "all tests" --runcommand "mvn" --commit "commit id" --branch "branch" --report "report" --pipeoutput "true"

Confirmation: Test Bind Turns Green

Upon successful Test Connection, in your Summary Page - your Test Bind Icon turns Green.

Push tests as part of CI process to build Model Maturity

Now that you've successfully Connected your Tests, automate them as part of the CI build process to build the Model Maturity wheel. Depending on how many Test Runs per day, this could take ~2-3 weeks. AKA, the more test data / runs - the stronger the results!

Once Model Maturity wheel has hit 50 Test runs or 100%, your Model will be trained over the following weekend, and will be subsequently ready for Predictive Test Selection mode - and your team can begin to enjoy the time and resource savings!

Additional Arguments - For Customization

ArgumentDefaultOptions / Details


newdefects reopeneddefects

Options: newdefects, reopeneddefects, flakybrokentests, newflaky, reopenedflaky, failedtests, brokentests



whether failed tests should be rerun



the number of times failed tests should be rerun



whether after each set of tests runs to determine if the build has failed (tests will still be rerun if this has been selected)


" "

command to be executed following completion of the script [[teststorun]] will be replaced with the formatted list of tests to run


" "

Percentage based prioritized test selection and execution. RECOMMENDED to start ~30%. Example: --percentage "20" would run top 20% of tests most likely to fail.


" "

Time based prioritized test execution. Example: --time "45s" would run top 45 seconds of tests most likely to fail. Example --time "10m" would run top 10 minutes of tests most likely to fail.

Last updated