Step 3: Connect Tests
Start pushing Test Results
Add Script in CI Pipeline to Connect Tests
Need to add an executable script in the CI to call the TestBrain API with the test conditions.
Requirements
Required: Python 3.7+
Installation Command
Available Arguments
--runtestswithappsurify
Call to Appsurify TestBrain API
--apikey
API Key from Appsurify Instance. Found in top right hand corner drop-down menu. Can be stored as a Secret in CI server secret storage capabilities. Reference the Secret via variable Script execution in the CI configuration.
--url
Url of Appsurify Instance i.e., https://dummy.appsurify.com/
--project
Name of the project
--testsuite
Name of the Test Suite
--branch
Name of the branch the tests are running against
--commit
SHA of the commit (need full SHA)
Recommended Arguments
--runtemplate
Options - all tests, no tests, prioritized tests. See below for more details.
--testtemplate
Options - mvn, pytest, cypress, mocha, mstest, vstest, opentest...etc. See Integrations for more supported Test Frameworks
--runcommand
Command to execute tests in the target environment, if command is custom to your test suite.
Note, --runcommand will override generic --testtemplate at time of final configuration and prioritized test execution.
--report
Location of XML reports created by the test run if this is not the default location for the test type.
--pipeoutput
Additional logging is printed to the screen from the Test Execution runner to help with Debugging and Logs
Test Execution Parameters
--all tests
Will run all tests. Will fail on any failure. Recommended for nightly or weekly test executions. Recommended for Appsurify to Learn
--no tests
Will run no tests. Will fail on any failure. Recommended when starting with Appsurify to just upload data.
--prioritized tests
Requires the Percentage or Time based Argument. Will fail the build on new / re-opened defects. Predictive Test Selection Technology
Example
Example in CI Pipeline
Confirmation: Test Bind Turns Green
Upon successful Test Connection, in your Summary Page - your Test Bind Icon turns Green.
Push tests as part of CI process to build Model Maturity
Now that you've successfully Connected your Tests, automate them as part of the CI build process to build the Model Maturity wheel. Depending on how many Test Runs per day, this could take ~2-3 weeks. AKA, the more test data / runs - the stronger the results!
Once Model Maturity wheel has hit 50 Test runs or 100%, your Model will be trained over the following weekend, and will be subsequently ready for Predictive Test Selection mode - and your team can begin to enjoy the time and resource savings!
Additional Arguments - For Customization
--fail
newdefects
reopeneddefects
Options: newdefects, reopeneddefects, flakybrokentests, newflaky, reopenedflaky, failedtests, brokentests
--rerun
"false"
whether failed tests should be rerun
--maxrerun
3
the number of times failed tests should be rerun
--failfast
"false"
whether after each set of tests runs to determine if the build has failed (tests will still be rerun if this has been selected)
--executioncommand
" "
command to be executed following completion of the script [[teststorun]] will be replaced with the formatted list of tests to run
--percentage
" "
Percentage based prioritized test selection and execution. RECOMMENDED to start ~30%.
Example:
--percentage "20" would run top 20% of tests most likely to fail.
--time
" "
Time based prioritized test execution. Example: --time "45s" would run top 45 seconds of tests most likely to fail. Example --time "10m" would run top 10 minutes of tests most likely to fail.
Next: Start Test Prioritization!
Last updated
