Smart Test Selection Explained

Why do we choose some Tests and not others?

How does Appsurify's Model determine Test Selection

Simple answer, based on recent developer changes and the trained model!

When Appsurify connects the Repository and begins to receive Incoming Commit data - Appsurify initially analyzes the Repository for basic structuring.

Once the Automated Tests are connected, Appsurify leverages Patent-Pending proprietary AI-Risk Based Testing Technology to train each company's unique AI-Model based off of incoming commits and the corresponding test results. Over a period of 2-3 weeks, Appsurify trains a robust AI Model that links Code to Tests - because it will know which Commits impacted which Tests to form a union between the two.

Once the AI Model is Trained, when a new Commit comes in, Appsurify will know which Tests are associated with that region of the Codebase - and subsequently; select and execute the relevant tests by order of Priority automatically in the CI/CD based on the Parameters set by the team

The AI Model is recalibrated on a rolling basis to ensure it's always up to date. For more information on on our Test Selection please see our Test Selection Whitepaper on the Appsurify website.

Will Appsurify catch every Failed Test?

Appsurify is designed to catch as many defects as possible by running as few tests as possible.

Appsurify is designed to be efficient in finding bugs in its Smart Subset while leaving room to find other potential defects that cause other test failures. 1 Bug often causes more than 1 Test Failure, often multiple. To be efficient and fast, Appsurify is designed to catch as many bugs in as few tests as possible.

Example:

• 1000 Tests.

• 50 Bugs.

• 200 Tests Fail because of the 50 bugs.

• Appsurify: 10% Smart Subsets

If Appsurify is enabled and running the Top "10%" of Tests, it will be forced to Select and Execute what the Model deems to be the most important 100 Tests most likely to Fail.

Therefore, Appsurify will Prioritize and Run the Tests most likely to Catch the underlying Defects. In this case, Appsurify will run Smart Subset of 100 Tests. In that example subset, it could be the case that 70 of them are Failed Tests and 30 of them are Passing Tests.

The IMPORTANT thing to note is that the 70 Failed tests run cover the majority of those 50 underlying bugs and Raise them as New or Reopened Defects. As is often the case with a single defect, it will likely cause multiple Failed Tests. Therefore, Appsurify needs to execute only one or a portion of those Failed Tests in order to raise the Defect while saving room in the Smart Subset to catch other defects. To further this case, if the 50 Bugs caused 50 Tests to Fail (1-to-1), if Appsurify's Smart Subset of 100 Tests caught 50 Failed Tests and 50 Passed Tests and still caught and Raised all 50 Bugs - that would be very successful. Appsurify is designed to find the underlying defect by running as few tests as possible. If the user decided to change the Smart Subset to "5%" in this case, Appsurify would still have room in the Smart Subset to find all 50 Bugs by only running 50 Tests.