Testing for Regressions: Recommended Approach

Intel® Inspector User Guide for Linux* OS

Download PDF

ID 767796

Date 5/15/2022

Version

Public

A newer version of this document is available. Customers should click here to go to the newest version.

Visible to Intel only — GUID: GUID-C229781B-5AB1-4904-BA49-25288FF03BDA

View Details

Testing for Regressions: Recommended Approach

The recommended approach for regression testing with the Intel Inspector involves these basic steps:

Create a gold standard:
- Use the GUI (or the inspxe-cl command) to create a result from which you have successfully eliminated all real problems.
- Use the inspxe-cl command to create a suppression file to match any remaining problems you have determined you can ignore.
Change your source code.
Use the inspxe-cl command (or the GUI) to check for new problems:
- Apply the suppression file when you run a new analysis.
- Determine if the changed source code introduced new problems that do not match those in the suppression file.

Create a Result

In the GUI (or the inspxe-cl command tool if you prefer), create a result from which you have successfully eliminated all real problems. Any problem remaining in the result is a positive you have determined you can ignore. For example:

You may have Data race problem caused by more than one thread accessing a variable without synchronization; however, you know there are other timing factors that make it impossible for the Data race to occur.
You may have an issue in a third-party library that is of no interest to you.

Create one result for each data set you plan to test, or one result for your entire application. You can use a result created with the same analysis type you plan to use for the new analysis or with the widest analysis type available.

Create a Suppression File

In the inspxe-cl command tool, use the create-suppression-file action to generate a suppress-all file for all remaining problems.

For example: The following inspxe-cl command creates a mySup suppression file with rules that suppress every error in the r002ti2 result.

$ inspxe-cl -create-suppression-file mySup -result-dir r002ti2

Create a suppress-all file for each data set you plan to test and store them all in one directory. If you prefer to use the GUI to perform the remainder of this recommended approach, use the Suppressions tab in the Project Properties dialog box to identify that directory.

CAUTION:

Applying many suppression files or large suppressions files during analysis may slow down finalization, so make sure you remove obsolete suppression files from the directory and generate new suppression files when previously known issues are eliminated.

Change Your Source Code

Use your development environment to edit your source code.

Suppression files remain effective for changed source code with the following caveats:

Intel Inspector identifies source code lines relative to the start of a function, not as absolute values. So you can insert and delete lines of code anywhere in source files without rendering a suppression file ineffective except inside a function where a problem is marked for suppression.
Adding or deleting lines in a function prior to the location of a suppressed problem causes a problem to no longer match the suppression rule. Therefore the problem appears as a new problem in subsequent analysis runs.

TIP:

Intel Inspector-created suppression files contain (usually multiple) narrow rules that include line numbers (to not accidentally suppress extra problems). Consider handwriting your own suppression files to create fewer rules with wider reach - or at least rules that do not include line numbers.

Run a New Analysis

In the inspxe-cl command tool (or the GUI if you prefer), run a new analysis with the suppress-all file.

TIP:

You can incrementally apply multiple suppress-all files if your gold standard result changes.

For example: The following inspxe-cl command uses the suppression-file action-option to apply all suppression files in the C:\My Inspector Results\suppressions directory when collecting a new result for the myApp application.

$ inspxe-cl -collect ti2 -suppression-file "C:\My Inspector Results\suppressions" -- myApp

Determine if the Changed Source Code Introduced New Problems

When analysis is complete, investigate the result to check for new problems:

In the inspxe-cl command tool: Check the automatically generated inspxe-cl.txt Summary report.
In the GUI: Check the Problems pane.

Any new problems are regressions.

Automate Your Regression Testing Process

Intel Inspector must execute your code path to find errors in it, so you should consider running the Intel Inspector on multiple versions of your code, on different workloads that stress different code paths, as well as on corner cases. However, multiple Intel Inspector analysis runs can take time.

It may be more efficient to automate: Batch process multiple analysis runs with different data sets passed in as arguments - possibly overnight - and have the computer do the work for you.

The simplest automation scenario involves adding the following steps to a script:

Set up the inspxe-cl command environment.
Invoke the inspxe-cl command to run an analysis and apply the suppress-all file(s).
Determine if the changed source code introduced new problems by checking the returned exit code in the automatically generated inspxe-cl.txt Summary report when analysis is complete. If the reported exit code equals 0, there are no new problems detected in unsuppressed code locations. If the reported exit code does not equal 0, consider sending the Summary report to a log file and email the log file to an engineer for further triage.
Repeat steps 2 and 3 as necessary.

Parent topic: Testing for Regressions

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Intel® Inspector User Guide for Linux* OS

Testing for Regressions: Recommended Approach

Create a Result

Create a Suppression File

Change Your Source Code

Run a New Analysis

Determine if the Changed Source Code Introduced New Problems

Automate Your Regression Testing Process

See Also