Scanning and snapshots

Scanning is the process of analysing a repository to detect various issues in it. A scan can be performed on a tag, branch or commit. When a branch is selected, latest available code on the branch will be scanned. A scan results in a snapshot. A snapshot is the representation of the state of a repository in a specific point in time.

Number of scans you can perform is determined by your BrowserStack Code Quality license.

---

Step to scan a repository:

1. A scan can be initiated from the repository context menu or from the Scan button on the top bar within a repository.
2. In the Scan pop-up, select a snapshot label for future identification.
3. Select a tag, branch or commit to scan.
4. Check Enable fast scan if you don’t need a complete scan. Read more about fast scan here.
5. Click the Scan button to start scanning. To know about monitoring the progress of the scan, read this article.

---

Steps to monitor the progress of a scan:

1. Navigate to Scan Queue page from left hand side navigation bar.
2. You can see a list of all ongoing scans here. Basic information and scan progress can be viewed here. An ongoing scan can be aborted using Abort button.
3. To get a detailed log of the scan, click any ongoing scan from the queue.
4. Scan details view will allow for detailed monitoring of scan progress. Log of completed or failed scan can be accessed using scan history feature. Scan can be aborted by clicking Abort Scan button from this page.

Steps to abort or stop a scan:

1. Navigate inside a repository being scanned.
2. Click the the Abort Scan button on top bar to abort the scan.
3. Alternatively an ongoing scan be aborted from the scan queue page or scan details page.

Scan history allows you to investigate scans executed in the past. For each scans that is successful, failed or aborted, a detailed log is available. You can view scan history of a repository through the “Scan History” option in the repository context menu.

You can find recently concluded scans including failed scans here.

Steps to view recent scans :

1. Navigate to Scan Queue page from left hand side navigation bar.
2. You can see a list of all recently concluded scans here. To scan repository again click Scan button.
3. To get a detailed log of the scan, click any scan from the Scan History list.
4. Detail log can be displayed on Scan Details view, click Scan button to scan repository again.

The scan configuration helps you to fine-tune the scan by excluding parts of your code from the scan, uploading additional included directories or adding parser options.

Scan Configuration pop-up can be opened from the repository context menu.

The scan configuration pop-up is displayed. Fill the information in different fields.

The scan configuration fields are described below:

Excludes

You can exclude specific source files by providing JavaScript formatted regular expressions.

Examples:

Default Exclusions

Language	Regular expression	Description
Java	(?i)(test|generated|mock|thirdparty|third-party|3rd-party|3rdparty|external)	Any file path containing 'test', 'generated', 'mock', 'thirdparty', 'third-party', '3rd-party', '3rdparty' and 'external' would be excluded.
C/C++	thirdparty;third-party;3rd-party;external;generated;mock;test;build	Any file path containing 'thirdparty', 'third-party', '3rd-party', 'external', 'generated', 'mock', 'test' and 'build' would be excluded.
C#	(?i)(.g.cs|example|mock|assemblyinfo.cs|.AssemblyAttributes.cs|.AnyCPU.Debug|.AnyCPU.Release)	Any file path containing '.g.cs', 'example', 'mock', 'assemblyinfo.cs', '.AssemblyAttributes.cs', '.AnyCPU.Debug' and '.AnyCPU.Release' would be excluded.
Objective-C	thirdparty;third-party;3rd-party;external;generated;mock;test;build	Any file path containing 'thirdparty', 'third-party', '3rd-party', 'external', 'generated', 'mock', 'test' and 'build' would be excluded.
Python	_init_.py;test	Any file path containing '_init_.py' and 'test' would be excluded.
JavaScript	node_modules;.min.js;dist;external;assets;gulp;grunt;libs;-bundle.js;.bundle.js;swagger-ui;.config.js;-config.js;UNKNOWN_FILE;.git;.ebextensions;test;.lib;.library;.zip	Any file path containing 'node_modules', '.min.js', 'dist', 'external', 'assets', 'gulp', 'grunt', 'libs', '-bundle.js', '.bundle.js', 'swagger-ui', '.config.js', '-config.js', 'UNKNOWN_FILE', '.git', '.ebextensions', 'test', '.lib', '.library' and '.zip' would be excluded.
TypeScript	node_modules;.min.js;dist;external;assets;gulp;grunt;libs;-bundle.js;.bundle.js;swagger-ui;.config.js;-config.js;UNKNOWN_FILE;.git;.ebextensions;test;.lib;.library;.zip;.d.ts	Any file path containing 'node_modules', '.min.js', 'dist', 'external', 'assets', 'gulp', 'grunt', 'libs', '-bundle.js', '.bundle.js', 'swagger-ui', '.config.js', '-config.js', 'UNKNOWN_FILE', ' .git', '.ebextensions', 'test', '.lib', '.library', '.zip' and '.d.ts' would be excluded.
PHP	vendor;wp-content/plugins;protected/extensions;test	Any file path containing 'vendor', 'wp-content', 'plugins', 'protected', 'extensions' and 'test' would be excluded.
Go	test	Any file path containing 'test' would be excluded.
Kotlin	test	Any file path containing 'test' would be excluded.
Solidity	test	Any file path containing 'test' would be excluded.
SQL	test	Any file path containing 'test' would be excluded.

Parser Options

Currently, parser options are only available for C and C++ repositories.

Parsing invalid code:

By default, invalid code parsing is enabled but it can be disabled using the following option:
–parse-invalid-code=OFF

Note: Support for invalid code parsing is limited and may result in parser failure. In case this happens, please disable invalid code parsing.

Define Macros:

Macros can be defined with GCC like option format.

Example:To define macro MY_MACRO -> –clang=”-DMY_MACRO=”

Any option to compiler can be given with –clang=“”. This needs to be added in the parser options field in the scan configuration.

Recursive include header search:

By default, the parser searches for any header file recursively in all the sub-directories of the source folder. This can lead to incorrect results if there are multiple header files with the same name but in different folders.

To disable searching in all sub-directories use the following option:
–include-all=OFF

Uploading Includes

This option is only relevant for scanning C or C++ repositories. If an included file is not found in the source directory, the C/C++ parser will throw a warning message which contains the location of the missing include.

The accuracy of the parser can be improved by uploading the directories which contain the missing files with the correct path. Includes can be uploaded in .zip format.

Note: Source code from additional include directories will not be considered for analysis. It is only used to resolve header dependencies (type resolution).

---

This section covers how BrowserStack Code Quality supports custom extensions for C and C++ languages.

UI Scan

On UI, custom extensions can be specified with Additional Options field of scan configuration page as shown below:

The flag for supplying C extension is “--xc“ and for CPP is “--xcpp“.
User can provide multiple C/CPP extensions with the help of pipe (|) separator as shown above.

And multiple options can be provided using semicolon (;) separator.

Remote Scan

Custom extensions can be specified in remote scan mode also by updating additionalOptions field of gammascan.json file as shown below:

In gammascan.json, multiple options are specified using comma (,) separator.

With these additional options, all the files in repository ending with .pc or .pcc will be considered as C and C++ files respectively, and parsed accordingly.

This section helps to understand how you can overwrite your scan configuration using a checked in file.

Default Configuration Settings

You can download default configuration settings from:

• Go to Projects
• Click on project where the repository belongs
• Click on the 3 dots at the top of the repository
• Click on Download repository configuration

Update configuration settings using embold.json

To update the configuration settings,

• You can use the downloaded repository configuration file (as shown in the previous section) as a template.
• Make your desired changes and save the file as embold.json.
• Commit this embold.json file to the base/root directory before scanning.

Update scan configurations from UI

• Go to “Scan configurations” from the repository settings and select “Overwrite default scan configuration” checkbox. By default, this checkbox is unticked.
• But, if the user modifies any parameter in UI such as excludes or Additional Options or anything from code checkers, tick the checkbox “Overwrite default scan configuration“.
• The original parameters will be overridden.

This section mentions how the data is stored in JSON (JavaScript Object Notation) and it can be customized by adding our own JSON objects.

Code Quality JSON structure

This section helps you to understand different field types, its description, its importance, and default values that areused in the JSON file.

Field Type	Usage	Type	Default
gammaAccess	This is the name of code quality project.
url	Specifies URL of the website	String	Empty
userName	Stores user name for basic authentication to connect to the remote machine	String	Empty
password	Stores password for basic authentication to connect to the remote machine. User can access either using username-password or by adding token for authentication.	String	Empty
token	Stores token for basic authentication to connect to the remote machine. User can access either using username-password or by adding token for authentication.	String	Empty
repositories	Specifies the repository information that will be scanned during your analysis process. For more information, refer this article
dataDir	This is used to store the temporary files.	String	Empty
uid	Stores repository uid which is unique for each repository	String	Saved from UI
projectName	Specifies the repository name		Saved from UI
sources	Path/directories/files that can be scanned during your analysis process. For more information, see as
baseDir	This is a base directory where source code resists. Moreover, this is a kind of main directory where all the operations such as cloning, scans, etc is performed.	String	Empty
exclusions	Specifies list of files/directories/languages that needs to be excluded from your analysis.	Array	Empty
settings	Specifies some additional options that can be used for scan configuration scanning process.
additionalOptions	Specifies the list or lists of additional options that needs to be scanned.	Array	Empty
includePaths	Specifies the file or list of files that needs to be scanned.	Array	Empty
modules	Modules can be used for scanning process. For more information, refer this article .
codeissues	This is a type of the module. This section is used to enabling/disabling code checker configuration. For more information, refer this article.
unittest	Specifies the input data value. For e.g. the input can be in CSV/XML or any other format.	List
coverage	Specifies the coverage information across different modules. For more information, refer this link.	List
relevance	Specifies the name of code issue.	List
name	Specifies the name that can have multiple values such as codeissues, unittest, coverage or relevance.	List	Saved from UI
enabled	Specifies whether the code checker is enabled or not.	Boolean	Saved from UI
config	Specifies the configuration for that particular code checker
rules	Specifies the rules for code issue	Array
options	These include additional options for code checkers like PMD.
name	Specify the rule name for different modules.	String	Saved from UI
type	Specifies the type of module.	String	Saved from UI
value	Specifies the value such as integer or string for that particular module.	String	Saved from UI
required	Specifies whether this module should be required while scanning and its value can be true or false.	Boolean	Saved from UI

Sample JSON file

BrowserStack Code Quality JSON helps you to construct a data hierarchy such as:

A remote scan is an alternate way to scan with no UI intervention. This can be done using a continuous integration toolchain, or manually via the command line. The results are then published on the UI automatically.

For languages such as C or C++, a strict mode remote scan can help to increase the accuracy of the scan.

How to download embold-scanner from Embold?

Download embold-scanner from your Embold Account’s section > Releases tab > CLI. There will be one file with names similar to the following: ‘embold-scanner-archive.tar.gz’.

Remote scan using embold-scanner

1. Download the Embold CLI tool from control panel. Extract that Embold CLI tool and make sure it has all the executable permissions.
2. Login to Embold server. Now create a Project in Ebold. Read more about creating a new project here.
3. Generate the Embold Access Token (EAT) for remote scan. Read more here for generating EAT.
4. Create and link a remote repository with the language you want to scan. Download the repository configuration (repository-configuration.json) for this new remote repository added.

Commands

1. embold-scanner – This command is applicable for Linux machines.
2. embold-scanner.bat – This command is applicable for Windows machines.

Sub-commands

1. analyse
2. local-analyse
3. gated-commit
   

Usage: embold-scanner analyse [-am <arg>] [-b <arg>] [-c <arg>] [-d <arg>] [-h] [-l <arg>] [-r <arg>] [-s <arg>] [-sh <arg>] [-si <arg>] [-sp <arg>] [-ssu] -t <arg> -u <arg> [-v]

Example

1. Linux:
   embold-scanner analyse -c <./repository configuration.json> -u <Embold URL> -t <Embold TOKEN> [-b <BASE_DIR>] [-d <DATA_DIR>] [-s <snapshot name>] [-r <REPO UID>] [-sh <CORONA PACKAGE PATH>] [-l <LOG_FOLDER>]
2. Windows :
   embold-scanner.bat analyse -c <./repository configuration.json> -u <Embold URL> -t <Embold TOKEN> [-b <BASE_DIR>] [-d <DATA_DIR>] [-s <snapshot name>] [-r <REPO UID>] [-sh <CORONA PACKAGE PATH>] [-l <LOG_FOLDER>]

Options

-t,–token <arg>	Embold Token
-u,–url <arg>	Embold URL
-am,–analysis-mode <arg>	Analysis Mode
-b,–repository-base-dir <arg>	Scan will happen for this directory
-c,–scan-config-file <arg>	Scan settings YAML/JSON file path
-d,–data-dir <arg>	Data directory for temporary use
-h,–help	Help
-l,–scanner-logs <arg>	Embold scanner logs directory path
-r,–repository-uid <arg>	The Data will be published in this Repository in Embold UI
-s,–snapshot-label <arg>	This label identifies the snapshot which will be published on Embold UI after successful scan. The value can also be an environment variable
-sh,–scanner-home <arg>	Embold scanner home path
-si,–session-id <arg>	Session ID
-sp,–scan-profile <arg>	Scan profile xml file path
-ssu,–skip-source-upload	Skip Source Upload
-v,–verbose	Enable verbose mode

After a successful remote scan, the below results will be displayed to the user.

Prerequisites

• File path use for -sh(scanner home), -l (scanner-logs) and -d(data directory) should have read write permission.
• The following linters/tools must be installed on the remote machine.

Language	Linter	version
CPP	cppcheck	2.8
C_SHARP	microsoft_security_codescan	5.6.2
TYPESCRIPT	eslint	v7.32.0
	tslint	5.9.1
JAVASCRIPT	jshint	2.9.5
	eslint	v7.32.0
GO	staticcheck	v0.2.1
	gosec	2.8.1
	gometalinter	v2.0.0
PYTHON	bandit	1.7.0
	pylint	2.11.1
	dlint	0.11.0
PHP	phpcs	3.2.3
	phpmd	2.6.1
RUBY	brakeman	5.1.1
KOTLIN	detekt	1.18.1
	mobsfscan	0.1.0
SOLIDITY	solhint	3.3.6
SWIFT	swiftlint	0.32.0
APEX	pmd	6.39.0
HTML	htmlhint	0.15.1
SQL	sqlcheck	1
CSS	stylelint	14.15.1
INFRASTRUCTURE	checkov	2.0.654
YAML	kubesec	2.11.5
LUA	luacheck	0.23.0

Prerequisites to run remotescan for Swift

• Install xcode on your macOS
• Run below command and the output should be similar to the below screenshot
  

xcode-select -p

• If your output is not as above screenshot then run the below command

sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

• Reverify the output of following command

xcode-select -p

• your final output should be as below screenshot

• Now run the remotescan

Configuration for https enabled Embold

• Import the same set of certificates used for Embold in default jre keystore on standalone Corona machine.
• Default jre path: Java/jre1.8.0_171/lib/security.
• Below is the command:
  “keytool -import -trustcacerts -alias gamma -file “ ”-keystore cacerts”

Additional configuration for Strict mode Embold Remote scan for C/CPP code

1. For understanding EMbold strict mode analysis, refer link https://docs.embold.io/installation-and-backup-guide/#c-strict-mode
2. In the above repository-configuration.json file, just add path of the compilation command json file generated by your build system under the section additionalOptions as follows
   “additionalOptions” : [“–cdb=<Folder where the compilaton command json file is generated>”]
   
   OR
   
   Add the generated compilation_commands.json file under the baseDirectory path
   
   embold-scanner will pick up the compilation command json file from either the path given with –cdb option or from the base directory.

Running analysis in strict mode

Set up your remote Embold instance. This is where your analysis results will be published. Follow the steps to set up remote analysis here.

1. On your build machine, navigate to the base directory of your source folder. Generate the compilation database as explained here. You should be able to see a compile_commands.json file in your base directory.
2. Install corona on your build machine as explained here for Ubuntu, Windows and RedHat Enterprise Linux/CentOS. For installer versions, refer this section.
3. Login to Embold. Create a project. Link the repository you want to scan. Select a repository type as “remote“. Read more info creating a project and linking a repository.
4. Select a repository and “Download repository configuration” from the drop down menu of a repository.
5. Open and edit the repository-configuration.json file. Provide the URL of your remote machine where the Embold is running and also, set your token based authentication.
   Note: Embold has deprecated the usage of username and password. Read more info. here.
6. In the sources section, set the value of baseDir as the base directory path of your source folder on your build machine. You can also give exclusions.
   e.g. If you want to exclude folders named “build” and “test”, simply give “.*build.*” , “.*test.*” in the exclusions section.
   For edting ANY settings involving paths, even if you are running Corona on Windows, you must use Unix style path seperator. E.g. forward slash and not backward slash. If you use normal Windows format, it will not work and result into an error message.
7. Set the dataDir to any desired location on your build machine.
8. If you wish to create a separate build folder for running trace-utility (intercept-build), you should use the –cdb additional option while running the scan.
   The compile_commands.json will be created inside your build directory and not in the base directory of your source folder. Since, the scanner looks for the compile_commands.json in the base directory by default, use the  –cdb option to specify the directory where your compile_commands.json is located.
9. In “settings” section, under “additionalOptions” set the directory where compilation database resides.
   Example: --cdb
10. Go to the bin directory inside the scanboxwrapper.
    e.g. $corona_home/scanboxwrapper/bin.
    Run below command:
    /gammascanner -c
/home/user/gamma/corona/scanboxwrapper/examples/gammascan_typical.json
11. If the scan is successful you should see “ANALYSIS SUCCESS”. On your build machine, the last two log messages indicate that the remote analysis was successful.
12. On your remote Embold instance, you will be able to see published results.

---

A snapshot represents the state of your source code at any given point in time. After every successful scan, a new snapshot is generated. While exploring various data visualisations, you can choose any snapshot to see results for that point in time.

To manage snapshots, select the Snapshots option from the repository context menu of any repository.

The number of snapshots you can store on each repository is dependent on your BrowserStack Code Quality license.

Example:
If your snapshot quota is 30 and your limit is exhausted, when you initiate the 31st scan, BrowserStack Code Quality will delete the earliest snapshot to make room for the new snapshot.

To avoid accidental deletion of snapshots by BrowserStack Code Quality, select the Keep snapshot option from Snapshots pop-up.

As per your preference, you can delete a snapshot manually by clicking the x button.