Scanning and snapshots

Scanning is the process of analysing a repository to detect various issues in it. A scan can be performed on a tag, branch or commit. When a branch is selected, latest available code on the branch will be scanned. A scan results in a snapshot. A snapshot is the representation of the state of a repository in a specific point in time.

Number of scans you can perform is determined by your Embold license.

---

Step to scan a repository:

1. A scan can be initiated from the repository context menu or from the Scan button on the top bar within a repository.
2. In the Scan pop-up, select a snapshot label for future identification.
3. Select a tag, branch or commit to scan.
4. Check Enable fast scan if you don’t need a complete scan. Read more about fast scan here.
5. Click the Scan button to start scanning. To know about monitoring the progress of the scan, read this article.

---

Steps to monitor the progress of a scan:

1. Navigate to Scan Queue page from left hand side navigation bar.
2. You can see a list of all ongoing scans here. Basic information and scan progress can be viewed here. An ongoing scan can be aborted using Abort button.
3. To get a detailed log of the scan, click any ongoing scan from the queue.
4. Scan details view will allow for detailed monitoring of scan progress. Log of completed or failed scan can be accessed using scan history feature. Scan can be aborted by clicking Abort Scan button from this page.

Steps to abort or stop a scan:

1. Navigate inside a repository being scanned.
2. Click the the Abort Scan button on top bar to abort the scan.
3. Alternatively an ongoing scan be aborted from the scan queue page or scan details page.

Scan history allows you to investigate scans executed in the past. For each scans that is successful, failed or aborted, a detailed log is available. You can view scan history of a repository through the “Scan History” option in the repository context menu.

You can find recently concluded scans including failed scans here.

Steps to view recent scans :

1. Navigate to Scan Queue page from left hand side navigation bar.
2. You can see a list of all recently concluded scans here. To scan repository again click Scan button.
3. To get a detailed log of the scan, click any scan from the Scan History list.
4. Detail log can be displayed on Scan Details view, click Scan button to scan repository again.

The scan configuration helps you to fine-tune the scan by excluding parts of your code from the scan, uploading additional included directories or adding parser options.

Scan Configuration pop-up can be opened from the repository context menu.

The scan configuration pop-up is displayed. Fill the information in different fields.

The scan configuration fields are described below:

Excludes

You can exclude specific source files by providing JavaScript formatted regular expressions.

Examples:

Default Exclusions

Language	Regular expression	Description
Java	(?i)(test|generated|mock|thirdparty|third-party|3rd-party|3rdparty|external)	Any file path containing 'test', 'generated', 'mock', 'thirdparty', 'third-party', '3rd-party', '3rdparty' and 'external' would be excluded.
C/C++	thirdparty;third-party;3rd-party;external;generated;mock;test;build	Any file path containing 'thirdparty', 'third-party', '3rd-party', 'external', 'generated', 'mock', 'test' and 'build' would be excluded.
C#	(?i)(.g.cs|example|mock|assemblyinfo.cs|.AssemblyAttributes.cs|.AnyCPU.Debug|.AnyCPU.Release)	Any file path containing '.g.cs', 'example', 'mock', 'assemblyinfo.cs', '.AssemblyAttributes.cs', '.AnyCPU.Debug' and '.AnyCPU.Release' would be excluded.
Objective-C	thirdparty;third-party;3rd-party;external;generated;mock;test;build	Any file path containing 'thirdparty', 'third-party', '3rd-party', 'external', 'generated', 'mock', 'test' and 'build' would be excluded.
Python	_init_.py;test	Any file path containing '_init_.py' and 'test' would be excluded.
JavaScript	node_modules;.min.js;dist;external;assets;gulp;grunt;libs;-bundle.js;.bundle.js;swagger-ui;.config.js;-config.js;UNKNOWN_FILE;.git;.ebextensions;test;.lib;.library;.zip	Any file path containing 'node_modules', '.min.js', 'dist', 'external', 'assets', 'gulp', 'grunt', 'libs', '-bundle.js', '.bundle.js', 'swagger-ui', '.config.js', '-config.js', 'UNKNOWN_FILE', '.git', '.ebextensions', 'test', '.lib', '.library' and '.zip' would be excluded.
TypeScript	node_modules;.min.js;dist;external;assets;gulp;grunt;libs;-bundle.js;.bundle.js;swagger-ui;.config.js;-config.js;UNKNOWN_FILE;.git;.ebextensions;test;.lib;.library;.zip;.d.ts	Any file path containing 'node_modules', '.min.js', 'dist', 'external', 'assets', 'gulp', 'grunt', 'libs', '-bundle.js', '.bundle.js', 'swagger-ui', '.config.js', '-config.js', 'UNKNOWN_FILE', ' .git', '.ebextensions', 'test', '.lib', '.library', '.zip' and '.d.ts' would be excluded.
PHP	vendor;wp-content/plugins;protected/extensions;test	Any file path containing 'vendor', 'wp-content', 'plugins', 'protected', 'extensions' and 'test' would be excluded.
Go	test	Any file path containing 'test' would be excluded.
Kotlin	test	Any file path containing 'test' would be excluded.
Solidity	test	Any file path containing 'test' would be excluded.
SQL	test	Any file path containing 'test' would be excluded.

Parser Options

Currently, parser options are only available for C and C++ repositories.

Parsing invalid code:

By default, invalid code parsing is enabled but it can be disabled using the following option:
–parse-invalid-code=OFF

Note: Support for invalid code parsing is limited and may result in parser failure. In case this happens, please disable invalid code parsing.

Define Macros:

Macros can be defined with GCC like option format.

Example:To define macro MY_MACRO -> –clang=”-DMY_MACRO=”

Any option to compiler can be given with –clang=“”. This needs to be added in the parser options field in the scan configuration.

Recursive include header search:

By default, the parser searches for any header file recursively in all the sub-directories of the source folder. This can lead to incorrect results if there are multiple header files with the same name but in different folders.

To disable searching in all sub-directories use the following option:
–include-all=OFF

Uploading Includes

This option is only relevant for scanning C or C++ repositories. If an included file is not found in the source directory, the C/C++ parser will throw a warning message which contains the location of the missing include.

The accuracy of the parser can be improved by uploading the directories which contain the missing files with the correct path. Includes can be uploaded in .zip format.

Note: Source code from additional include directories will not be considered for analysis. It is only used to resolve header dependencies (type resolution).

---

This section covers how Embold supports custom extensions for C and C++ languages.

UI Scan

On UI, custom extensions can be specified with Additional Options field of scan configuration page as shown below:

The flag for supplying C extension is “--xc“ and for CPP is “--xcpp“.
User can provide multiple C/CPP extensions with the help of pipe (|) separator as shown above.

And multiple options can be provided using semicolon (;) separator.

Remote Scan

Custom extensions can be specified in remote scan mode also by updating additionalOptions field of gammascan.json file as shown below:

In gammascan.json, multiple options are specified using comma (,) separator.

With these additional options, all the files in repository ending with .pc or .pcc will be considered as C and C++ files respectively, and parsed accordingly.

This section helps to understand how you can overwrite your scan configuration using a checked in file.

Default Configuration Settings

The default configuration is basically a committed JSON file. This JSON file should be committed to the base/root directory before scanning.

On Scan Configuration pop-up, the user can view the “Overwrite default scan configuration” checkbox.

• By default, this checkbox is unticked.
• But, if the user modifies any parameter in UI such as excludes or Additional Options or anything from code checkers, tick the checkbox “Overwrite default scan configuration“.
• The original parameters will be overridden.

Repository Configuration

In JSON file, by default, exclusions are empty.

For example: In the below image, we can see that whatever excludes we have added in Scan Configuration pop-up and if a user downloads the repository_configuration file (JSON), similar changes will be seen here.

For more information, refer to the “Embold scan configuration JSON”.

This section mentions how the data is stored in JSON (JavaScript Object Notation) and it can be customized by adding our own JSON objects.

Embold JSON structure

This section helps you to understand different field types, its description, its importance, and default values that areused in the JSON file.

Field Type	Usage	Type	Default
gammaAccess	This is the name of Embold project.
url	Specifies URL of the website	String	Empty
userName	Stores user name for basic authentication to connect to the remote machine	String	Empty
password	Stores password for basic authentication to connect to the remote machine. User can access either using username-password or by adding token for authentication.	String	Empty
token	Stores token for basic authentication to connect to the remote machine. User can access either using username-password or by adding token for authentication.	String	Empty
repositories	Specifies the repository information that will be scanned during your analysis process. For more information, refer this article
dataDir	This is used to store the temporary files.	String	Empty
uid	Stores repository uid which is unique for each repository	String	Saved from UI
projectName	Specifies the repository name		Saved from UI
sources	Path/directories/files that can be scanned during your analysis process. For more information, see as
baseDir	This is a base directory where source code resists. Moreover, this is a kind of main directory where all the operations such as cloning, scans, etc is performed.	String	Empty
exclusions	Specifies list of files/directories/languages that needs to be excluded from your analysis.	Array	Empty
settings	Specifies some additional options that can be used for scan configuration scanning process.
additionalOptions	Specifies the list or lists of additional options that needs to be scanned.	Array	Empty
includePaths	Specifies the file or list of files that needs to be scanned.	Array	Empty
modules	Modules can be used for scanning process. For more information, refer this article .
codeissues	This is a type of the module. This section is used to enabling/disabling code checker configuration. For more information, refer this article.
unittest	Specifies the input data value. For e.g. the input can be in CSV/XML or any other format.	List
coverage	Specifies the coverage information across different modules. For more information, refer this link.	List
relevance	Specifies the name of code issue.	List
name	Specifies the name that can have multiple values such as codeissues, unittest, coverage or relevance.	List	Saved from UI
enabled	Specifies whether the code checker is enabled or not.	Boolean	Saved from UI
config	Specifies the configuration for that particular code checker
rules	Specifies the rules for code issue	Array
options	These include additional options for code checkers like PMD.
name	Specify the rule name for different modules.	String	Saved from UI
type	Specifies the type of module.	String	Saved from UI
value	Specifies the value such as integer or string for that particular module.	String	Saved from UI
required	Specifies whether this module should be required while scanning and its value can be true or false.	Boolean	Saved from UI

Sample JSON file

Embold JSON helps you to construct a data hierarchy such as:

Fast scan enables you to perform a scan only on the changed files, thus giving you a significantly fast scanning experience.

Steps to enable fast scan:

1. Check Enable fast scan (beta) option from the scan pop-up.

Limitations:

In some circumstances such as the very first scan of a repository, fast scan option will not be available.

---

A remote scan is an alternate way to scan with no UI intervention. This can be done using a continuous integration toolchain, or manually via the command line. The results are then published on the UI automatically.

For languages such as C or C++, a strict mode remote scan can help to increase the accuracy of the scan.

The remote scan is offered with an on-premise installation of Embold.

How to download embold-scanner from Embold?

Download embold-scanner from your Embold Account’s section > Releases tab > CLI. There will be one file with names similar to the following: ‘embold-scanner-1.0.0-archive.tar.gz’.

Remote scan using embold-scanner

1. Download the Embold CLI tool from control panel. Extract that Embold CLI tool and make sure it has all the executable permissions.
2. Login to Embold server. Now create a Project in Embold. Read more about creating a new project here.
3. Generate the Embold Access Token (EAT) for remote scan. Read more here for generating EAT.
4. Create and link a remote repository with the language you want to scan. Download the repository configuration (repository-configuration.json) for this new remote repository added.

Commands

1. embold-scanner – This command is applicable for Linux machines.
2. embold-scanner.bat – This command is applicable for Windows machines.
   

Usage: embold-scanner [-am <arg>] [-b <arg>] -c <arg> [-d <arg>] [-h] [-i <arg>] [-l <arg>] [-la] [-od <arg>] [-p <arg>] [-r <arg>] [-s <arg>] [-sh <arg>] [-sp <arg>] -t <arg> -u <arg> [-v] -am,--Analysis Mode <arg> Analysis Mode

Example

1. Linux:
   embold-scanner -c <./repository configuration.json> -u <EMBOLD URL> -t <EMBOLD TOKEN> [-s <snapshot name>] [-r <REPO UID>] [-sh <CORONA PACKAGE PATH>] [-l <LOG_FOLDER>]
2. Windows :
   embold-scanner.bat -c <./repository configuration.json> -u <EMBOLD URL> -t <EMBOLD TOKEN> [-s <snapshot name>] [-r <REPO UID>] [-sh <CORONA PACKAGE PATH>] [-l <LOG_FOLDER>]

Options

-b <arg>,–Repository Base Directory, Scan will happen for this directory
-c <arg>,–scan settings, Scan settings JSON file path. <./repository configuration.json> is an argument passed for this parameter.
-d <arg>,–Data Directory, Data directory for temporary use
-h,–help Help
-i <arg>,–DB IP Address, This IP Address would replace Embold DB IP Address
-l <arg>,–scanner_logs Embold scanner logs directory path
-la,–Local Analysis Local Analysis
-od <arg>,–Output Director, Output dir
-p <arg>,–DB Port, This port would replace Embold DB Port
-r <arg>,–Repository UID, The data will get published in this Repository on Embold UI. Read more info. here.
-s <arg>,–Snapshot label, This label identifies the snapshot which will publish on Embold UI after a successful scan. The value can also be an environment variable
-sh <arg>,–scanner_home, Embold scanner home path
-sp <arg>,–Scan Profile, Scan profile XML file path
-t <arg>,–token, Embold Token. <EMBOLD TOKEN> is an argument passed for this parameter. Read more here for generating Embold Access Token.
-u <arg>,–URL, Embold URL. <EMBOLD URL> is an argument passed for this parameter.
-v,–verbose Enable verbose mode Process finished with exit code 0

After a successful remote scan, the below results will be displayed to the user.

Remote Analysis

The remote analysis is an alternate way of analysis with no UI intervention. Here, Embold analysis is done remotely, and results are published on Embold UI.

Prerequisites

• In order to give read, write and execute permission to a user running remote scan:
  • create a group ‘gamma’ with the help of following command: sudo groupadd gamma
  • Change the group of the gamma installation directory to ‘gamma’ sudo chgrp gamma -R /opt/gamma
  • Add permission to gamma installation directory: sudo chmod 774 -R /opt/gamma
  • Add a user, running remote scan, to ‘gamma’ group with following command: sudo usermod -a -G gamma user
  • Restart Embold sudo service gamma restart
• Setup Embold Remote instance if not already available. This is where your analysis results will be published.
• Create a subsystem on Embold UI to be analyzed.

Configuration for https enabled Embold

• Import the same set of certificates used for Embold in default jre keystore on standalone Corona machine.
• Default jre path: Javajre1.8.0_171libsecurity.
• Below is the command:
  “keytool -import -trustcacerts -alias gamma -file “ ”-keystore cacerts”

---

Overview

The Embold CLI (Command Line Interface) analyses modified files or a complete repository.

It also generates a report containing code issues, anti-patterns, and metrics.

Prerequisites

• A valid Embold account is required. Contact the administrator to create an account.
• Git version (recommended version 2.18.0) should be installed.
• The project to be analyzed should be a Git repository.
• NodeJS (version 10.xx) For JavaScript/Typescript repository analysis.
• Install CORONA through the installer. Verify if the CORONA_HOME environment variable is set.

Configuration

1. Edit the file $CORONA_HOME/coronacli/config/cli.properties to add embold url and embold token. Read more about generating Embold Access Token here.
   Example :
   embold.url= https://myembold.io/
   embold.token = <embold_token>
2. “$CORONA_HOME/tools/bin ” should be added in the PATH environment variable.

Embold CLI commands

Syntax: embold-analyse [options]

The Embold CLI provides multiple options like:

• -f: Analyses all the files in the current directory. If the option “-f” is not given, only the changed files will get scanned. For more information, refer to the supported languages link.
• -o: By default, it generates the output on the console. Moreover, the supported formats are CSV and XML.
• -od: Saves the output in the corresponding directory.
• -h: Displays the usage.

Usages of Embold CLI:

Following are different usages of the Embold CLI :

CLI Commands	Usage
embold-analyse	Analyses all the staged, unstaged and the untracked files in the current directory and the output is displayed on the console.
embold-analyse -f	Analyses all the files in the current directory and the output is displayed on the console.
embold-analyse -o	Analyses all the modified files in the current directory. But, if the option "-o" is not given, the output is displayed on the console.
embold-analyse -o CSV -od [output_dir_path]	Analyze all the modified files in the current directory and generates the output in the CSV file. This CSV file is saved in the output directory.
embold-analyse -o XML -f -od [output_dir_path]	Analyses all the modified files in the current directory and generates output in the XML file. This XML file is saved in the output directory.
embold-analyse -o CSV -f -od [output_dir_path]	Analyses all the modified files in the current directory and generates the output in CSV file. This CSV file is saved in the output directory.

A snapshot represents the state of your source code at any given point in time. After every successful scan, a new snapshot is generated. While exploring various data visualisations, you can choose any snapshot to see results for that point in time.

To manage snapshots, select the Snapshots option from the repository context menu of any repository.

The number of snapshots you can store on each repository is dependent on your Embold license.

Example:
If your snapshot quota is 30 and your limit is exhausted, when you initiate the 31st scan, Embold will delete the earliest snapshot to make room for the new snapshot.

To avoid accidental deletion of snapshots by Embold, select the Keep snapshot option from Snapshots pop-up.

As per your preference, you can delete a snapshot manually by clicking the x button.