Scans
Scanning is the process of analysing a repository to detect various issues in it. A scan can be performed on a tag, branch or commit. When a branch is selected, latest available code on the branch will be scanned. A scan results in a snapshot. A snapshot is the representation of the state of a repository in a specific point in time.
Number of scans you can perform is determined by your Embold license.
Scan a repository
Step to scan a repository:
- A scan can be initiated from the repository context menu or from the Scan button on the top bar within a repository.
- In the Scan pop-up, select a snapshot label for future identification.
- Select a tag, branch or commit to scan.
- Check Enable fast scan if you don’t need a complete scan. Read more about fast scan here.
- Click the Scan button to start scanning. To know about monitoring the progress of the scan, read this article.
Monitoring scan progress
Steps to monitor the progress of a scan:
- Navigate to Scan Queue page from left hand side navigation bar.
- You can see a list of all ongoing scans here. Basic information and scan progress can be viewed here. An ongoing scan can be aborted using Abort button.
- To get a detailed log of the scan, click any ongoing scan from the queue.
- Scan details view will allow for detailed monitoring of scan progress. Log of completed or failed scan can be accessed using scan history feature. Scan can be aborted by clicking Abort Scan button from this page.
Aborting or stopping an ongoing scan
Steps to abort or stop a scan:
- Navigate inside a repository being scanned.
- Click the the Abort Scan button on top bar to abort the scan.
- Alternatively an ongoing scan be aborted from the scan queue page or scan details page.
Scan History
Scan history allows you to investigate scans executed in the past. For each scans that is successful, failed or aborted, a detailed log is available. You can view scan history of a repository through the “Scan History” option in the repository context menu.
You can find recently concluded scans including failed scans here.
Steps to view recent scans :
- Navigate to Scan Queue page from left hand side navigation bar.
- You can see a list of all recently concluded scans here. To scan repository again click Scan button.
- To get a detailed log of the scan, click any scan from the Scan History list.
- Detail log can be displayed on Scan Details view, click Scan button to scan repository again.
Scan configurations
The scan configuration helps you to fine-tune the scan by excluding parts of your code from the scan, uploading additional included directories or adding parser options.
Scan Configuration pop-up can be opened from the repository context menu.
The scan configuration pop-up is displayed. Fill the information in different fields.
The scan configuration feilds are described below:
Excludes
You can exclude specific source files by providing JavaScript formatted regular expressions.
Examples:
| Example | Code |
|---|---|
| Filer out files containing the keyword “test” | “.*test.*” |
| Filter out everything but one file | “^(?!.*parse-this-file-only.cpp*$).*” |
| Using escape characters to match special characters (+,.) | “.*test.c++*” |
Default Exclusions
| Language | Regular expression | Description |
|---|---|---|
| Java | (?i)(test|generated|mock|thirdparty|third-party|3rd-party|3rdparty|external) | Any file path containing 'test', 'generated', 'mock', 'thirdparty', 'third-party', '3rd-party', '3rdparty' and 'external' would be excluded. |
| C/C++ | thirdparty;third-party;3rd-party;external;generated;mock;test;build | Any file path containing 'thirdparty', 'third-party', '3rd-party', 'external', 'generated', 'mock', 'test' and 'build' would be excluded. |
| C# | (?i)(.g.cs|example|mock|assemblyinfo.cs|.AssemblyAttributes.cs|.AnyCPU.Debug|.AnyCPU.Release) | Any file path containing '.g.cs', 'example', 'mock', 'assemblyinfo.cs', '.AssemblyAttributes.cs', '.AnyCPU.Debug' and '.AnyCPU.Release' would be excluded. |
| Objective-C | thirdparty;third-party;3rd-party;external;generated;mock;test;build | Any file path containing 'thirdparty', 'third-party', '3rd-party', 'external', 'generated', 'mock', 'test' and 'build' would be excluded. |
| Python | _init_.py;test | Any file path containing '_init_.py' and 'test' would be excluded. |
| JavaScript | node_modules;.min.js;dist;external;assets;gulp;grunt;libs;-bundle.js;.bundle.js;swagger-ui;.config.js;-config.js;UNKNOWN_FILE;.git;.ebextensions;test;.lib;.library;.zip | Any file path containing 'node_modules', '.min.js', 'dist', 'external', 'assets', 'gulp', 'grunt', 'libs', '-bundle.js', '.bundle.js', 'swagger-ui', '.config.js', '-config.js', 'UNKNOWN_FILE', '.git', '.ebextensions', 'test', '.lib', '.library' and '.zip' would be excluded. |
| TypeScript | node_modules;.min.js;dist;external;assets;gulp;grunt;libs;-bundle.js;.bundle.js;swagger-ui;.config.js;-config.js;UNKNOWN_FILE;.git;.ebextensions;test;.lib;.library;.zip;.d.ts | Any file path containing 'node_modules', '.min.js', 'dist', 'external', 'assets', 'gulp', 'grunt', 'libs', '-bundle.js', '.bundle.js', 'swagger-ui', '.config.js', '-config.js', 'UNKNOWN_FILE', ' .git', '.ebextensions', 'test', '.lib', '.library', '.zip' and '.d.ts' would be excluded. |
| PHP | vendor;wp-content/plugins;protected/extensions;test | Any file path containing 'vendor', 'wp-content', 'plugins', 'protected', 'extensions' and 'test' would be excluded. |
| Go | test | Any file path containing 'test' would be excluded. |
| Kotlin | test | Any file path containing 'test' would be excluded. |
| Solidity | test | Any file path containing 'test' would be excluded. |
| SQL | test | Any file path containing 'test' would be excluded. |
Parser Options
Currently, parser o
Parsing invalid code:
By default, invalid code parsing is enabled but it can be disabled using the following option:
–parse-invalid-code=OFF
Note: Support for invalid code parsing is limited and may result in parser failure. In case this happens, please disable invalid code parsing.
Define Macros:
Macros can be defined with GCC like option format.
Example:To define macro MY_MACRO -> –clang=”-DMY_MACRO=”
Any option to compiler can be given with –clang=
Recursive include header search:
By default, the parser searches for any header file recursively in all the sub-directories of the source folder. This can lead to incorrect results if there are multiple header files with the same name but in different folders.
To disable searching in all sub-directories use the following option:
–include-all=OFF
Uploading Includes
This option is only relevant for scanning C or C++ repositories. If an included file is not found in the source directory, the C/C++ parser will throw a warning message which contains the location of the missing include.
The accuracy of the parser can be improved by uploading the directories which contain the missing files with the correct path. Includes can be uploaded in .zip format.
Note: Source code from additional include directories will not be considered for analysis. It is only used to resolve header dependencies (type resolution).
Custom extensions for C & C++
This section covers how Embold supports custom extensions for C and C++ languages.
UI Scan
On UI, custom extensions can be specified with Additional Options field of scan configuration page as shown below:
The flag for supplying C extension is “--xc“ and for CPP is “--xcpp“.
User can provide multiple C/CPP extensions with the help of pipe (|) separator as shown above.
And multiple options can be provided using semicolon (;) separator.
Remote Scan
Custom extensions can be specified in remote scan mode also by updating additionalOptions field of gammascan.json file as shown below:
In gammascan.json, multiple options are specified using comma (,) separator.
With these additional options, all the files in repository ending with .pc or .pcc will be considered as C and C++ files respectively, and parsed accordingly.
Overwrite default scan configuration
This section helps to understand how you can override your scan configuration using a checked in file.
Default Configuration Settings
The default configuration is basically a committed JSON file. This JSON file should be committed to the base/root directory before scanning.
On Scan Configuration pop-up, the user can view the “Overwrite default scan configuration” checkbox.
- By default, this checkbox is unticked.
- But, if the user modifies any parameter in UI such as excludes or Additional Options or anything from code checkers, tick the checkbox “Overwrite default scan configuration“.
- The original parameters will be overridden.
Repository Configuration
In JSON file, by default, exclusions are empty.
For example: In the below image, we can see that whatever excludes we have added in Scan Configuration pop-up and if a user downloads the repository_configuration file (JSON), similar changes will be seen here.
For more information, refer to the “Embold scan configuration JSON”.
We cannot overwrite changes for remote repositories.
Embold Scan Configuration JSON
This section mentions how the data is stored in JSON (JavaScript Object Notation) and it can be customized by adding our own JSON objects.
Embold JSON structure
This section helps you to understand different field types, its description, its importance, and default values that areused in the JSON file.
| Field Type | Usage | Type | Default |
|---|---|---|---|
| gammaAccess | This is the name of Embold project. | ||
| url | Specifies URL of the website | String | Empty |
| userName | Stores user name for basic authentication to connect to the remote machine | String | Empty |
| password | Stores password for basic authentication to connect to the remote machine. User can access either using username-password or by adding token for authentication. | String | Empty |
| token | Stores token for basic authentication to connect to the remote machine. User can access either using username-password or by adding token for authentication. | String | Empty |
| repositories | Specifies the repository information that will be scanned during your analysis process. For more information, refer this article | ||
| dataDir | This is used to store the temporary files. | String | Empty |
| uid | Stores repository uid which is unique for each repository | String | Saved from UI |
| projectName | Specifies the repository name | Saved from UI | |
| sources | Path/directories/files that can be scanned during your analysis process. For more information, see as | ||
| baseDir | This is a base directory where source code resists. Moreover, this is a kind of main directory where all the operations such as cloning, scans, etc is performed. | String | Empty |
| exclusions | Specifies list of files/directories/languages that needs to be excluded from your analysis. | Array | Empty |
| settings | Specifies some additional options that can be used for scan configuration scanning process. | ||
| additionalOptions | Specifies the list or lists of additional options that needs to be scanned. | Array | Empty |
| includePaths | Specifies the file or list of files that needs to be scanned. | Array | Empty |
| modules | Modules can be used for scanning process. For more information, refer this article . | ||
| codeissues | This is a type of the module. This section is used to enabling/disabling code checker configuration. For more information, refer this article. | ||
| unittest | Specifies the input data value. For e.g. the input can be in CSV/XML or any other format. | List | |
| coverage | Specifies the coverage information across different modules. For more information, refer this linkk. | List | |
| relevance | Specifies the name of code issue. | List | |
| name | Specifies the name that can have multiple values such as codeissues, unittest, coverage or relevance. | List | Saved from UI |
| enabled | Specifies whether the code checker is enabled or not. | Boolean | Saved from UI |
| config | Specifies the configuration for that particular code checker | ||
| rules | Specifies the rules for code issue | Array | |
| options | These include additional options for code checkers like PMD. | ||
| name | Specify the rule name for different modules. | String | Saved from UI |
| type | Specifies the type of module. | String | Saved from UI |
| value | Specifies the value such as integer or string for that particular module. | String | Saved from UI |
| required | Specifies whether this module should be required while scanning and its value can be true or false. | Boolean | Saved from UI |
Sample JSON file
Embold JSON helps you to construct a data hierarchy such as:
Fast scan (Beta)
Fast scan enables you to perform a scan only on the changed files, thus giving you a significantly fast scanning experience.
Steps to enable fast scan:
1. Check Enable fast scan (beta) option from the scan pop-up.
Limitations:
In some circumstances such as the very first scan of a repository, fast scan option will not be available.
Remote scanning
A remote scan is an alternate way to scan with no UI intervention. This can be done using continuous integration toolchain, or manually via the command line. The results are then published on the UI automatically.
For languages such as C or C++ remote scan can help to increase the accuracy of the scan in some environments.
The remote scan is currently only offered with an on-premise installation of Embold. If you want to hear more about it, please get in touch with us and we will be happy to provide more information!
Embold CLI
Overview
The Embold CLI (Command Line Interface) analyses modified files or a complete repository.
It also generates a report containing code issues, design issues and metrics.
Prerequisites
- A valid Embold account is required. Contact the administrator to create an account.
- Git version (recommended version 2.18.0) should be installed.
- The project to be analyzed should be a Git repository.
- NodeJS (version 8.11) For JavaScript/Typescript repository analysis.
- Install CORONA through the installer. Verify if the CORONA_HOME environment variable is set.
Configuration
- Edit the file $CORONA_HOME/coronacli/config/cli.properties to add username, password and server address. Contact the administrator to get the username and password.
Example :
gamma.username= example@mygamma.io
gamma.password=password123
gamma.url= https://myembold.io/ - “
$CORONA_HOME/tools/bin” should be added in the PATH environment variable.
Embold CLI commands
Syntax: embold-analyse [options]
The Embold CLI provides multiple options like:
- -f: Analyses all the files in the current directory. For more information, refer to the supported languages link.
- -o: By default, generates the output on the console. Other options are CSV and XML.
- -h: Displays the usage.
Usages of Embold CLI:
Following are different usages of the Embold CLI :
| CLI Commands | Usage |
|---|---|
| embold-analyse | Analyses all the staged, unstaged and the untracked files in the current directory and the output is displayed on the console. |
| embold-analyse -f | Analyses all the files in the current directory and the output is displayed on the console. |
| embold-analyse -o | Analyses all the modified files in the current directory and by default displays the output on the console. |
| embold-analyse -o CSV | Analyze all the modified files in the current directory and generates the output in the CSV file. This file is by default saved in the base directory. |
| embold-analyse -o XML -f | Analyses all the modified files in the current directory and generates output in the XML file. This file is by default saved in the base directory. |
| embold-analyse -o CSV -f | Analyses all the modified files in the current directory and generates the output in CSV format. |
Snapshots
A snapshot represents the state of your source code at any given point in time. After every successful scan, a new snapshot is generated. While exploring various data visualisations, you can choose any snapshot to see results for that point in time.
To manage snapshots, select the Snapshots option from the
The number of snapshots you can store on each repository is dependent on your Embold license.
Example:
If your snapshot quota is 30 and your limit is exhausted, when you initiate the 31st scan, Embold will delete the earliest snapshot to make room for the new snapshot.
To avoid accidental deletion of snapshots by Embold, select the Keep snapshot option from Snapshots pop-up.
As per your preference, you can delete a snapshot manually by clicking the x button.