Installation with docker
Prerequisites
Operating system
For Docker-based installation, any Linux distribution with Docker support is required. Further information on the compatibility of Docker CE and EE can be found on the Docker help center: https://docs.docker.com/install/overview/.
Hardware
- We recommend a minimum of 2 dedicated cores
- For projects > 1 Mio. Lines of code 4 cores are recommended
- For projects > 3 Mio. Lines of code 8 cores are recommended.
Memory
- Minimum 8 GB RAM (dedicated)
- For projects > 1 Mio. Lines of code 16 GB RAM are recommended
- For projects > 3 Mio. Lines of code 32 GB RAM are recommended.
Storage
- Minimum of 100 GB storage
- Should be expandable to support a growing analysis history
- The table below gives an indication on the growing storage requirements:
Lines of Code | Storage size of 1 full snapshot |
100.000 | About 70-90MB |
1.000.000 | About 700MB – 1.2GB |
2.000.000 | About 1.2 – 2GB |
Table 1:Storage estimations
Network
During Installation
- The instance needs to be accessible through SSH.
After Installation
- One port needs to be exposed to access BrowserStack Code Quality via HTTP on the browser (by default port 3000)
- For remote scans where scans run on a build machine and results are published to the BrowserStack Code Quality server, port 5432 also needs to be exposed
- Connectivity to all external systems that should be integrated is required on the relevant ports-
- This includes connectivity to the repositories (i.e. git, GitHub, GitLab, TFS, SVN) on the required ports (i.e. HTTPS, SSH)
- If CI integration is required, connectivity between BrowserStack Code Quality on the CI system is required
- If task management is required, connectivity between BrowserStack Code Quality and the task management system needs to be there (i.e. for Jira, Teamforge)
Installing Docker engine
Docker engine needs to be installed on the host prior to installing the BrowserStack Code Quality Docker image.
Depending on your OS, refer the following steps to setup Docker Engine.
A. Docker Engine on CentOS
Follow the steps listed at this link.
B. Docker Engine on Ubuntu
Follow the steps listed at this link.
Environment Variables
Environment variable | Use |
---|---|
ssl_key | Used for ssl .key file path |
ssl_cert | Used for ssl .cert file path |
ssl_port | Used to specify ssl port |
ssl_passphrase | Passphrase for ssl certificates |
EMB_SCM_TRUST_SERVER_CERT | True/False, if scm has self signed certificates |
EMB_USE_NATIVE_PYPARSER | True/false, if we want to ue native python parser |
re_host | RE host url |
enableREScan | True/false. To enable rescan |
ACCEPT_EULA | Y, mandatory. ACCEPT_EULA confirms your acceptance of the End-User Licensing Agreement. |
ANALYSER_XMX | Analyser xmx value |
EMB_ANALYSER_THREADS | Number of analyser threads |
EMB_PARSER_THREADS | Number of parser threads |
EMB_CHECKER_THREADS | Number of checker threads |
polling_pr_cron_time | Cronjob for polling PR |
emboldSecurityChecker | True/false, enable security code checker |
gamma_ui_public_host | Specifies URL that can be accessible from the public domain |
Installation on docker
Using archive file
This article gives step-by-step instructions for the installation of docker using a docker archive file.
This document assumes the following BrowserStack Code Quality versions:
EMBOLD_VERSION=1.9.27.2
EMBOLD_RE_VERSION=0.9.1
If you are using a later version, please replace EMBOLD_VERSION and EMBOLD_RE_VERSION above with the corresponding values.
You can export these variables for the rest of the commands in this document to work.
Code Quality Components
Code Quality is made up of the following components:
- Code Quality Server (Docker container)
- This is the server component that hosts the results via a web interface and exposes results via APIs.
- The server can also initiate scans directly from the web interface by connecting it to a Git/SVN/TFS repo or just by uploading a ZIP file.
- Although for C/C++, we recommend running the scan on a build machine so BrowserStack Code Quality sees the source exactly how the compiler sees it.
- Code Quality Analyser (a.k.a. Corona)
Note: This is optional but recommended for C/C++ scans; Native installation / ZIP file- This component is used to run “Remote” scans, i.e. the Analyser runs on a different machine (typically the build machine) then the server, and publishes results to the server.
- Typically it is installed on a build machine (esp for C/C++) which has the build tools, compilers, and dependent libraries and headers.
- For C/C++, the Analyser component contains 2 sub-components:
- Code Quality Trace tool – Used to monitor the build and generate a compilation database file (e.g: embold-trace make).
To know more about embold-trace tool, refer to this article. - Analyser – Uses the generated compilation database and source files to run the scan and publish results.
- Code Quality Trace tool – Used to monitor the build and generate a compilation database file (e.g: embold-trace make).
To know more about C++ configuration, refer to this article.
- Code Quality Recommendation Engine for Docker container (optional)
- This component (when enabled) analyses Git/SVN history and its correlation with defects in Jira/Redmine to build a history-based defect correlation model, which is then used to predict defects in new code.
To know more about the Recommendation Engine for the beta version, check the link here.
Code Quality behind Proxy
Refer to this article, to know more about BrowserStack Code Quality behind proxy.
Code Quality
Installation Steps:
- There are two ways you can download Code Quality Docker package file:
- Recommended way is to use
wget
command as shown below
BROWSERSTACK_CQ_VERSION=1.9.27.2 wget https://v1.embold.io/nfs/BrowserStackCodeQuality_$BROWSERSTACK_CQ_VERSION/Docker/BrowserStackCodeQuality_$BROWSERSTACK_CQ_VERSION.tar.gz
- Alternate way is to Login to Customer Portal and go to Releases section and download from latest release folder.
- Now, Load the docker image using below command
docker load -i BrowserStackCodeQuality_$BROWSERSTACK_CQ_VERSION.tar.gz
New setup
Create the necessary directories and set the appropriate permissions using the following commands
mkdir BrowserStackCodeQuality cd BrowserStackCodeQuality mkdir gamma_data mkdir gamma_data/tmp mkdir logs chown -R 1001:1001 gamma_data/ logs/
You can either connect to an external PostgreSQL database or use a PostgreSQL Docker container as specified below.
Option 1: Using Docker compose.
If using an external PostgreSQL, skip the db
service and configure the database settings in the app
service accordingly.
version: '2' services: db: image: "postgres:16.3-alpine" container_name: BrowserStackCodeQuality-DB environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres ports: - "5432:5432" healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s timeout: 5s retries: 5 volumes: - /home/${USER}/BrowserStackCodeQuality/gamma_pg_data:/var/lib/postgresql/data app: image: "browserstack/code-quality:1.9.27.2" tty: true container_name: BrowserStackCodeQuality environment: - ACCEPT_EULA=Y - gamma_ui_public_host=http://localhost:3000 - RISK_XMX=-Xmx1024m - ANALYSER_XMX=-Xmx6072m - PGHOST=db - PGPORT=5432 - PGUSER=postgres - PGPASSWORD=postgres - GAMMA_DATABASE=gamma - ANALYTICS_DATABASE=corona depends_on: db: condition: service_healthy deploy: resources: limits: memory: 10G ports: - "3000:3000" volumes: - /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data - /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs - /home/${USER}/BrowserStackCodeQuality/gamma_data/tmp:/tmp
Option 2: Using Docker
If using an external PostgreSQL, skip the PostgreSQL container and provide the database configuration in the following Docker command.
- Run postgres 16.3 (if not using an external PostgreSQL)
docker run -d \ --name BrowserStackCodeQuality-DB \ -e POSTGRES_USER=postgres \ -e POSTGRES_PASSWORD=postgres \ -p 5432:5432 \ -v /home/${USER}/BrowserStackCodeQuality/gamma_pg_data:/var/lib/postgresql/data \ postgres:16.3-alpine
Get IP address of BrowserStackCodeQuality-DB and use in below docker command in place of DB_HOST
- Run CodeQuality
docker run -d \ --name BrowserStackCodeQuality \ -e ACCEPT_EULA=Y \ -e gamma_ui_public_host=http://YOUR_IP_OR_SERVER:PORT \ -e RISK_XMX=-Xmx1024m \ -e ANALYSER_XMX=-Xmx6072m \ -e PGHOST=DB_HOST \ -e PGPORT=5432 \ -e PGUSER=postgres \ -e PGPASSWORD=postgres \ -e GAMMA_DATABASE=gamma \ -e ANALYTICS_DATABASE=corona \ -p 3000:3000 \ -v /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data \ -v /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs \ -v /home/${USER}/BrowserStackCodeQuality/gamma_data/tmp:/tmp \ --user 1001:1001 \ --read-only \ browserstack/code-quality:1.9.27.2
Upgrade existing setup
Execute the following steps if you are existing Embold docker version is lower than 1.9.26.0.
Important: RDBMS (Postgres) has been separated from Docker. You can connect to your own RDBMS. Ensure that the Postgres version is >= 13.
Step 1: Backup using Postgres 16
- Create a Backup Directory
- Create a backup directory under the mapped
gamma_data
folder on your host machine. You can find the mapped path in the existing Docker command forgamma_data
. For example
- Create a backup directory under the mapped
-v /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data
- Run the following command to create the backup directory:
mkdir gamma_data/backup
- Take a Backup
- Execute the following Docker command to take a backup of the
corona
andgamma
- Execute the following Docker command to take a backup of the
docker exec -it BrowserStackCodeQuality sh -c "pg_dump --no-owner -U postgres corona -f /opt/gamma_data/backup/corona.sql && pg_dump --no-owner -U postgres gamma -f /opt/gamma_data/backup/gamma.sql"
Step 2: Restore the Database. You can connect to your RDBMS or use a Postgres Docker container. Choose one of the following options to connect to the database:
- Option 1: Use an external database
To restore a database on your RDBMS, follow these steps:- Create the databases
- Execute the following command to create the
corona
andgamma
databases. ReplaceYOUR_DB_HOST
andYOUR_DB_USERNAME
with your actual database host and username.
- Execute the following command to create the
- Create the databases
docker exec -it BrowserStackCodeQuality sh -c 'su - postgres -c "psql -h YOUR_DB_HOST -U YOUR_DB_USERNAME -d postgres -c \"CREATE DATABASE corona;\" && psql -h YOUR_DB_HOST -U YOUR_DB_USERNAME -d postgres -c \"CREATE DATABASE gamma;\""'
- Enter the database password (if prompted).
- Restore the Database
- Execute the following command to restore the
corona
andgamma
databases from backup files. ReplaceYOUR_DB_HOST
andYOUR_DB_USERNAME
with your actual database host and username.
- Execute the following command to restore the
docker exec -it BrowserStackCodeQuality sh -c 'su - postgres -c "psql -h YOUR_DB_HOST -U YOUR_DB_USERNAME -d corona -f /opt/gamma_data/backup/corona.sql && psql -h YOUR_DB_HOST -U YOUR_DB_USERNAME -d gamma -f /opt/gamma_data/backup/gamma.sql"'
- Enter the database password (if prompted).
- Remove the Existing Embold Container
docker rm -f BrowserStackCodeQuality
- Option 2: Using a Postgres Docker Container
- Run Postgres Container
- Execute the following command to run a Postgres container:
- Run Postgres Container
docker run -d --rm \ --name BrowserStackCodeQuality-DB \ -e POSTGRES_USER=postgres \ -e POSTGRES_PASSWORD=postgres \ -v /home/${USER}/BrowserStackCodeQuality/gamma_pg_data:/var/lib/postgresql/data \ -v /home/${USER}/BrowserStackCodeQuality/gamma_data/backup:/backup \ postgres:16.3-alpine
- Create the databases
- Execute the following command to create the
corona
andgamma
databases:
- Execute the following command to create the
docker exec -it BrowserStackCodeQuality-DB sh -c 'su - postgres -c "psql -U postgres -c \"CREATE DATABASE corona;\" && psql -U postgres -c \"CREATE DATABASE gamma;\""'
- Restore the Database
- Execute the following command to restore the
corona
andgamma
databases:
- Execute the following command to restore the
docker exec -it BrowserStackCodeQuality-DB sh -c 'su - postgres -c "psql -U postgres -d corona -f /backup/corona.sql && psql -U postgres -d gamma -f /backup/gamma.sql"'
- The database restoration is now complete.
- Remove the DB and Existing Embold Container
docker rm -f BrowserStackCodeQuality-DB docker rm -f BrowserStackCodeQuality
Step 3: Final step: Prepare and Run Docker-Compose
- Execute the following commands to create a temporary directory and set the appropriate permissions:
mkdir gamma_data/tmp chown -R 1001:1001 gamma_data/ logs/
- Run Docker-Compose
If using an external PostgreSQL, skip thedb
service and configure the database settings in theapp
service accordingly.
version: '2' services: db: image: "postgres:16.3-alpine" container_name: BrowserStackCodeQuality-DB environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres ports: - "5432:5432" healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s timeout: 5s retries: 5 volumes: - /home/${USER}/BrowserStackCodeQuality/gamma_pg_data:/var/lib/postgresql/data app: image: "browserstack/code-quality:1.9.27.2" tty: true container_name: BrowserStackCodeQuality environment: - ACCEPT_EULA=Y - gamma_ui_public_host=http://locahost:3000 - RISK_XMX=-Xmx1024m - ANALYSER_XMX=-Xmx6072m - PGHOST=db - PGPORT=5432 - PGUSER=postgres - PGPASSWORD=postgres - GAMMA_DATABASE=gamma - ANALYTICS_DATABASE=corona depends_on: db: condition: service_healthy deploy: resources: limits: memory: 10G ports: - "3000:3000" volumes: - /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data - /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs - /home/${USER}/BrowserStackCodeQuality/gamma_data/tmp:/tmp
Reference diagram
Below illustration helps you understand how the scans work on Code Quality Server.
Recommendation Engine
Overview:
- In this section, we will see how to setup Recommendation Engine and Embold on same Linux host.
- We will also see how an existing embold server can be integrated with RE .
Installation steps
- Prerequisites:
- docker engine
- docker-compose, version 1.25.0 or up
- In order to install Embold RE with docker-compose, you need 3 images:
- bitnami/mongodb:4.2.21 (Available on docker hub as public image)
- bitnami/elasticsearch:7.17.5 (Available on docker hub as public image)
- embold/gammare-snapshot:<current ver> (Available on Embold private docker repo,or as a tar.gz file in your Embold account)
- If the host where you will install RE has internet access, you need to download and load only embold/gammare-snapshot: prior to running docker-compose
- Image procurement and setup process
- Download Embold RE image (tar.gz) from your Embold account (Releases section : https://codequality.browserstack.com/self-managed/v1/portal/)
Or, You can run below command to downloadwget https://v1.embold.io/nfs/embold_1.9.16.1/RE/embold_re_0.9.2-SNAPSHOT.tar.gz
- On the docker host machine that will run RE, run:
docker load -i embold_re_0.9.2-SNAPSHOT.tar.gz
- If you need to download the mongodb and elasticsearch images as well, run the steps below: On some machine with internet access:
docker pull bitnami/mongodb:4.2.21
docker save -o mongodb.tar.gz bitnami/mongodb:4.2.21
docker pull bitnami/elasticsearch:7.17.5
docker save -o elasticsearch.tar.gz bitnami/elasticsearch:7.17.5
- On the docker host machine that will run RE, run:
docker load -i mongodb.tar.gz
docker load -i elasticsearch.tar.gz
- Prepare the docker-compose script, directories and environment variables
- Create an environment file (e.g. re_env) in a folder with these 2 variables in it:
# Edit to your actual host paths!!
GAMMA_RE_DATA=/some/path/on/host/gamma_re_data
GAMMA_RE_LOGS=/some/log/path/gamma_re_logs
- IMPORTANT!!
Create the directories/some/path/on/host/gamma_re_data/mongo_data/data
and /some/path/on/host/gamma_re_data/elastic_data
on the host- Change recursive ownership of to the user/group 1001:1001 like below:
chown -R 1001:1001 /some/path/on/host/gamma_re_data
chown -R 1001:1001 /some/log/path/gamma_re_logs
- Download Embold RE image (tar.gz) from your Embold account (Releases section : https://codequality.browserstack.com/self-managed/v1/portal/)
- Creation of docker-compose yaml
- Now with all components in place, you can run docker-compose with this sample script i.e docker-compose.yaml file:
version: "3" services: db: image: bitnami/mongodb:4.2.21 # Internal port: 27017 volumes: - ${GAMMA_RE_DATA}/mongo_data:/bitnami/mongodb indexer: image: bitnami/elasticsearch:7.17.5 # Internal port: 9200 volumes: - ${GAMMA_RE_DATA}/elastic_data:/bitnami/elasticsearch/data gammare: image: embold/gammare-snapshot:0.9.2-SNAPSHOT container_name: gammare environment: - ES_DB_URL=mongodb://db:27017/ - ES_INDEXER_URL=http://indexer:9200 ports: - "9090:9090" # Only port exposed to outside world volumes: - ${GAMMA_RE_DATA}:/opt/gammare_data - ${GAMMA_RE_LOGS}:/opt/gammare_logs gamma: image: embold/gamma:1.9.16.1 container_name: gamma environment: - SCM_THREADS=1 - EMB_SCM_TRUST_SERVER_CERT=true - EMB_USE_NATIVE_PYPARSER=TRUE - re_host=http://gammare:9090 - enableREScan=true - polling_pr_cron_time="*/2 * * * *" - gamma_ui_public_host=http://localhost:3000 - RISK_XMX=-Xmx1024m - ACCEPT_EULA=Y - ANALYSER_XMX=-Xmx3072m ports: - "3000:3000" volumes: - /home/gamma:/opt/gamma_data - /home/pg:/var/lib/postgresql - /home/logs:/opt/gamma/logs
- Save above file as “docker-compose.yaml” in the same directory where you have created re_env file
- Start the embold services with the following command from directory where re_env file is created:
docker-compose --env-file=./re_env up -d
- If all goes well, the last few messages in $GAMMA_RE_LOGS/cmodsrv.log (
GAMMA_RE_LOGS=/some/log/path/gamma_re_logs
) should look like:cmoddb.py:27 INFO| Initialized CmodDb
cmoddb.py:27 INFO| Initialized CmodDb
cmoddb.py:266 INFO| Cleared 0 requests from build queue
To Activate BrowserStack Code Quality goto activation.
Integrate existing standalone embold docker with RE
NOTE : Before you jump to this section , complete all the above steps except execution of docker-compose file.
Assume below is the docker command you have used to run standalone embold docker.
docker run -m 10GB -d -p 3000:3000 --name EMBOLD -e gamma_ui_public_host=
http://<IP/domain name>:<Port> -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx6072m -v /home/${USER}/embold/gamma_data:/opt/gamma_data -v /home/${USER}/embold/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/embold/logs:/opt/gamma/logs embold/gamma:$EMBOLD_VERSION
So In above docker-compose.yaml, you have to edit only gamma section,
In environment, add -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx6072m
from the standalone docker command
In volumes, replace values and use -v /home/${USER}/embold/gamma_data:/opt/gamma_data -v /home/${USER}/embold/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/embold/logs:/opt/gamma/logs
from the standalone docker command
So finally your docker-compose.yaml will be as below
version: "3" services: db: image: bitnami/mongodb:4.2.21 # Internal port: 27017 volumes: - ${GAMMA_RE_DATA}/mongo_data:/bitnami/mongodb indexer: image: bitnami/elasticsearch:7.17.5 # Internal port: 9200 volumes: - ${GAMMA_RE_DATA}/elastic_data:/bitnami/elasticsearch/data gammare: image: embold/gammare-snapshot:0.9.2-SNAPSHOT container_name: gammare environment: - ES_DB_URL=mongodb://db:27017/ - ES_INDEXER_URL=http://indexer:9200 ports: - "9090:9090" # Only port exposed to outside world volumes: - ${GAMMA_RE_DATA}:/opt/gammare_data - ${GAMMA_RE_LOGS}:/opt/gammare_logs gamma: image: embold/gamma:1.9.16.1 container_name: gamma environment: - re_host=http://gammare:9090 - enableREScan=true - gamma_ui_public_host=http://<IP/domain name>:<Port> - RISK_XMX=-Xmx1024m - ACCEPT_EULA=Y - ANALYSER_XMX=-Xmx3072m ports: - "3000:3000" volumes: - /home/${USER}/embold/gamma_data:/opt/gamma_data - /home/${USER}/embold/gamma_psql_data:/var/lib/postgresql - /home/${USER}/embold/logs:/opt/gamma/logs
Save above file as “docker-compose.yaml” in the same directory where you have created re_env file
- Start the embold services with the following command from directory where re_env file is created:
docker-compose --env-file=./re_env up -d
If all goes well, the last few messages in $GAMMA_RE_LOGS/cmodsrv.log (GAMMA_RE_LOGS=/some/log/path/gamma_re_logs
) should look like:
cmoddb.py:27 INFO| Initialized CmodDb
cmoddb.py:27 INFO| Initialized CmodDb
cmoddb.py:266 INFO| Cleared 0 requests from build queue
For configuration of a repository with a Recommendation Engine for the beta version, check the link here.
Memory settings
Whenever you want to scan a large repository (above 2 Millions lines of code) remember you need these memory settings. Before you run such scan make sure to tweak few environment values such as -m
and ANALYSER_XMX
. Note that, such values may not be known upfront when the docker run command is triggered. We recommend values as shown in the table below:
Lines of code | ANALYSER_XMX | -m | Embold Server/UI Scan (RAM) | Remote Scan (RAM) |
Upto 1 Million | 6GB | 12GB | 16GB | 16GB |
Upto 2-10 Millions | 15GB | 30GB | 32GB | 32GB |
The below docker run command is an example where we increased the -m
and ANALYSER_XMX
values.
docker run -m 30GB -d -p 3000:3000 --name BrowserStackCodeQuality -e gamma_ui_public_host=http://: -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx15360m -v /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data -v /home/${USER}/BrowserStackCodeQuality/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs browserstack/code-quality:$BROWSERSTACK_CQ_VERSION
For your information, a significant part of memory allocated using -m
to the docker is used by the various processes of Code Quality which includes the UI, Controller Process, and Processes involved in the actual analysis of the source code. Among these processes, the analyzer process a.k.a ANALYSER_XMX
uses the maximum memory.
It is also observed in most instances that the ANALYSER_XMX
value is proportional to the actual number of Lines Of Code (LOC) scanned for a repository. The ANALYSER_XMX
value should not be more than 70% of the allocated value for -m
. Also, the environment variables passed to docker such as ANALYSER_XMX
and RISK_XMX
are mutually exclusive for a given analysis.
For example, a repository with 1 Million lines of code will require a minimum of 6 to 7GB ANALYSER_XMX
. Said that, few other factors may impact the ANALYSER_XMX
value – like within this repository, there could be a surge of code duplication that will demand the ANALYSER_XMX
value to be set to 8GB and the -m
value to set to 12GB.
Therefore, a general thumb rules to follow while setting ANALYSER_XMX
value are:
- For a repository with less than 1 Million lines of code, that equals 35-37% of the total memory allocated for the container.
- For a repository with less than 3 Millions lines of code, that equals 40-42% of the total memory allocated for the container.
- For a repository with less than 5 Millions lines of code, that equals 50-55% of the total memory allocated for the container.
Note: For C++ repositories to achieve more accuracy we recommend to perform Remote scan with Strict mode. For more details click here
Multi-Threaded Execution
Enhance the analysis process by running the application in multi-threaded mode for faster results.
Ensure your system configuration includes an adequate number of processes to support multi-threaded execution.
Run the Analyzer in Multi-Threaded Mode : By default, the analyzer operates with four threads. For further customisation, set the EMB_ANALYSER_THREADS environment variable on the machine where Embold scan is running. If using Docker, include this variable in the Docker command; for installations, incorporate it into the installer machine.
Run Java Parser in Multi-Threaded Mode : The Java Parser utilizes two threads by default. To adjust the thread count for the Embold Java Parser, set the EMB_PARSER_THREADS environment variable on the machine running Embold Scan. For Docker, integrate this variable into the Docker command; for the installer, incorporate it into the installer machine.
Run CPP Parser in Multi-Threaded Mode : By default CPP Parser operates with single thread. To enable multi-threaded mode, follow these steps:
- Go to Projects and on Repository list click on repository context menu (ooo) and then click Scan configuration menu item
- Add “–jobs=[no of multiple process to be launched]” under Additional Options field (for example see below screenshot)
- This settings can also be configured using repository_configuration.json. For that, go to repository context menu (ooo) and click on Download repository configuration and add/edit additional options under settings section (as shown below)
"settings": {
"additionalOptions": [
"--jobs=4"
],
"includePaths": []
},
Upgrade to newer version
Steps:
- Before upgrading we recommend you take backup of the volumes mounted. For example as below:
/home/${USER}/BrowserStackCodeQuality/gamma_data
/home/${USER}/BrowserStackCodeQuality/gamma_psql_data
/home/${USER}/BrowserStackCodeQuality/gamma_psql_data
- Stop and remove existing running docker container
- Login to Customer Portal and go to Releases section
- Download latest docker
<BROWSERSTACK_CQ_VERSION>.tar.gz
file - Load docker container on your local machine
docker load -i BrowserStackCodeQuality_<BROWSERSTACK_CQ_VERSION>.tar.gz
- Re-run docker container using below command (or use the same docker run command which you used earlier)
docker run -m 12GB -d -p 3000:3000 --name BrowserStackCodeQuality -e ACCEPT_EULA=Y -e gamma_ui_public_host=http://:3000 -e EMB_USE_NATIVE_PYPARSER=TRUE -e RISK_XMX=-Xmx2024m -e ANALYSER_XMX=-Xmx8072m -v /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data -v /home/${USER}/BrowserStackCodeQuality/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs browserstack/code-quality:<BROWSERSTACK_CQ_NEW_VERSION>
Installation on Windows
Before you begin, please verify that you have a supported version of Windows here.
NOTE: Always use clean Windows VM or clean Bare metal for Installation with min 16GB RAM and 100GB HDD
Pre-installation checks
Following are the prerequisites for installing Code Quality on Windows:
- .NET Framework 4.5 (required for running Code Quality windows installer)
- PowerShell version 5.1
- Google chrome browser
For more information, refer specifications section.
Below are the pre-installation check steps:
.NET Version check:
- Open PowerShell in admin mode and run below command:
(Get-ItemProperty "HKLM:SOFTWAREMicrosoftNET Framework SetupNDPv4Full").Release -ge 378389
- If the above command is “True”, then the default framework is installed.
- If the above command is “False”, then install .NET Framework 4.5 or a newer version.
- .NET Framework 4.5 is the default version.
PowerShell version check:
- Open PowerShell in admin mode and run below command to check the current version:
$PSVersionTable
. - If the version is not 5.1, install Windows Management Framework “Win7AndW2K8R2-KB3191566-x64.msu” placed under “..preInstallationsRequiredpowershell5.1 Win7AndW2K8R2-KB3191566-x64.zip”.
- Ensure that PowerShell terminal is closed before installing.
Set PowerShell execution policy:
- After confirming PowerShell version to be 5.1, open PowerShell terminal as administrator and enter the command “Set-ExecutionPolicy RemoteSigned” and choose Yes in the flow.
Dependencies for C# language support:
If the user wants C# language support for Code Quality, ensure to have:
- .NET Core 5.0
- In addition to the pre-requisites, you can install more pre-requisites based on your repository settings.
Note:
- Ensure that the machine on which Code Quality is getting installed has no Node.js installed.
- While installing .net framework, it may ask for user intervention. Choose all the default settings and complete the installation.
Installation steps
Below are the steps to install BrowserStack Code Quality on Windows:
- Download the installer file from your BrowserStack Code Quality Customer Portal > Releases tab > Installers. There will be file with name similar to the following: BrowserStackCodeQuality_windows_1.9.21.0.zip.
- Right-click the zip folder, click Properties > General tab.
- In the ‘Security‘ section, if the Security button is blocked, unblock it and then proceed with the unzipping of the file.
Note:- If the security button is not blocked, you can directly proceed with the unzipping of the file. - Unzip the BrowserStackCodeQuality_windows_1.9.21.0.zip file.
- Go to setup folder.
- Open command prompt in admin mode, change the directory (folder) using cd to the path (gammaWindowsInstallersetup) where setup.bat file is located and run the command:
setup.bat
- This will install the pre-requisites for the gamma including required frameworks, tools, etc.
The script will set all the required environment variables. - On successful installation, BrowserStack Code Quality login page will be available at localhost:3000.
- Activate BrowserStack Code Quality.
Enable C# Security Checks
- To enable Microsoft Security CodeScan tool for C#, install the following SDKs in order:
- https://download.visualstudio.microsoft.com/download/pr/f92c52da-2ef6-44f2-a296-487f94c2c37a/258dc2e61ff8bec7d90aee3ca1e7d8a3/dotnet-sdk-5.0.406-win-x64.exe
- https://download.visualstudio.microsoft.com/download/pr/962fa33f-e57c-4e8a-abc9-01882ff74e3d/23e11ee6c3da863fa1489f951aa7e75e/dotnet-sdk-3.1.417-win-x64.exe
- Restart all 3 Gamma related services from services.msc
Code Quality behind Proxy
If Code Quality is located behind a proxy, additional information should be provided in the configuration file or docker command.
For Docker
Use -e to pass environment variable or --env-file
to provide file of environment variables.
e.g. with -e option
docker run -m 8GB -d -p 3000:3000 -p 5432:5432 --name BrowserStackCodeQuality -e gamma_ui_public_host=http://<IP/domain name>:<Port> -e ACCEPT_EULA=Y -e EMB_USE_NATIVE_PYPARSER=TRUE -e RISK_XMX=-Xmx1024m -e ANALYSER_XMX=-Xmx3072m -e no_proxy=localhost,127.0.0.0/8,::1 -e http_proxy=http://PROXY_IP:PROXY_PORT -e https_proxy=http://PROXY_IP:PROXY_PORT -e CATALINA_OPTS="-Dhttps.proxyHost=PROXY_IP -Dhttps.proxyPort=PROXY_PORT -Dhttp.proxyHost=PROXY_IP -Dhttp.proxyPort=PROXY_PORT -Dhttp.NoproxyHost='localhost|127.*|::1|192.168.*' -DGAMMA_ROOT=/opt/gamma -Xmx1024m -Djava.awt.headless=true -Djava.library.path=/usr/lib/x86_64-linux-gnu/jni/:/usr/lib64/" -v /home/${USER}/docker/data:/opt/gamma_data -v /home/${USER}/docker/db:/var/lib/postgresql -v /home/${USER}/docker/logs:/opt/gamma/logs browserstack/code-quality:${VERSION}
With --env-file
option,
Use --env-file=env.list
to read a file of environment variables.
env.list
file looks like this:
ACCEPT_EULA=Y
RISK_XMX=-Xmx1024m
ANALYSER_XMX=-Xmx3072m
http_proxy=http://PROXY_IP:PROXY_PORT
https_proxy=http://PROXY_IP:PROXY_PORT
no_proxy=localhost,127.0.0.0/8,::1
-e gamma_ui_public_host=http://<IP/domain name>:<Port>
CATALINA_OPTS=-Dhttps.proxyHost=PROXY_IP -Dhttps.proxyPort=PROXY_PORT -Dhttp.proxyHost=PROXY_IP -Dhttp.proxyPort=PROXY_PORT -Dhttp.NoproxyHost='localhost|127.*|::1|192.168.*' -DGAMMA_ROOT=/opt/gamma -Xmx1024m -Djava.awt.headless=true -Djava.library.path=/usr/lib/x86_64-linux-gnu/jni/:/usr/lib64/
For Installer
Modify /etc/default/gamma
1. Change CATALINA_OPTS environment variable with given settings:
export CATALINA_OPTS="-Dhttps.proxyHost=PROXY_IP -Dhttps.proxyPort=PROXY_PORT -Dhttp.proxyHost=PROXY_IP -Dhttp.proxyPort=PROXY_PORT -DGAMMA_ROOT=/opt/gamma -Xmx1024m -Djava.awt.headless=true -Djava.library.path=/usr/lib/x86_64-linux-gnu/jni/:/usr/lib64/"
2. Also add new environment variables as below:
export https_proxy=http://PROXY_IP:PROXY_PORT
export http_proxy=http://PROXY_IP:PROXY_PORT
export no_proxy='127.0.0.1,localhost'
Backup & Restore
This guide will help you to backup your BrowserStack Code Quality data and restoring it. With this process, you can backup your source code, scans with all its prerequisites.
Prerequisites for database backup
- The BrowserStack Code Quality version on the source and destination machines should be the same.
- Stop BrowserStack Code Quality service on the destination machine.
- You must have administrative privileges.
- The script works with default database names as corona and gamma.
- Share/mount the directory which shall be accessible to source machine if the backup/restore is been done on the remote machine.
- Create a backup directory beforehand and provide write permissions where the backup is to be saved.
- You must be aware of the BrowserStack Code Quality installation directory path for backup/restore on the same platforms.
- In the case of Linux platform, to restore BrowserStack Code Quality, remove the password authentication from the remote destination.
Backup & restore on Windows
This will include the information on database backup and restoration of the Windows operating system.
Database backup
A. Database backup on windows
The databases from the source BrowserStack Code Quality machine is backed up using the following steps:
- Get the batch file at C:\ProgramData\Gamma\gamma_ui\dbscripts\scripts
\backup_restore.bat . - Execute the file. A command prompt will be displayed. Follow the inquiries to backup the database.
B. Files backup on windows
The source files from the source BrowserStack Code Quality machine are backed up using the following steps:
- Get the batch file at
C:\ProgramData\Gamma\gamma_ui\dbscripts\scripts\backup_restore.bat . - Execute the file. The command prompt will be displayed. Follow the inquiries to back up the source and logs.
- To back up the logs, provide the destination directory as shown in the below screens.
Database restoration
A. Database restoration on windows
The databases from the source machine is restored using the following steps:
- Get the batch file at
C:\ProgramData\Gamma\gamma_ui\dbscripts\scripts\backup_restore.bat. - Execute the file. The command prompt will be displayed. Follow the inquiries to restore the database.
- Provide the same sql paths which we have generated in destination folder during Backup-Process
B. Files restoration on windows
The source files from the source machine are restored using the following steps:
- Get the batch file at
C:\ProgramData\Gamma\gamma_ui\dbscripts\scripts\backup_restore.bat . - Execute the file. The command prompt will be displayed. Follow the inquiries to back up the source and logs.
- Provide the same destination data folder path which we have generated during Backup-Process
- To restore the logs provide the destination directory as shown in the below screen.
- Provide the same destination logs folder path which we have generated during Backup-Process
Backup & restore on Docker
This will include the information on backup and restoration of BrowserStack Code Quality data for docker setup. Below are the 2 scenarios
- backup & restore on same machine i.e on same docker
- backup & restore from one machine to another machine i.e from one docker to another docker on another machine
Follow the steps based on source and destination platforms.
Docker : backup & restore on same machine:
Suppose you have used below command for execution of docker
docker run -m 10GB -d -p 3000:3000 --name CODEQUALITYBACKUP -e gamma_ui_public_host=http://3.72.41.7:3000 -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx3072m -v /Users/saurabh/dockervolumes/data:/opt/gamma_data -v /Users/saurabh/dockervolumes/db:/var/lib/postgresql -v /Users/saurabh/dockervolumes/logs:/opt/gamma/logs embold/gamma:1.9.18.0
NOTE : In below document above volume mapping host paths are used for reference only. Make sure you use your volume paths used in your docker command.
Backup phase :
- First login to Code Quality with your credentials and enable backup from settings and schedule it as per your requirements. PFA Below image for reference.
- Once schedule make sure backup_path shown in above image is mapped to host_path via docker volumes . By default it is mapped to
/opt/gamma_data/
inside the docker - Inside host at path
/Users/saurabh/dockervolumes/data
(As per our reference docker command at start of document) it will generate .sql files PFA snapshot for reference. There will be two .sql files created in the format as belowgamma_database_backup_
DDMMYYYY_HHMMSS.sqlcorona_database_backup_
DDMMYYYY_HHMMSS.sql. - There can be multiple sql generated due to backup execution at different intervals as per schedule interval, but make sure we should use both sql with same date and time for restoration.
For e.g :gamma_database_backup_07092023_102900.sql
andcorona_database_backup_07092023_102900.sql
- For reference see below image.
- Inside docker, By default backup path is
/opt/gamma_data
which hold all the sql with data. - Now create a directory for DB sql on docker host machine.
For eg: –/home/user/Documents/dbrestore
- Move only 2 sql file i.e gamma and corona sql from the backup_path to
/home/user/Documents/dbrestore
. Make sure we should use both sql with same date and time for restoration
For e.g :gamma_database_backup_07092023_102900.sql
andcorona_database_backup_07092023_102900.sql
- Get docker container ID and navigate inside it through command
docker exec -it <container-id> bash
Create “dbrestore” folder inside /home directory and exit the container. - Execute the below commands from host machine to move data from
/home/user/Documents/dbrestore
to/home/dbrestore
folder inside the docker:
docker cp /home/user/Documents/dbrestore/gamma_database_backup_07092023_102900.sql CODEQUALITYBACKUP:/home/dbrestore docker cp /home/user/Documents/dbrestore/corona_database_backup_07092023_102900.sql CODEQUALITYBACKUP:/home/dbrestore
For reference see below image.
Restore phase :
- Get inside the container and run below command
pm2 stop app
- To Restore execute shell script located at
/opt/gamma//gamma_ui/dbscripts/scripts/backup_restore.sh
. It will prompt you for inputs see below image for reference and provide same except the name of your sql files.
- Once DB restore script is executed , Run command
pm2 start app
- Exit the docker.
- You have successfully restored your database to the backup point of generated sql.
Docker : backup & restore from one machine to another machine.
Backup phase :
- First login to Code Quality with your credentials and enable backup from settings and schedule it as per your requirements. PFA Below image for reference.
- Once schedule make sure custom backup_path is mapped to host_path via docker volumes . By default it is mapped to
/opt/gamma_data/
inside the docker - Inside host at path
/Users/saurabh/dockervolumes/data
(As per our reference docker command at start of document) it will generate .sql files PFA snapshot for reference. There will be two sql files created in the format as belowgamma_database_backup_DDMMYYYY_HHMMSS.sql
corona_database_backup_DDMMYYYY_HHMMSS.sql
. - There can be multiple sql generated due to backup execution at different intervals as per schedule interval, but make sure we should use both sql with same date and time for restoration.
For e.g :gamma_database_backup_07092023_102900.sql
andcorona_database_backup_07092023_102900.sql
- For reference see below image
- Copy these sql files to new host machine at place :
/home/backup_sql
- Copy your entire gamma_data volume on host machine, in our case as per our docker command i.e
/Users/saurabh/dockervolumes/data
(As per our reference docker command at start of document) to the new host machine and store it at location :/home/backup_data
.
Restore Phase :
- On new machine use /home/backup_data to map to /opt/gamma_data, execute below docker command
docker run -m 10GB -d -p 3000:3000 --name CODEQUALITYRESTORE -e gamma_ui_public_host=http://3.72.41.7:3000 -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx3072m -v /home/backup_data:/opt/gamma_data -v /newpath/db:/var/lib/postgresql -v /newpath/logs:/opt/gamma/logs embold/gamma:1.9.18.0
- After activation, get inside the container and run below command
pm2 stop app
- Now create a directory for DB sql on docker host machine.
For eg: –/home/user/Documents/dbrestore
- Move only 2 sql file i.e gamma and corona sql from
/home/backup_sql
to/home/user/Documents/dbrestore
. Make sure we should use both sql with same date and time for restoration
For e.g :gamma_database_backup_07092023_102900.sql
andcorona_database_backup_07092023_102900.sql
- Get docker container ID and navigate inside it through command
docker exec -it <container-id> bash
Create “dbrestore” folder inside /home directory and exit the container. - Execute the below commands from host machine to move data from
/home/user/Documents/dbrestore
to/home/dbrestore
folder inside the docker:
docker cp /home/user/Documents/dbrestore/gamma_database_backup_07092023_102900.sql CODEQUALITYRESTORE:/home/dbrestore docker cp /home/user/Documents/dbrestore/corona_database_backup_07092023_102900.sql CODEQUALITYRESTORE:/home/dbrestore
For reference see below image.
- To Restore execute shell script located at
/opt/gamma//gamma_ui/dbscripts/scripts/backup_restore.sh
. It will prompt you for inputs see below image for reference and provide same except the name of your sql files.
- Once DB restore script is executed , Run command
pm2 start app
- Exit the docker.
- You have successfully restored your database to the backup point of generated sql.
C++ Configuration
For setting up an environment for C++, you can perform configuration steps for installing files, databases and so on.
C++ Strict Mode
In strict mode, CXXParser uses a compilation database to parse files. Compilation database contains actual compile commands and options used to compile source files in a project which helps CXXParser in accurate parsing. Compilation database can be generated using build tools such as cmake, make, waf, scons, etc.
Embold Trace
Introduction
BrowserStack Code Quality uses Compilation Database (CDB) to scan C/C++ repository in strict mode. BrowserStack Code Quality provides a tool ‘embold-trace’ to generate CDB from build process. BrowserStack Code Quality-trace tool facilitates users to configure which compilers to intercept during build. It then intercepts build process and generates CDB based on configured compilers.
Supported Platforms
Before you begin, please verify that you have a supported version of operating systems here.
Prerequisites
- Linux
strace: A tool for debugging and troubleshooting programs in Unix-like operating systems such as Linux.
Read more about strace here. - Installation of strace
For Ubuntu→$ sudo apt-get install strace
For CentOS/Red Hat→$ sudo yum install strace
How to use
- To run the embold-trace, run the below command:
$ embold-trace [embold-trace-options] {your build command} [build-options]
- Use `
--help
` to know more about the embold-trace options.
MSBuild on Windows
BrowserStack Code Quality-trace has two modes of operation for tracing MSBuild projects:
- Using .tlog files→Uses MSBuild generated tracker log files of the form CL.command.1.tlog to generate CDB
This mode can be enabled using -t option of embold-trace
$
embold-trace
-t
msbuild project.sln
- Trace files → Uses intermediate trace files to generate CDB.
This is the default mode of operation
Output Directory
Embold-trace supports generating CDB in custom directory with option -o $
embold-trace
-o
{custom/directory}
Configure embold-trace
BrowserStack Code Quality-trace uses json configuration files to configure compilers to intercept. There are two types of configuration files.
1. Top-level configuration file:
This file list down the compilers names to intercept and corresponding translation file name (excluding extension). BrowserStack Code Quality-trace provides compiler configuration for all the standard compilers.
The default name is ’embold-trace-default.json’
- Source Extensions
The default configuration file lists supported extensions for C/C++ source files. Users should add the extension which needs to be supported to this configuration file. (if not already present)
- Translation file:
It contains the corresponding compiler options to specify include directory, include file and define macro.
Key-Value description in translation json as follows :
Key | Value | Description | Example |
---|---|---|---|
includeDirPrefix | Option used to specify include directory | -I (GNU Family) | -I (GNU Family) |
defineMacroPrefix | Option used to define macro | -D (GNU Family) | -D (GNU Family) |
includeFilePrefix | Option used to include single file during compilation | -include (GNU Family) | -include (GNU Family) |
sourceFilePrefix | Prefix used to specify source files in compile command | Prefix used to specify source files in compile command. Green Hills Compiler uses flag --source to specify source file in compile command. Example: $ ease850 --source=abc.cpp | --source (GH Compiler) |
optionFilePrefix | Prefix used to specify build system options file. Options file contains compiler options for include directory, define macro etc. | Instead of directly specifying options like -I, -D in compile commands, some compilers or build systems uses a special file called option file which contains compiler flags. This file is then provided in compile command using a prefix. The value of this key is that prefix. Example: $ cctc --option-file=option-file.txt abc.cpp | --option-file (Tasking Tri-core) |
optionFilePattern | A pattern to specify build system options file. Each build system or compiler has different name format for options file. So, a regex should be specified as value. | Option files can also be specified using pattern instead of prefix. Example: CMake build system generates response files whose names are like includes_CXX.rsp for C++ compilation and includes_C.rsp for C compilation. These files contain includes and define options required for compilation. And these response files are supplied usually using ‘@’ character at the start but not necessarily. Example: "command": "g++ @CMakeFiles/TestProject/includes_C.rsp -o CMakeFiles\ TestProject\test.o -c C:\TestProject\test.c" A regex which match these file names should be provided as value of optionFilePattern in translation file. Then, if embold-trace finds such file pattern in compile command, then it replace the content of the option file in compile command for Embold to parse correctly. | .*includes_CXX.rsp (CMake Build) |
Where to find
By default, all the provided configuration files (top-level and translation) are present in the directory where embold-trace binary is present. Any new translation file should also be added to the same directory.
However, you can keep all the configuration files in a separate directory. In this case, the top-level configuration file path should be given as input to embold-trace while invocation using option “-c
”$ embold-trace -c
2. Adding Non-Standard Compiler Name
If you have a standard compiler from the above list but a non-standard name, you must edit the default configuration file.
Example:
If a GCC compiler 4.9 is installed and compiler name is ‘gcc-4.9’
Then, add an entry to top-level configuration file.
Adding Unsupported Compiler
If your compiler is name is not found in top-level configuration file and it is not a standard compiler like GCC or Clang, then that compiler is non supported by embold-trace by default.
Supporting a non-supported compiler involves two steps:
- Add an entry in the top-level configuration file
- Add corresponding translation file
3. Add an entry in the top-level configuration file
An entry must be added to top-level configuration file for this compiler name and corresponding translation file name (excluding extension).
Example:
Suppose your new compiler name is ‘cctc’ which takes following options
“-inc” – to include directory
“-def” – to defining macro
“-ifile” – to include a file during compilation
Where, tasking_tricore is a JSON translation file named tasking_tricore .json in the same directory as top-level configuration file.
- tasking_tricore json file will look like below:
Suppose the new compiler takes includeFile of the form @test/.includeDirs_c23dewff34.txt Hence, the regex provided is “.*.includesDirs_.*.txt”
Supported Compilers
Compiler | Compiler name | Translation config name |
---|---|---|
GCC | gcc, g++, cc, c++, clang | gnu.json |
Clang | clang, clang++ | gnu.json |
Green Hills (GH) | ease850 | gh.json |
Tasking Tri-core | cctc | tasking_tricore.json |
MSBuild | cctc | tasking_tricore.json |
Locate Compilation Database
After the build is finished successfully, the Compilation Database is generated in the current working directory where the build is run. A file named compile_command.json is the CDB. If the embold-trace successfully generates CDB, a log message will be printed displaying the number of compile entries in CDB.
Locate Configuration Files
By default, all the provided configuration files (top-level and translation) are present in the directory where the embold-trace binary is present. Any new translation file should also be added to the same directory.
However, you can keep all the configuration files in a separate directory. In this case, the top-level configuration file path should be given as input to embold-trace while invocation using option “–c”
$ embold-trace -c {top/level/config/file/path} {your build command} [build-options]
Configuring Unknown Compiler
Configuring embold-trace for an unknown compiler is an iterative process.
Steps to configure
- Run build using embold-trace (Check How to use section)
- After the build, embold-trace will print number of non-configured executable and log all the entries in a text file embold-trace-unknown-exe.txt. This file will be createdin current working directory. Not all the executable are compilers so user need to identify which ones are the compilers.
- If you found your compiler in non-configured list then go to next step else configuration is done no further action is needed
- Create a translation file for your compiler
- Add compiler name and translation file name entry to embold-trace-default.json
- Run embold-trace with trace file as input (No need to run build again)
- Go to step 2
Example: Configuring g++
Assumption: No compilers are configured.
Empty embold-trace-default.json
Our sample repository is cppcheck. We will build the repository using embold-trace and configure compiler g++ and create compilation database.
Step 1: Build repository using embold-trace
Step 2: Build output
As can be seen from above image, embold-trace intermediate trace file path is C:\Users\Hemant\AppData\Local\Temp\embold-trace-log_1590487077
. Non-configured executable is 4 and they ate written to C:\Workspace\repos\cppcheck\build_make\embold-trace-unknown-exe.txt
. Also, no commands have been intercepted as no compilers are configured.
Step 3: Inspect the contents of C:\Workspace\repos\cppcheck\build_make\embold-trace-unknown-exe.txt
There are 4 entries in unknown executable list. Not all executable are compilers only g++ is. So we will go to next step and configure g++.
Step 4: Create a translation file gnu.json (filename can be anything) for g++ compilers
For all GNU family compilers like gcc, g++
includeDirPrefix → -I
defineMacroPrefix → -D
includeFilePrefix → -include
So we will create gnu.json like this
Step 5: Add compiler name (g++) and translation file name (gnu) entry to embold-trace-default.json
Step 6: Rerun embold-trace with a new configuration and trace file as input
This time the compilation database is generated and non-configured executable count is reduced by 1 as we have configured g++ compiler. Also we didn’t run the build again. Contents of C:\Workspace\repos\cppcheck\build_make\embold-trace-unknown-exe.txt
There is no entry for g++. So g++ is configured correctly. Other executable are not required to be configured. So embold-trace configuration is complete and we got valid compilation database.
Step 7: No need to go to step 2
Likewise, we can configure any possible compiler iteratively
Non-configured Executable file format
Its a text file which provides hints to user regarding probable compilers in build. It contains unique executable path per line. Each line is further divided into 3 parts separated by semicolon. Meaning of each part is explained in below image.
As you can see, the first thing user need to check is whether the executable is compiler or not is by inspecting the second part, configure all* executable with “C“ first as they probably are compilers. And executable with most occurrences and “N“ might be custom compiler but they are can only be identified by build engineers.
*Some compiler executable internally also invokes another executable in that case we may ignore it like cc1plus. g++ internally invokes cc1plus.
Custom Source Extensions
By default, embold-trace supports following C/C++ extensions
“cc”,”c++”,”c”,”cpp”,”cxx”,”cplusplus”
However, user can add custom extensions like this
Here ‘850’ & ‘pc’
are custom C extensions and 'pcpp'
is custom C++ extension
Running analysis in strict mode
Set up your remote Embold instance. This is where your analysis results will be published. Follow the steps to set up remote analysis here.
- Follow the steps to set up remote scan
- If you wish to create a separate build folder for running trace-utility (intercept-build), you should use the –cdb additional option while running the scan.
The compile_commands.json will be created inside your build directory and not in the base directory of your source folder. Since, the scanner looks for the compile_commands.json in the base directory by default, use the –cdb option to specify the directory where your compile_commands.json is located. - In “settings” section, under “additionalOptions” set the directory where compilation database resides.
Example: –cdb= - If the scan is successful you should see “ANALYSIS SUCCESS”. On your build machine, the last two log messages indicate that the remote analysis was successful.
- On your remote Embold instance, you will be able to see published results.
C++ Fuzzy Mode
If any included file is not found in the source directory, cxxparser will throw a warning message which contains the location of missing include. Accuracy of the parser can be improved by providing the path to the directories which contain the missing files.
Improving accuracy in fuzzy mode
Steps to navigate to Scan Configuration Option:
- Click on Projects option on the left side navigation pane.
- Inside your project, click on a repository and then click on three dots “…” on the top right corner. Select the Scan configuration option.
Scan Configuration window consists of:
- Exclusion
- Additional Options
- Inclusion
- Includes
Exclusion:
We can exclude particular source files by providing regular expressions. Regular Expression Format: JavaScript
Examples:
- To filter out files containing “test” keyword -> “.*test.*”
- To filter out everything but one file -> “^(?!.*parse-this-file-only.cpp*$).*”
- Use escape character to match special characters like +, . -> “.*test.c++*”
Additional Options:
- Parsing invalid code:
By default, invalid code parsing is enabled but, it can be disabled using the following option.–parse-invalid-code=OFF
There is limited support for invalid code parsing, and it is prone to parser crash (very rare), so if for some codebase parser is crashing, please disable the invalid code parsing.
- Recursive include header search
By default, the parser searches for any header file recursively in all the sub-directories of the source folder. This can lead to incorrect results if there are multiple header files with the same name but in different folders.
To disable searching in all sub-directories use the following option
—include-all=OFF
- Define macros
Macros can be defined with GCC like option format.
Example: To define macro MY_MACRO -> –clang=”-DMY_MACRO=”
Any option to compiler can be given with –clang=””
Inclusion
- A comma-separated list of regular expressions should be added in Scan Configuration pop-up window >> inclusion
Includes
- Look for Includes label and upload zip of include files.
- Click Save.
Integration with CI/CD tools
Embold is integrated into the Software Development Life Cycle (SDLC), enabling the early detection of code issues, anti-patterns, etc. Embold scan can be automatically triggered after the source is built. Integration of Embold with CI/CD tools requires sign-in with the Embold Server deployed within the organization.
GitLab Integration
After integrating Embold with GitLab, users can scan their code after every GitLab build pass. Users can also configure Embold modules (tools) using gamma_scansettings.json at the GitLab level.
CI Integration pipeline
In this mode, Embold integrates with the CI system (Gitlab in this case) to run scans as part of the pipeline run. The following diagram shows the Embold setup with Gitlab:
In this case, the “Corona” component of Embold is deployed additionally on the Gitlab Runner host. The Corona component analyses code which was pulled on the Gitlab Runner during pipeline runs. If running C/C++ pipelines, it is recommended to run Corona in “strict” mode. This involves intercepting the build process to extract compiler command invocations, generating a compilation database, and using that database for the analysis. This approach results in maximum accuracy for C/C++ systems.
Prerequisites
- Remote Analysis should work from GitLab machine (where GitLab runner is running) to Embold machine. (Refer Remote Scan for more info).
Installations Steps
- Install standalone Corona for Ubuntu, Windows or RedHat Enterprise Linux/CentOS at your desired location (e.g. /opt/gamma ). It will be your GAMMA_ROOT folder.
- Recursively change the owner of the GAMMA_ROOT folder (e.g. /opt/gamma folder) to GitLab user (e.g. sudo chown -R gitlab-runner /opt/gamma)
- PostgreSQL on your remote Embold instance should be configured.
- Recursively give read/write/execute permission to the GAMMA_ROOT folder for GitLab users. (e.g. sudo chmod -R 744 /opt/gamma)
- Download and update scan configuration json from gamma UI
- Update the .gitlab-ci.yml file the source code. Refer below table.
- In case of running gcov, Gcovr should be installed on the build machine.
With the above settings, when the runner is launched by Gitlab while executing the pipeline, it will now also include Embold code scans.
Operating system | .gitlab-ci.yml changes |
---|---|
Windows | set GAMMA_ROOT= C:\workspace\gitlab\gitlab_corona ‘%GAMMA_ROOT%\corona\tools\bin\gitlab_wrapper.bat gammascan.json’ |
Linux | export GAMMA_ROOT=/opt/gamma $GAMMA_ROOT/corona/tools/bin/gitlab_wrapper.sh gammascan.json |
GAMMA_ROOT is where the standalone corona is installed on the runner (GAMMA_ROOT should have a folder “corona” directly in it)
Jenkins Integration
Embold Jenkins Plugin allows users to scan code after Jenkins build is triggered. Users can configure Embold modules from this plugin. The plugin requires sign-in with the Embold Server deployed within the organization.
Prerequisites
- Jenkins version 2.289+.
- Remote Scan should work from Jenkins machine (where build is happening) to Embold machine.
- Download Embold scanner from your Embold Customer Portal > Releases tab > CLI. There will be one file with names similar to the following: ‘browserstack-codequality-scanner.tar.gz’.
- For running gcov, Gcovr should be installed on the build machine.
- JDK (Java Development Kit)
- JAVA_HOME environment variable need to set for build machine (master or slave) which should point to
/bin
Installation Steps
Embold integration in Jenkins can be done using 2 approaches:
- Using Embold plugin (Check limitations) or
- Using Embold CLI / Remote scan
Jenkins Plugin
Limitation in Jenkins Embold plugin:
- It is not supported in pipeline jobs.
- Embold scanner packages will not update automatically, you need to replace the new Embold scanner folder for every new release.
Installation steps
- Untar embold-scanner-1.0.0-archive.tar.gz to some location on Jenkins master/slave.
- Add the environment variables
CORONA_HOME
,CORONA_LOG
,EMBOLD_SCANNER_HOME
.CORONA_HOME
: Specify the location where Embold packages will be downloaded (e.g /opt/gamma/corona).CORONA_LOG
: Specify the location where logs for Embold Analysis will be generated.
Note: The directory should have write permissions.
- Download Jenkins plugin from your Embold Account’s section > Releases tab > Plugins > CI_CD > jenkins. There will be file with a name similar to the following: Embold-Jenkins-Plugin-1.9.17.0.hpi.
- Click on “Manage Jenkins” on Jenkins’s home page.
- Jump to the Advanced tab.
- Go to the Upload Plugin section and upload “Embold-Jenkins-Plugin-1.9.17.0.hpi”.Click the “Upload” button.
- After Embold Jenkins Plugin is updated, Jenkins needs to be restarted.
Using CLI
Installation using Embold CLI / Remote scan
- Download the embold-scanner binaries from Embold control panel.
- Put the binaries in some directory where Jenkins user can access them (Make sure that the Jenkins user has
read-write-execute
permission). - Select Execute shell / Execute Windows batch command depending on OS where your job is running. (For pipeline jobs you can add following command in pipeline steps after build steps )
- Add a following command in ‘Command’ section
- For Windows, run the below command:
<Embold scanner path>/embold-scanner.bat -c <repository configuration path>/repository-configuration.json -u <Embold server URL> -t <Embold token> -sh <unique path where scanner packages will be downloaded> -r <Embold repository UID> -l <custom log directory>
- For Linux, run the below command:
<Embold scanner path>/embold-scanner -c <repository configuration path>/repository-configuration.json -u <Embold server URL> -t <Embold token> -sh <unique path where scanner packages will be downloaded> -r <Embold repository UID> -l <custom log directory>
For E.g./var/jenkins_home/embold-scanner-1.0.0/bin/embold-scanner -c ./repository-configuration.json -u http://192.168.2.38:3001/ -t eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MTYyODA2NTAxOTcyMSwiaWF0IjoxNjI4MDY1MDE5fQ.RehuKmSsWu7BaRvNgKGBMreOsy4d1GHXju53ujAr5QQLqqOOG48_sf7l4L_9aHcq40_hSWSiMR2coR4kXTJutPQ -sh ./embold -r 75b058cd126d1316af094632dfc47b01 -l ./logs
- For Windows, run the below command:
Make sure your Embold scanner package path (argument for -sh) is unique for each job
Integration Modes
- As Post Steps: Hooking into the build process so that Jenkins Job can be marked unstable if Embold’s quality gate check fails.
- As Post-build Actions:
- Hooking into Jenkins job workflow and gets triggered after the build is completed.
- In this mode, Embold Jenkins plugin will run irrespective, if the build has failed for any reason.
Configure Plugin Logs
- Go to the Jenkins> Manage Jenkins> System log. (Steps to add logging support)
- “Add new log recorder” for Jenkins Plugin.
- The Logger should be com.acellere.plugins and the log level should be mentioned as “All”.
- You should only view Embold Jenkins Plugin related logs.
Post steps configuration of Jenkins Plugin
- Go to the “Configure” page.
- Go to Post Steps and from Add post-build step dropdown list, select “
Embold Jenkins Plugin” option. - Embold Configuration Page is displayed to the user. Specify valid Embold URL (http://IP address of Embold Server:3000).
- Click on the Add button for adding Embold Credentials.
- In the Jenkins Credentials Provider pop-up, from Kind dropdown, choose “GAMMA Credentials”.
- Specify Name and Description which could be any value (it can be your Repo name or job name).
- Generate an Embold Access Token from the deployed Embold application website and add the Access Token under the Token section. Refer this article for generating Embold Access Token.
- After adding the Jenkins Credentials, select the added credential from the dropdown of Embold Credentials.
- Fill up the details in Subsystems section:
- Subsystem UID: Repository UID. Can be obtained from Repository information in corresponding project on Embold. Incorrect UID will throw an error.
- Snapshot Label (Optional): Provide the snapshot level value for Embold analysis. You can also provide environment variable. For example: BUILD_NUMBER
Configure Jenkins Plugin as Post-build Action
- Go to Post build Actions tab and add Embold Jenkins Plugin.
- You should see the Embold Jenkins configuration page.
- Embold Configuration Page is displayed to the user. Specify valid Embold URL (http://IP address of Embold Server:3000).
- Click on the Add button for adding Embold Credentials.
- In the Jenkins Credentials Provider pop-up, from Kind dropdown, choose “GAMMA Credentials”.
- Specify Name and Description which could be any value (it can be your Repo name or job name).
- Generate an Embold Access Token from the deployed Embold application website and add the Access Token under the Token section. Refer this article for generating Embold Access Token.
- After adding the Jenkins Credentials, select the added credential from the dropdown of Embold Credentials.
- Fill up the details in Subsystems section:
- Subsystem UID: Repository UID. Can be obtained from Repository information in corresponding project on Embold. Incorrect UID will throw an error.
- Snapshot Label (Optional): Provide the snapshot level value for Embold analysis. You can also provide environment variable. For example: BUILD_NUMBER
Configure Analysis Configuration file for Embold Jenkins Pugin for both modes
- Use Analysis configuration from Embold:
- Embold analysis configuration file with name ‘.gammascan.json’ containing subsystem configuration will be picked at runtime from. Embold server associated with the Subsystem ID specified.
- With this option, the configuration file will take the default path for Junit and Clover.
- Use Analysis configuration file from Source code base directory:
- Embold configuration file with name ‘.gammascan.json’ containing subsystem configuration should be present in the base(root) directory.
- Windows: C:/Program Files (x86)/Jenkins/workspace/{Project_Name}
/.gammascan.json - Linux: /var/lib/Jenkins/workspaces/{Project_Name}
/.gammascan.json
- Windows: C:/Program Files (x86)/Jenkins/workspace/{Project_Name}
- Use Analysis configuration from Jenkins:
- Login to Embold. Create a project. Link the repository you want to scan.
- On your remote Embold instance, navigate to the repository you want to scan. Click on “…”. Download configuration json from Embold UI.
- Update the following fields in json.
- Replace baseDir value with $WORKSPACE OR %WORKSPACE%
Note: WORKSPACE = Jenkins workspace - Give custom directory path for Data directory
Note: Make sure Jenkins user has Write permission for dataDir.
- Replace baseDir value with $WORKSPACE OR %WORKSPACE%
- Give the analysis configuration JSON in Jenkins UI. Click on the “Apply” button. Check if any errors are displayed in any configuration field. Click on the “Save” button.
For more information to understand repository-configuration.json, refer to this article.
Steps for Master Slave:
- Go to Jenkins -> Manage Jenkins -> Manage Nodes (only required for master – slave configuration)
- Open the particular Slave Node (only required for master – slave configuration).
- Go to Configuration.
- Add the new environment variables
CORONA_HOME
,CORONA_LOG
andEMBOLD_SCANNER_HOME
.
Steps for master only
- Go to Jenkins -> Manage Jenkins -> Configure System page and add the following 3 environment variables in “Global properties” section (In a case of master slave add these variables into slave configuration)
- Add the environment variables
CORONA_HOME
,CORONA_LOG
andEMBOLD_SCANNER_HOME
.CORONA_HOME
: Specify the location where Embold Analyzer will be downloaded (e.g /opt/gamma/corona).CORONA_LOG
: Specify the location where logs for Embold Analysis will be generated. Note: The directory should have write permissions.EMBOLD_SCANNER_HOME
: Specify the path till embold-scanner.
Embold Quality Gate
Below are the steps to show Embold details as an HTML report on Jenkins:
- Master Environment: Go to Jenkins > Manage Jenkins > Configure System and then add JENKINS_BASE environment variable in Node properties. Add path of Jenkins base directory of master’s machine.
- Master-Slave Environment: Go to Jenkins > Manage Jenkins > Manage Nodes > Click on the Configuration button of a particular node (slave) and then add JENKINS_BASE environment variable in Node properties. Add path of Jenkins base directory of master’s machine.
- Enable CSS and Javascript for Jenkins.
- Windows: add -Dhudson.model.DirectoryBrowserSupport.CSP=”” for Jenkins war in Jenkins.xml.
E.g.-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Dhudson.model.DirectoryBrowserSupport.CSP=”” -jar “%BASE%jenkins.war” –httpPort=8080 –webroot=”%BASE%war” - Linux: For Debian distributions, update the file- /etc/default/Jenkins.
- Centos/RedHat Distributions update the file /etc/sysconfig/Jenkins. E.g. JENKINS_JAVA_OPTIONS=”-Djava.awt.headless=true -Dhudson.model.DirectoryBrowserSupport.CSP=”””
- Windows: add -Dhudson.model.DirectoryBrowserSupport.CSP=”” for Jenkins war in Jenkins.xml.
- Restart the Jenkins. Install HTML Publisher Jenkins plugin.
- Add a “Publish HTML reports” plugin. Make sure the HTML Publisher plugin should be executed/added after the Embold plugin.
- Add the required details in the HTML publisher configuration.
- After build success, you can check the HTML report into HTML Publisher.
Update Embold Jenkins Plugin
- Go to Jenkins -> Manage Jenkins -> Manage Plugins. In the Advanced tab, upload the latest embold-jenkins-plugin.hpi file.
- Restart Jenkins using http://
/safeRestart.
Coverlet Integration
For more information about the execution of Coverlet integration, refer to this article.
UI output of coverage through coverlet can be seen in the Coverage by components section as below:
For Linux users, please check below points:
There is an issue with MSBuild on Linux that affects the ability to escape quotes while specifying multiple comma-separated values. Linux MSBuild automatically translates to /
in properties, tasks, etc. before using them, which means if you specified /p:CoverletOutputFormat="json,opencover
” in an MSBuild script, it will be converted to /p:CoverletOutputFormat=/"json,opencover/"
before execution. This yields an error similar to the following:MSBUILD : error MSB1006: Property is not valid. [/home/vsts/work/1/s/default.proj]
Switch: opencover/
You’ll see this if directly consuming Linux MSBuild or if using the Azure DevOps MSBuild task on a Linux agent.
The workaround is to use the .NET Core Dotnet MSBuild command instead of using MSBuild directly. The issue is not present in Dotnet MSBuild and the script will run with correctly escaped quotes.
Limitations
For more information, refer to this article.
Troubleshooting
- Coverlet module logs are generated inside
workspace/projectName/gamma_logs/{repositoryUID}/ext_coverlet.log
MSTest Integration
This section will help you to integrate MSTest (.Net unit test framework) tool on your local instance.
Prerequisites
- MSBuild version should be 15 or 16 depending on the type of source code that needs to be scanned.
- MSTest (does not require Visual Studio) can be separately downloaded as Test Agent Module.
Assumption
- MSTest/MSBuild should run in the client environment’s existing setup.
For more information about the execution of MSTest, refer to this article.
Limitations
- MSTest exe takes only a DLL executable; hence, the path of the executable is very specific to the arguments provided.
Troubleshooting
- In spite of having Visual Studio setup, if MSTest.exe is not present on your Jenkins machine, make sure you update the installation because you may have installed minimal components and not the full IDE. Also, sometimes the test components are explicitly loaded on drives, other than C drive.
- If Embold analysis succeeds only with an “INITIALIZING GAMMASCANNER” success message and git exceptions for fetching the repository, the JSON given for the scan config is invalid.
- MSTest module logs are generated inside
workspace/projectName/gamma_logs/{repositoryUID}/ext_mstest.log
Building C# projects in Jenkins:
While building the project from Jenkins, the following errors may occur:
MSBUILD : error MSB1009: Project file does not exist
MSBUILD : error MSB1008: Project file does not exist
Solutions
%WORKSPACE%
value has space in it ( C:/Program Files(x86/Jenkins/jobs/MSTest-Examples.Tests-Build/workspace)
), hence, it can break it into 2 arguments.
In this case, you need to move your Job’s workspace to a location that won’t have spaces in it. You can do in either of the following ways:
- Relocate your whole Jenkins’ installation out of “Program Files“,
- Or, Use custom workspace (click on the Advanced… button on the project configuration page, under Advanced Project Options)
%WORKSPACE%
in MSBuildConfig build step with Custom Workspace value set in the above step. SonarQube Integration
SonarQube integration with Embold allows users to see code issues reported by SonarQube on Embold.
Prerequisites
- Make sure Sonar analysis for a particular project is completed and code issues are visible on Sonar UI.
Steps for Integrating SonarQube with Embold
The following are the steps for running the SonarQube module with Remote Analysis:
- Download the JSON file from Embold. To know more about downloading configuration JSON from Embold UI, click here.
- Add the following config parameters for the SonarQube module in the code issue section.
- Specify the “Sonar User Token” for authentication.
- Specify the URL where SonarQube is running. This is a mandatory field.
- Specify the unique Sonar project key. This is a mandatory field.
- Specify one or more severities for which code issues will be fetched from SonarQube. If not specified, code issues for all the severities will be retrieved.
Example: SonarQube default severity values are: BLOCKER, CRITICAL, MAJOR, MINOR and INFO.
Configuration Notes
- Criticality filter value can contain custom severities or can be blank.
- Example: criticalityFilter=”blocker;severe;medium”
Note: Values should be separated using semicolon (;).
Mapping between SonarQube and Embold severity
Sonar | Embold |
---|---|
Blocker | Critical |
Critical | High |
Major | Medium |
Minor | Low |
Info | Info |
TeamCity Integration
TeamCity is a user-friendly Java-based build management and continuous integration server. Embold plugin integrates with TeamCity to:
- Automatic code scan upon triggered builds.
- Upload project’s code to Embold with scan results
- Embold tab to view scan summary and trends.
Prerequisites
- For seamless installation and execution, Embold should be installed on a clean Machine/VM.
- Install standalone corona on build machine (master/node).
- Requires an account to sign in to Embold from the browser. Please contact the administrator to create an account.
- Download file from your Embold Account’s section > Releases tab > Plugins > CI_CD > teamcity. There will be a file with a name similar to the following: embold-teamcity_1.8.3.0.zip.
- Embold uses the configuration file ‘.gammascan.json’ while analyzing the project. This file should be a part of the project to be analyzed.
- You can download configuration JSON from Embold UI.
Installation Steps
- On the top right corner of the TeamCity home page, click Administration.
- On the left navigation panel under Server Administration, click Plugin List.
- On the Plugin List page, click Upload plugin zip. Upload the downloaded plugin.
- On successful uploading, Embold plugin is listed in the External plugins table.
- Go to the Home page. On the left navigation pane, click Embold to validate the user.
- Embold can be validated using either API token, then click on the Test Connection button.
- On successful validation, the “Connection successful” message is displayed.
- Click on the project and build that needs to be analyzed through Embold.
- Click on the Edit Configuration Setting of the build.
- On the left navigation pane, click Parameters > Add new parameter.
- Add the environmental variables env.GAMMA_HOME, env.GAMMA_DATA and env.EMBOLD_SCANNER_HOME. Click Save.
-
env.GAMMA_HOME
: Specify the location where corona is installed (e.g /opt/gamma/corona) env.GAMMA_DATA
: Temporary directory location. (Used to store temporary data created while analyzing repository). Note: The directory should have write permissions.env.EMBOLD_SCANNER_HOME
: Specify the location for scanner home.
-
- The environmental variables are added successfully.
- Go to the left navigation pane, Click Build Step: Embold > Add build setup.
- Choose EMBOLD from the build runner drop-down list.
- There are 2 options through which scanning can be done:
- Using Configuration File: Configuration file can be obtained from Embold and need to be uploaded here.
- Using Subsystem ID: Obtain Subsystem ID from Embold.
This step is optional if the user wants to run the scan using Subsystem UID.
- You have successfully added a new Embold Built setup.
- Click on the Run button to build the project.
- After building the project, click on the Overview tab to view the result, Artifacts tab to view the generated PDF and Embold tab to view the analyzed project in a graphical format.
IDE Plugins
Embold IDE plugin allows developers to scan their code locally within the IDE environment and find issues in code as it is developed, before committing it to version control systems. Developers can install Embold IDE Plugin in their local eclipse/VS installation. The plugin requires sign-in with the Embold server deployed within the organization. It supports both windows and Linux OS.
Eclipse Plugin
Prerequisites
- Eclipse versions supported by Embold Eclipse Plugin are Mars, Neon, Oxygen.
- Requires a valid Embold username and a password. This is the account you usually use to sign in to Embold from the browser. If you don’t have one, please contact the administrator to create an account for you.
- Requires CPPCheck version 1.78 onwards to be installed for CPP analysis. Set CPPCheck in the environment variable. Verify if the environment variable is set or not by typing ‘cppcheck –version’ on the command line.
- Install Corona through Embold Installer.
- Make sure the Corona folder has right permission for the user running the Eclipse Plugin.
- Requires Eclipse CDT plugin
Installation Steps
- Download file from your Embold Account’s section > Releases tab > Plugins >IDE > eclipse. There will be file with a name similar to the following: site.zip.
- Extract zip file at any location
For e.g : /Users/admin/Embold/Site - Go to “Help” -> “Install new software”
- Click on “Add” button
- Click on local button.
- Browse parent dir of extracted zip.
- Click on Add.
- Check the “Embold Eclipse Feature”
- Accept ‘Terms and conditions’
- Finish and Restart Eclipse
- Enable the Embold Analysis tool in the menu bar (After restarting Eclipse): Go to Window -> Preferences > Embold settings.
- Configuration setup and authentication process:
- Set Embold Home path and fill Embold credentials for authentication.
Embold Home path should refer to the path where corona was installed.
e.g. D:/GammaHome/corona/ - Contact the administrator to get Embold the URL.
- Enter the Embold Authentication Token.
- Validate the Embold credentials and click on ‘Apply and close’ button.
- Set Embold Home path and fill Embold credentials for authentication.
- After authentication, Go to Embold Settings > Scan Configuration option. Select language and modules to run (e.g. CPPCheck for CPP).
- Click on Embold analysis icon from the menu bar to run analysis.
Analysis of output
After running an analysis successfully, view the results with the help of following views on Eclipse. These views can be opened as follows:
- Click on Window -> Show View -> Other.
- Open Embold -> Click on Embold related views
(e.g. Component Report View, Hotspot View, and Component Info View):
- Component Report View:
This view gives you all the code issues and design issues for the currently opened file (Java/CPP) in the editor.
- Hotspot View:
This view shows all the Hotspots in the project.
- Component Info View:
This view opens when you right click on any code issue or design issue and click on Show info option, it gives you information related to any code issue or design issue.
- Design Issue Info (God Class)
- Code Issue Info
- Metric Details
Hovering on ruler icon on vertical bar in editor gives the metric details of respective component
IntelliJ Plugin
In the Embold IntelliJ plugin, the detected code issues are highlighted on the fly and you can have quick-fix solutions as you edit the code. This plugin helps to enable or disable your code issues from Embold Global Settings by filtering the code rules according to the criticality. Moreover, anti-patterns can aid in providing error-prone solutions.
Code issues and anti-patterns’ count with its criticality can be visualized graphically. It can analyse the code written in the Java language.
For more information, refer to this link.
Prerequisites
- Java version: 1.8+
- Supported Intellij version:
- IntelliJ IDEA Community : 2019.3+
- IntelliJ IDEA Educational: 2019.3+
- IntelliJ IDEA Ultimate: 2019.3+
- IntelliJ IDEA 2021.2 (Community Edition)
- Android Studio version: 4.0+
- Android Studio build: 193.2252+
Installation steps
Install the Embold plugin from the IntelliJ IDEA marketplace.
Embold plugin is now available at the marketplace.
Scanning options
Embold supports 2 scanning options to scan the files in IntelliJ. They are as below:
- Scan single file
- Scan all files with Embold.
Scan a single file
Embold will automatically scan the file when a change is triggered in the file.
Scan all files with Embold
- Right click on editor.
- Select option “Scan all files with Embold”.
- Scanning all files with Embold may take a considerable amount of time to complete the analysis. Accept the pop-up to proceed. Click “Proceed” button.
- Embold will scan all files and publish the analysis result on “Embold Issues Inspector” and “Fly (Yellow Ballon)“ option.
Analysis of output
In the Embold IntelliJ plugin, the code issues are highlighted on the fly, and quick feedback is provided.
This ensures that the code issues are fixed even before the code is committed.
The code issues with its recommended solution can be seen under the “Embold Issues Inspector” section.
For example in the above screenshot, the code is analyzed at runtime, with the below output:
- On the left side, you see the type of issue with its description, criticality, and other details. Whereas, when you click on a specific issue, detailed information with examples can be seen on the right side.
Code issues and anti-patterns’ count can be visualized graphically.
The below output is seen in the graphical representation:
1. Issues by criticality– The issue count is calculated according to its criticality such as critical, high, medium, low, and info level issues.
2. Issues by type: Code issue and anti-pattern count are calculated.
- Clickable graphical representation is enhanced with in-depth granularity. Each section is segregated according to criticality and issue type (i.e. code issues and anti-patterns) displayed in a new tab known as the “Issues” tab.
- In the Embold IntelliJ plugin, the lines are highlighted on which code issues are present, and quick feedback is provided.
- On the fly, the “Disable Embold rule” option is available.
- This ensures that the code issues are fixed even before the code is committed.
In the above screenshot, the code issues are present on line number 2 (i.e. highlighted), and on the fly “Disable Embold rule ” option is available.
AutoDoc
Overview
- Embold AutoDoc generator is an AI-based comment generator tool that currently supports Java only for the IntelliJ plugin.
- The support for more languages is coming soon.
- Embold generates Java docs that help to produce a set of documentation comments for a method. This implementation delivers you faster and accurate results.
License Agreement for Embold service
The user needs to accept the terms of service by clicking on the “Accept” button.
- This will allow user to utilize Embold features by making an agreement.
- Once, license agreement is made, a pop-up window is displayed to the user. This AutoDoc premium feature is available for free use for a limited period.
How to generate the Embold AutoDoc?
- You need to install the latest Embold IntelliJ plugin from the marketplace.
AutoDoc can be generated using 3 ways:
A. Follow the below steps:
- Right-click on the method name
- Click on ‘generate…’ (Alt+Insert)
- Click on ‘Embold AutoDocs’
4. All set now! Embold will generate Java docs for you.
- Note: Embold will replace old Java docs with the new ones if already present.
B. You can also select the method name and use the shortcut key as ‘Alt + Shift + D’ to generate Java docs using Embold.
C. When user hovers on method name, a bulb icon is displayed. User can select “Generate comments using Embold AutoDoc”.
The comments get generated as shown in the below output:
Code checker configuration
This section helps to understand the supported code checks for Java language. Embold code checks can be enabled and disabled according to user requirement.
Critical Level Issues
Rule | Description | Example | KPI | Tags |
---|---|---|---|---|
Compile Regex Once | Pattern object compiles the regular expressions which are passed to them and this compilation happens in the memory. If a regular expression is used many times, then this compilation should be performed only once. | Non-compliant code:Class Foo { Compliant code: Class Foo { | Efficiency | |
Disabled Spring Securitys CSRF | CSRF protection is enabled by default in the Java configuration. Disabling CSRF protection can create a major security threat. Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they're currently authenticated. More Info - OWASP Top 10 A6:2017 - Security Misconfiguration CWE-352 - Cross-Site Request Forgery (CSRF) | Non-compliant code:@Configuration Compliant code: @Configuration | Security | |
Resource Leak | Resource should be closed in finally block. Another way is to use try-with-resource. In case of either exception or no exception, close() should always be put in finally clause. | Non-compliant code:class Demo { Compliant code: class Demo { | Resource Utilization | |
Weak Cipher Algorithm | Use strong cryptographic algorithms as they are less vulnerable to brute force attacks among others. For more information : : OWASP Top 10 2017 Category A3 - Sensitive Data Exposure MITRE, CWE-327 - Use of a Broken or Risky Cryptographic Algorithm | Non-compliant code: class CipherExample Compliant code:
| Security | |
Possible Thread Leak In Executor Service | Thread leak can be possible if ExecutorService is not getting shutdown.There is a queue of tasks between client threads and thread pool. When the normal thread finishes the "run" method (Runnable or Callable), it will be passed to garbage collector for collection. But with ExecuterService, threads will be simply put on hold and they will not be selected for garbage collection. Therefore, the shutdown is needed for ExecutorService. | Non-compliant code: class Demo { Compliant code: class Demo { | Resource Utilization | |
Non Private Field In Synchronized Block | Non-private, non-final field accessed in synchronized block indicates possibly partial synchronization, and does not protect direct access to the field from other parts of the system without synchronization. Make the field private and/or final, and provide accessors which can enforce synchronization. More Info - CWE-820 - Missing synchronization | Non-compliant code:class NonPrivateFieldAccessInSynchronizedBlock { Compliant code class NonPrivateFieldAccessInSynchronizedBlock { | Robustness, Security | |
Database Should Be Password Protected | Database connection should always be password protected. Attacker can access the sensitive data from the database, in case it is not secured by password. More Info - [OWASP Top 10 2017 Category A3] - Sensitive Data Exposure - [MITRE, CWE-521] - Weak Password Requirements | Non-compliant code:class CipherExample { Compliant code class CipherExample { | Security | |
Should Not Use getRequestedSessionId API | This method returns the session ID specified by the client and may not be the same as the ID of the current valid session for this request. If the client does not mention a session ID, it returns null. The session ID returned is either transmitted in a cookie or a URL parameter, hence by definition nothing prevents the end-user from manually updating the value of this session ID in the HTTP request. More Info - [CWE-807] - Reliance on Untrusted Inputs in a Security Decision - [OWASP] - Broken Authentication |
| Security | |
Web Application Contains Main Method | This method returns the session ID specified by the client and may not be the same as the ID of the current valid session for this request. If the client does not mention a session ID, it returns null. The session ID returned is either transmitted in a cookie or a URL parameter, hence by definition nothing prevents the end-user from manually updating the value of this session ID in the HTTP request. More Info - [CWE-489] - Leftover Debug Code - [OWASP] - Sensitive Data Exposure |
``` | Security | |
Authenticate LDAP Connection | Anonymous binds and unauthenticated binds allow access to information in the LDAP directory without providing a password. Therefore using these binds is strongly discouraged. More Info - [OWASP Top 10 2017 Category A2]- Broken Authentication - [CWE-521] - Weak Password Requirements - [ldapwiki.com] - Simple Authentication |
| Security | |
Potential Command Injection | Avoid using unfiltered input to process executor APIs because that can lead to arbitrary command execution. More Info - [OWASP] - Command Injection - [OWASP: Top 10 2013 A1] -Injection - [CWE-78] - Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection') |
| Security | |
Potential Path Traversal | Path traversal is a web security vulnerability that allows an attacker to read arbitrary files on the server that is running an application. This might include application code and data, credentials for back-end systems, and sensitive operating system files. More Info - [CWE-22] - Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal') - [OWASP] - Path Traversal |
| Security | |
Accessing Android external storage is security-sensitive | Files or data stored in the android application should be security-sensitive. Files can be globally readable or writable as they are stored on external storage. Avoid storing sensitive information on the external storage as anyone can tamper with the data or remove the files by using any application. **More Info** - [OWASP Top 10 2017 Category A1](https://owasp.org/www-project-top-ten/2017/A1_2017-Injection.html) - Injection - [OWASP Top 10 2017 Category A3](https://owasp.org/www-project-top-ten/2017/A3_2017-Sensitive_Data_Exposure) - Sensitive Data Exposure - [CWE-312](https://cwe.mitre.org/data/definitions/312.html) - Cleartext Storage of Sensitive Information - [CWE-20](https://cwe.mitre.org/data/definitions/20.html) - Improper Input Validation accessing android external storage |
| Security | |
Using unsafe Jackson deserialization configuration is security-sensitive | JThe untrusted data could cause abuse to application logic. Jackson's deserialization should be configured securely. Implement @JsonTypeName and @JsonSubTypes to prevent serialized data from an untrusted source. It can help to safeguard the data from external attackers. **More Info** - [OWASP Top 10 2017 Category A8](https://owasp.org/www-project-top-ten/2017/A8_2017-Insecure_Deserialization) - Insecure Deserialization - [CWE-502](https://cwe.mitre.org/data/definitions/502.html) - Deserialization of Untrusted Data |
| Security | |
Setting JavaBean properties is security-sensitive | The JavaBean property uses a set or get functions that are exposed to other applications. An attacker can modify its properties, attack malicious code that can be risky. Avoid storing sensitive information under this JavaBean property as it may help the user to retain its software integrity. **More Info** - [OWASP Top 10 2017 Category A8](https://owasp.org/www-project-top-ten/2017/A8_2017-Insecure_Deserialization) - Insecure Deserialization - [CWE-502](https://cwe.mitre.org/data/definitions/502.html) - Deserialization of Untrusted Data |
| Security | |
Run Should Not Be Called Directly | The program accidentally calls the Thread.run() method and causes the code to be executed in the current thread, just like any other method call. Instead, use Thread.start() to actually create a new thread so that the runnable's 'run()' method is executed in parallel. More Info [CWE-572] - Call to Thread run() instead of start() | Non-compliant code class Demo { Compliant code class Demo { | Functionality, Efficiency | |
Exclude SpringBootApplication And ComponentScan From The Default Package | Exclude "SpringBootApplication" and "@ComponentScan" from the default package. | Non-compliant code import org.springframework.boot.SpringApplication; Compliant code package javacodechecker; | Efficiency | spring |
High-Level Issues
Rule | Description | Example | KPI |
---|---|---|---|
Initialization Of Secure Random At Method Level | "SecureRandom" should not be initialized in method. Every SecureRandom generation is seeded from some entropy pool. Creating a new SecureRandom method on every call might slow down the application as it might block the creation of seed. Use a statically created instance instead. | Non-compliant code class Demo { Compliant code class SecureRandomGenerator { | Security |
Security Sensitive Regex | Regular expressions are security-sensitive. Evaluating regular expressions with input string can be an extremely CPU-intensive task. The regular expression naive algorithm builds a Nondeterministic Finite Automaton (NFA), the attacker might use the knowledge of states of finite state machine to look for applications that use regular expressions, containing an Evil Regex, and send a well-crafted input, that will hang the system. Alternatively, if a Regex itself is affected by a user input, the attacker can inject an Evil Regex, and make the system vulnerable. | Examples of Evil Patterns: Examples of Evil pattern : 1) (a+)+ 2) ([a-zA-Z]+)* 3) (a|aa)+ 4) (a|a?)+ 5) (.*a){2} for x > 10 E.g Pattern: /(a+)+b/; Input string:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab Instead use Pattern:(/a+b/) //fixed the hard-coded regex pattern to avoid possible captures, possessive quantifiers and back-references To avoid possible attacks, if the regex pattern is defined with an user-controlled input, it should be sanitized in order to escape characters which are part of the regular expression syntax. Non-compliant code class Demo { public boolean validate1(javax.servlet.http.HttpServletRequest request) { String regex = " /(a+)+b/"; String input = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab"; input.matches(regex); //Noncompliant Compliant code class Demo { public boolean | Efficiency, Security |
Use Of System.err.println | 1. System.err.println is an IO-operation and therefore is time consuming. 2. The better approach is to use a logging framework for a message queue. 3. Moreover, you can configure separate log files for different purposes. | Non-compliant code class Demo { Compliant code class Demo { | Analyzability |
Invalid Logging Class Name | Avoid incorrect class names while creating logger. Ignoring this may create confusion while analysing logs. | Non-compliant code public class LoggingClass extends BaseChecker { Compliant code public class LoggingClass extends BaseChecker { | Analyzability |
Use Of System.out.println | 1.Sending messages to stdout is usually inappropriate in a production environment. E.g. If you are coding a GUI app, the information should be presented to the user and not to the stdout method anywhere. 2.System.out.println is an IO-operation and therefore is time consuming. The better approach is to use a logging framework for a message queue. 3.Moreover, you can configure separate log files for different purposes. | Non-compliant code class Demo { Compliant code public void process2() { | Analyzability |
Complex Regex Pattern | Avoid usage of complex Regex pattern as it involves heavy processing.In some cases complex Regex performance test shows that regex is significantly inefficient. | //Non-compliant code class Demo { | Efficiency |
Empty Catch Block | Empty Catch Block finds instances where an exception is caught, but nothing is done. In most circumstances, this ignores an exception which should either be acted on or reported. Avoid keeping the catch block empty. | // Non-compliant code class Demo { // compliant code public void process2() { | Analyzability |
Empty Catch Block | Empty Catch Block finds instances where an exception is caught, but nothing is done. In most circumstances, this ignores an exception which should either be acted on or reported. Avoid keeping the catch block empty. | // Non-compliant code class Demo { // compliant code public void process2() { | Analyzability |
Sensitive Info Logged | Logger is used for recording application activity. Information displayed on the logs is visible to different stakeholders. Avoid logging sensitive information such as password, key, social security number etc. Not complying may compromise the system security. | Non-compliant code:class Demo { Compliant code class Demo { | Security |
Main Should Not Throw Anything | The main method should not throw any checked exception, instead, it needs to handle properly. If a non-checked exception is thrown (and not catch) in the main method, it will terminate. | Non-compliant code: Compliant code public static void main(String[] args) { | Functionality |
Avoid Synchronized At Method Level | Avoid synchronized at method level. When new code is added to the existing method then method-level synchronization can cause problems so to avoid these kind of problems use block-level synchronization. Block-level synchronization helps to ensure that only the code that needs synchronization gets it. | Non-compliant code:
Compliant code public class SynchronizedCounter { | Efficiency |
Shortcircuit Logic Should Be Used In Boolean Contexts | Using non-short-curcuit logic is a mistake it can also cause serious problem error. | Non-compliant code:
Compliant code
| Efficiency |
Medium Level Issues
Rule | Description | Example | KPI |
---|---|---|---|
Return Empty Array Or Collection Instead Of Null | Returning null value instead of an empty array or collection can lead to denial-of-service vulnerabilities when the client code fails to explicitly handle the null return value. For methods that return a set of values using an array or collection, returning an empty array or collection is an excellent alternative to returning a null value. More Info MET55-J - Return an empty array or collection instead of a null value for methods that return an array or collection MSC19-C - For functions that return an array, prefer returning an empty array over a null value | Non-compliant code: class Demo { Compliant code: class Demo { | Robustness |
Preserve Stack Trace In Logs | Preserve stack trace in logs. This will help to analyse the logs in case of any exception. | Non-compliant code:class Demo { Compliant code: class Demo { | Analyzability |
Read Only Transaction | Spring components support database transactions using "@Transactional" annotation. If readOnly attribute is not explicitly set to true, we will have read/write transactions for select queries. Hence, it is always recommended to explicitly specify the readOnly attribute. | Non-Compliant Code:class Demo { Compliant Code: class Demo { | Robustness |
Unusual REST Practice | The best practices while creating REST API's are : 1. URL should contain resources (E.g. nouns) only; not actions or verbs. 2. Singular and plural noun should not be mixed together. 3. Use plural noun only for all the resources. 4. Use GET method, instead of the POST method to fetch the data. 5. Use PUT, POST and DELETE methods to alter the state. | Non-compliant code class Demo { Compliant code class Demo { | Maintainability |
Variables Should Not Be Self Assigned | Self-assignment of the variables can be confusing and leads to bugs; however, one should not assign a variable to itself. Hence, this statement can be redundant and removed. | Non-compliant code class Demo { Compliant code class Demo { | Functionality |
Externalizable Must Have No Arguments Constructor | Externalizable interface cannot be deserialized without a non-argument constructor,so non-argument constructor must be implemented. | Non-compliant code public class Car implements Externalizable { Compliant code public class Car implements Externalizable { | Efficiency |
Getters And Setters Should Access The Expected Fields | Getter and Setter methods must access the expected fields. For each instance variable, a getter method returns its value, while a setter method sets or updates its value. For example, 'active' is an instance variable and 'setActive' (boolean value) is a setter method. Instead of unexpectedly updating any field, it must update or set the active variable. | Non-compliant code class Demo { Compliant code private boolean active; | Functionality |
RunFinalizersOnExit Should Not Be Called | Remove Runtime::runFinalizersOnExit and System::runFinalizersOnExit methods. It can be enabled with "System.runFinalizersOnExit" and "Runtime.runFinalizersOnExit".It may result in finalizers being called on live objects while other threads are concurrently manipulating those objects, resulting in unexpected behavior or deadlock. - [CWE-586]- Explicit Call to Finalize() | Non-compliant code public static void main(String [] args) { Compliant code public static void main(String [] args) { | Efficiency |
Big Integer Instantiation | Use already existing BigIntegers (BigInteger.ZERO, BigInteger.ONE, BigInteger.TEN).Instead of creating a new object with new BigInteger better use one static object which is created once when the BigInteger class is loaded. It is likely to yield significantly better space and time performance. Zero and One is probably the most fundamental number in mathematics. Using static objects avoids the allocation about 48 bytes and the need to collect them back later in a tight loop that can matter. | Non-compliant code BigInteger bigInteger = new BigInteger("1"); Compliant code BigInteger bigInteger = BigInteger.ONE; | Efficiency |
Maps With Enum Values Replace With EnumMap | If Map has all the key values from the same Enum then Map should be replaced with EnumMap because the underlying data structure is a simple array so it will be more efficient than other sets. | Non-compliant code public class MyClass { Compliant code public class MyClass { | Functionality |
Mismatch Regex Boundaries Should Not Be Used | In the regular expression by switching $ and ^ boundaries it will never match and it can be misused. | Noncompliant Code pattern.compile("$[a-z]+^"); Compliant Code pattern.compile("^[a-z]+$"); | Functionality |
Avoid Concatenating Char As String | Avoid concatenating characters as strings because using string rather than char creates unnecessarily space accommodation in heap space. Appending a character as a char will always be faster than appending it as a String. | Noncompliant Code class Demo { Compliant Code class Demo { | Functionality |
Empty String Should Not Be Used | Concatenating empty string with literals during conversion is inefficient. | Noncompliant Code String s = "" + 456; Compliant Code String t = Integer.toString(456); | Functionality |
Exceptions Should Not Be Thrown In Finally Block | Exception that is thrown in finally block will mask any previous exception in try or catch block and the stack trace and exception message will be lost. | Noncompliant Code try { Compliant Code try { | Functionality |
Low-Level Issues
Rule | Description | Example | KPI |
---|---|---|---|
Log Level Info In Catch Block | The catch block handles exception that occurs in associated try block. In this block, the logger level is expected to be as "error". This is a good list describing the log levels : 1. Debug : Used for development and testing. 2. Information : Used to output information that is useful to run and manage your system. 3. Warning : Used for handled 'exceptions' or other important log events. 4. Error : Used to log all unhandled exceptions. This is typically logged inside a catch block at the boundary of your application. 5. Fatal : Used for special exceptions/conditions where it is imperative that we can quickly pick out these events. | Non-compliant code: class Demo { Compliant code: class Demo { | Analyzability |
Non Thread Safe Field Declaration | A field declared inside the spring component should be thread safe. By default, beans in spring are singleton. Multiple threads accessing the same component may produce inconsistent results, if they access fields declared globally. | Non-compliant code: class Demo { Compliant code: class Demo { | Robustness |
Info Level Issues
Rule | Description | Example | KPI |
---|---|---|---|
String Concatenation In Logging | Instead of string concatenation use parameterized logging. Concatenation is calculated before the condition check. If you call your logging framework multiple times which turns to be false, this effort will be a waste of time. E.g. if you call your logging framework 10K times conditionally and all of them evaluates to be false, concatenation will happen 10K times. | Non-compliant code: class Demo { Compliant code: class Demo { | Efficiency |
Tightly Coupled Class | A spring component such as repository, service and controller should auto-wire the interface, instead of its implementation class. Ignoring this, will increase tight coupling among the classes | Non-compliant code:class Demo { Compliant code: class Demo { | Maintainability |
Visual Studio Plugin
Prerequisites
- Visual Studio versions supported by Embold Visual Studios Plugin: Visual Studios 2015 14.0.25431.01 Update 3, Visual Studios 2017 15.9.25, Visual Studios 2019 16.6.2.
- A valid Embold user registered with the Embold Instance deployed for the organization. Please contact the administrator for creating a valid Embold user account.
- Install Corona from Embold Installer.
- Contact administrator to download gamma_ide_plugins.tar.gz file.
- These files “embold_vs2015_1.2.vsix”, “embold_vs2017_1.2.vsix” and “embold_vs2019_1.2.vsix” are present in folder “visual studio” that are used for Embold-VSPlugin installation.
- Before installing Embold Visual Studio Plugin ensure that all Visual Studio instances are closed.
- Languages supported: C/C++, C#
- Frameworks supported for C# Language:
- .NET Framework-based projects supported for Embold plugin for Visual Studios 2015 and Visual Studios 2017.
- .NET Framework, .NET Standard and .NET Core based projects supported with Embold Plugin for Visual Studios 2019
Installation Steps
- Download the installer file from your Embold Account’s section > Releases tab > Plugins > IDE > visual_studio. There will be three file with names similar to the following: embold_vs2015_1.2.vsix, embold_vs2017_1.2.vsix, and embold_vs2019_1.2.vsix.
- Double click on the relevant vsix file version that is available for Visual Studio installed on the machine.
- Check the appropriate option (E.g. the Visual Studio 2017 option) from the given options and click “Install”.
- A successful installation message of the plugin will be displayed to the user. (E.g. Visual Studio 2017)
User Authentication for Visual Studio Plugin
- Analysis can be done in 2 ways:
- Open Visual Studio and select open Project/Solution option in File-> Open Menu option OR
- Navigate to the solution which needs to be analysed and open the .sln file in Visual Studio.
- fter opening a solution, if the Embold Visual Studio Plugin was installed successfully, the user should see the “Embold” Menu in the Visual Studio Menu bar.
- Inside the Settings Option, select the ”Gamma” option. There are two sub-options Analysis and User Registration.
A. Analysis
- Here a user can configure the mode of analysis.
Analyse on Save: Checking this option would enable the user to analyse the file on every “Save” operation.- Scan Type
- Full Scan: This is a fresh scan option where user can analyse the source code irrespective of any scan history.
- Incremental Scan: This option will enable incremental analysis of a single file or set of files with reference to the previous full scan analysis. Currently, only “Full” Analysis mode is supported.
- External Tools
- CPP Check: For a C++ project, the user can check the CPP Check option to enable CPP Check and see the code issues in the Component Issues View inside the Visual Studio instance.
- Clang Static Analysis: Not yet supported.
- Logging
- File Logging: Checking this option enables the user to get file logs as per the instance. The logs can be found at the following location: “C:UsersTestUserAppDataLocalAcellerePluginsVS”
- Name Obfuscation: Checking this option generates obfuscated logs at the above location.
- Scan Type
B. User Registration
- Corona Home Directory: Refer to the path where Corona is installed.
- Embold URL: The Embold URL should be given in the below format:
“http://emboldServer:3000”. Here “emboldServer” is the Embold Server deployed for the organization. - Embold Access Token: This is a personnel access token to access Embold APIs authentically.
- Verify:
1. It is mandatory to get these credentials verified by clicking on the “Verify” button. Users will get a “Verified” message on successful verification.
2. Ensure that a valid Embold User’s Embold server instance is up before entering the credentials; else user will get the “Embold server not reachable” status as shown below.
3. Once the user is authenticated to use the plugin, a message “Embold is ready” will be displayed to the user in the leftmost corner of the Visual Studio status bar. It will indicate that the user can now proceed with code analysis.
Code Analysis on Visual Studio Plugin
- Various options are provided to enable the user in getting a detailed analysis report. Embold analysis helps the user get all the Dependencies and Inter-dependencies between classes and methods. It is recommended that the user does a complete scan using the “Analyse Solution” option first and later the user can run scans on individual files that are modified. Below options are available for scanning the source code: Analyse Solution: The entire solution is scanned using this option. This is the currently recommended option.
- On selecting any analysis option, the ”Analysing” status message is displayed on the leftmost corner of the status bar and also a processing box appears as long as the analysis is in progress. The time taken for analysis is dependent on the scanning option selected and the size of code to be analysed.
- On successful analysis, an “Analysis completed” message will be displayed on the output tool window of Visual Studio.
- If the analysis fails due to any reason, the “Analysis failed” status will be displayed on the Embold output tool window.
- The user can also stop analysis in between by using the “Stop Analysis” option from the Embold Menu on the Menu bar. An “Analysis Aborted” status will be notified on this operation in the same Embold Output Tool Window.
Analysis of output
After running analysis successfully, results can be viewed through the following views:
- Hotspot View:
Click on Embold -> Hotspot View: This view shows all the Hotspots in the project.
- Component Issues Summary View:
Click on Embold -> Component Issues Summary View: This view gives all the code issues and design issues for the currently opened file in editor.
- Component Issues Summary View:
Right-click on a particular issue in Component Issue Summary View -> ShowInfo: This view gives detailed information related to the code or design issue.
- Metric adornments:
The four-blue coloured ellipses above Classes and Methods shows the Metrics Issues and the Design Issues related that particular Class or Method. Hovering over this adornment will display the analysis similar to below image.
Limitations
- Currently, Embold Visual Studio Plugin does not support the analysis of both C# and C++ code together in a single solution.
- Users cannot analyse the same code in multiple instances of Visual Studio’s.
VS Plugin Marketplace Installation
Prerequisites
- Supported Visual Studio version: VS2019
- Supported Visual Studio OS: Windows
- Supported Language: C#
Installation steps
Install the Embold plugin from the marketplace.
While it is getting installed, the pop-up window is displayed as below:
Embold plugin is now available in the marketplace.
Analysis of output
In the Embold Visual Studio plugin, the code issues are highlighted on the fly, and quick feedback is provided. This ensures that the code issues are fixed even before the code is committed.
Moreover, the code issues are suppressed and suppression information occurs where the violation occurs.
The code issues detected on the fly can be seen under the “Error List” section. Users can collapse/expand specific issues accordingly.
For example in the above screenshot (E.g. under Error List > Warnings“) section, the code is analyzed at runtime, with the below output:
- Under Error List, various types of errors with its error code, description, suppression state, and other details will be displayed. Whereas, when you click on a specific issue, detailed information can be seen in this documentation link.
Code Checker Configuration
Code Issue Name | Description | Example |
---|---|---|
MethodLinesOfCodeVoilationRule | Method having high number of lines of code(LOC)/statements are harder to understand and may violate single responsibility principle. Consider refactoring the method to modular methods. The default threshold for maximum allowed LOC/statements for a method is 50. When the LOC/statements for a method exceeds the threshold limit, the violation is notified. | Non-complaint code:/// |
CyclomaticComplexityForMethodVoilationRule | Cyclomatic Complexity (CC) is a measure of the program's complexity achieved by measuring the number of linearly independent paths through a program's source code. The number of linearly independent paths also means the minimum number of paths that should be tested. The more paths, the higher the number of test cases that need to be implemented. Consider splitting this method into smaller methods. The default threshold for the maximum allowed Cyclomatic Complexity for a method is 15. When the Cyclomatic Complexity for a method exceeds the threshold limit, the violation is notified. | Non-compliant code:static bool Compare(object val1, object val2, Type type) |
InvalidLoggingClassNameRule | Consider creating the logger name in context to the current class name. | Non-Compliant code:class ExampleClass Compliant code: class ExampleClass |
WeakEncryptionCheckRule | Weak encryption algorithms provide very little security and insufficient protection for sensitive data hence it is recommended to use a more secure encryption algorithm, such as AES. | Non-compliant code :using (var tripleDES = new TripleDESCryptoServiceProvider()) //Noncompliant Compliant code : AesManaged aes = new AesManaged |
VulnerableEncryptionModeCheckRule | Secure the encryption mode by combining Cipher Block Chaining with an authenticity check (HMAC-SHA256 for example) on the ciphertext. | Non-compliant code :AesManaged aes = new AesManaged |
CsrfTokenCheckRule | The action might be vulnerable to CSRF attack. It is advisable to check if it is secured with [ValidateAntiForgeryToken] attribute globally. | Non-compliant code:[HttpPost] Compliant code: [HttpPost] |
DisposableTypePublicallyExposedRule | Disposable types should not be publicly exposed as properties or fields. | Non-compliant code:public IDisposable Test { get { return null; } } |
DoNotChangeLoopVariablesRule | Do not change a loop variable inside the loop. Changing variable inside the loop may cause inconsistent behavior and a possible runtime exception. | Non-compliant code void TestForLoopVariableChange() |
AsyncMethodShouldNotHaveThreadSleepRule | An asynchronous method should not use a synchronous thread sleep mechanism. Thread.Sleep blocks the current thread; instead, use Task.Delay. Task.Delay pauses execution but will free the thread for other execution. | Non-compliant codeTask MyMethod() Compliant code: Task MyMethod1() |
CatchBlockShouldThrowExceptionRule | A catch block should throw exceptions instead of eating the exception. Swallowing an exception without re-throwing it's details, tampers the origin of exception and makes debugging difficult. | Non-compliant code:{ Compliant code: { |
EmptyCatchBlockRule | Code should not swallow exceptions using generic catch blocks like catch {} or catch(Exception){}. | Non-compliant code:Something() Compliant code: try |
FinallyBlockShouldNotThrowExceptionRule | Throwing an exception from a finally block may hide an exception thrown from the try or catch block. | Non-compliant codeSomething() |
IneffectiveExceptionHandlingRule | While handling exceptions, the 'Message' property does not contain any useful information which can assist in debugging the cause for the exception. Useful information can be extracted from the 'InnerException' property and should be used while handling and logging exceptions. | Non-compliant code:public static void Foo() |
UsingMoreSpecificExceptionTypeRule | Throw statements should not directly use types such as Exception, SystemException, and ApplicationException because they are too vague and don't have information about the failure reason. Use more specific types of exceptions or create your own that represents a given exceptional situation. Having a specific type for all types of failures makes logs filtering and aggregating much easier. Custom exception types allow also to convey in semantic form more contextual information inside the exception object. | Non-compliant code :try Compliant code : try |
MultiThreading_AcquireLockInsideTryRule | When manually managing scope of locking by directly calling methods responsible for acquiring and releasing the lock , ensure that the lock is acquired inside try clause. | Non-compliant code :class MyClass Compliant code : class MyClass |
MultiThreading_MethodLevelSynchronizationRule | Do not lock on the publicly accessible member as it can result in a deadlock.Type encapsulating access to a resource that requires synchronization should ensure that resource is not public. | Non-compliant code:
|
MultiThreading_NoLockOnPublicMembersRule | Do not lock on the publicly accessible member as it can result in a deadlock. Type encapsulating access to a resource that requires synchronization; it should be ensured that synchronization should be encapsulated. | Non-compliant code :public class SomeObject |
MultiThreading_ReleaseLockInsideFinallyRule | When manually managing the scope of locking by directly calling methods responsible for acquiring and releasing, ensure that the lock is released inside the finally clause to ensure that the lock is manually released in case of any exception. | Non-compliant code :class MyClass Compliant code : class MyClass |
DontCallVirtualMethodsInConstructorRule | When a virtual method is called, the actual type that executes the method is not selected until run time. When a constructor calls a virtual method, it's possible that the type that invokes the method has not been initialized yet. | Non-compliant code :DontCallVirtualMethodsInConstructorRule_Test Compliant code : class DontCallVirtualMethodsInConstructorRule_Test |
DateTimeNowUsageRule | Storing date/time values in UTC makes the data transportable, as the data is no longer tied to a particular time zone. To get the current date and time in UTC, it is recommended to use DateTime.UtcNow instead of DateTime.Now | Non-compliant code:void Main1(string[] args) Compliant code: void Main1(string[] args) |
DoNotReturnNullFromMethodRule | A method returning values representing strings or collections or task should never be null. The consumer of the APIs returning NULL would break if not handled. | Non-compliant code :public IEnumerable Compliant code : public IEnumerable |
UseArrayEmptyAPIRule | While initializing an array, avoid using 'new T[0]'. Instead use 'Array.Empty | Non-compliant code :
Compliant code : public void UseEmptyArrayForInit_Test() |
MakeMethodStaticRule | If a method is not referencing any instance variable and if you are not creating a virtual/abstract/new or partial method and if it is not a method override, your instance method may be changed to a static method. Therefore we recommend declaring the method as static. | Non-compliant code:public void Foo() Compliant code: public static void Foo() |
ClassImplementsICloneableCheckRule | ICloneable does not define which kind of cloning is done - deep or shallow copies. Therefore we recommend defining your Clone() or Copy() methods and document whether they perform deep or shallow copying instead of using ICloneable interface. | Non-compliant code public class Person : ICloneable Compliant code : public class IdInfo |
SwitchWithoutDefaultRule | The switch statement should include a default clause. | Non-compliant code :switch (value) Compliant code : switch (value) |
DoNotHideInheritedMembersRule | On hiding an inherited member using the "new" modifier, the derived version of the member replaces the base version. | Non-compliant code :BaseClass Compliant code variant 1: class BaseClass Compliant code variant 2: class BaseClass |
UsingContextWhileThrowingExceptionRule | While throwing an exception, add contextual information that can be used for constructing an appropriate error message. The rule ensures using context-aware constructors in throw statements. | Non-compliant code :try Compliant code : ``` try { // do something } catch (Exception ex) { // do something throw new Exception ("Put more context here", ex) } |
UsingInnerExceptionInNewExceptionThrownRule | Catch block could throw another exception that would be more meaningful for the clients. When throwing a new exception from the catch clause, always pass caught exception as an innerException parameter for the new exception. | Non-compliant code :public void Test() Compliant code : public void Test() |
DisposableFieldNotDisposedRule | The class having disposable field should call the dispose method of the field. | Non-compliant code :public class TypeA : IDisposable Compliant code : protected virtual void Dispose(bool disposing) |
DisposablesShouldCallSuppressFinalizeRule | Classes implementing IDisposable should call the GC.SuppressFinalize method in their finalize method to avoid any finalizer from being called. This rule should be followed even if the class does not have a finalizer as a derived class could have one. | Non-compliant code :public class MyType : System.IDisposable Compliant code : ``` public class MyType : System.IDisposable { void IDisposable.Dispose() { Dispose(true); System.GC.SuppressFinalize(this); } protected virtual void Dispose(bool disposing) { } } |
ClassMustBeSealedRule | A class should be marked as sealed class if there is no inheritor from the class. | Non-compliant code :class Calculations Compliant code : sealed class Calculations |
VSCode Plugin
Embold VSCode extension uncovers potential code issues, vulnerabilities, metrics, and hard-to-detect anti-patterns that make your code difficult to maintain and can lead to error-prone solutions. The extension currently supports C/ C++, Java, JavaScript, and Typescript.
Prerequisites
- Supported Vscode version: 1.62.0+
- Cppcheck version : 2.4.1
- Eslint version: v7.32.0
- Tslint version: 5.9.1
- Jshint(Optional -Only if you are enabling in Embold server): version : 2.9.5
- Java version: 1.8 or higher
- Supported Languages: C/CPP, Java, JavaScript, Typescript, C#
- Supported OS: Linux, Windows and Mac OS
Note: For Mac OS following languages are not supported- JavaScript
- TypeScript
- C#
- C/CPP
Installation steps
Install the Embold plugin from the VSCode marketplace.
- Go to the ‘view’ menu.
- Select the ‘Extensions‘ sub-menu.
- SSearch ‘Embold’ in the search box.
- Click “Install” button on the left navigation pane.
- Extension successfully installed.
Install Embold Server Certificate
This is most likely because of the firewall setting on your network, which rejects HTTPS requests if the Embold server certificate is not installed in the Java trust store of your machine.
Please follow these steps to install the same:
- In Edge, go to
https://packages.embold.io, click on the certificate icon that appears in the address bar, click the “Connection is Secure” line item, which then looks like this:
2. Click on the certificate is valid, go to Details Tab and click on Export button:
3. In the Export window, select file name as “embold.io” and Save as Type: DER-Encoded binary, single certificate:
4. Once the file is saved, go to command line and run the following command:
- “c:\Program Files\Java\jre-1.8\bin\keytool.exe” -importcert -trustcacerts -alias embold.io -file c:\Users\embold\Downloads\embold.io.der -keystore “C:\Program Files\Java\jre-1.8\lib\security\cacerts”
- Please adjust the paths above as per your installation (e.g. JRE Path and the certificate file path where you saved it)
- It will prompt for keystore password, and the default keystore password for Java is “changeit”
- It will prompt you to verify the certificate, and type “yes” at the end.
- This will then install the certificate on your computer, which the firewall should then allow.
Install Cppcheck
You have to install cppcheck (Version 2..4.1) manually on your machine.
Installation on Windows
- Go to ‘
https://sourceforge.net/projects/cppcheck/files/cppcheck/2.4/cppcheck-2.4.1-x64-Setup.msi/download
‘. - Open downloaded
.msi
file. - Install Cppcheck.
- Check version on cmd:
cppcheck --version.
Installation on Linux
Execute commands on the terminal:
git clone --branch 2.4.x https://github.com/danmar/cppcheck.git /cppcheck
mkdir /cppcheck/build
cd /cppcheck/build
cmake ..
cmake --build . -- -j16
make install
cd /
rm -rf /cppcheck
cppcheck --version
Install Eslint
Execute command from cmd/terminal:
npm install -g eslint@v7.32.0 –save-dev
# or
yarn add eslint@v7.32.0 –dev
Install Tslint
Execute command from cmd/terminal:
npm install -g tslint@5.9.1 –save-dev
# or
yarn add tslint@5.9.1 –dev
Install Jshint
Note: Jshint (Optional – Only if you are enabling in Embold server)
Execute command from cmd/terminal –
npm install -g jshint@2.9.5 –save-dev
# or
yarn add jshint@2.9.5 –dev
Proxy Settings
We have added a proxy setting in the VSCode extension.
Add the following environment variables if you are using any custom proxy settings.
1.EMB_PROXY_HOST
E.g. 192.168.1.127
2.EMB_PROXY_PORT
E.g. 3128
3.EMB_PROXY_USERNAME
(Optional)
4.EMB_PROXY_PASSWORD
(Optional)
How does it work?
Right-click on any .c/.cpp/.h/.hpp
file or folder containing .c/.cpp/.h/.hpp
any of these files. Select ‘Analyze Using Embold‘.’.
Analysis of output
After scan completion, analysis results will be displayed on the “PROBLEMS” view.
Show Description option is available on a quick fix. For metric violation – a quick fix will redirect to the documentation page.
For Design Issue – It will show a new tab for insights with an animated progress bar.
For Code Issue – It will redirect to the Rules page documentation.
Modules
BrowserStack Code Quality supports below modules:
Module Name | Module Type | Language | Availability | Pre-requisites |
---|---|---|---|---|
PMD | codeissues | Java | On-premise / Cloud | - |
Spotbugs | codeissues | Java | On-premise / Cloud | User should generate Spotbugs report (xmls) before analysis |
Findsecbugs | codeissues | Java | On-premise / Cloud | User should generate Findsecbugs report ( xmls) before analysis |
Sonar | codeissues | Java | On-premise | Sonar should be setup and configured |
Clover | coverage | Java | On-premise | User should generate clover report (xmls) before analysis |
Junit | unittest | Java | On-premise | User should generate junit report (xmls) before analysis |
cppcheck | codeissues | Cpp | On-premise / Cloud | - |
gamma_cxx | codeissues | Cpp | On-premise / Cloud | - |
Gcov | coverage | Cpp | On-premise | User should generate gcov report (xmls) before analysis |
Gtest | unittest | Cpp | On-premise | User should generate gtest report ( xmls) before analysis |
Gamma_cs | codeissues | CSharp | On-premise / Cloud | - |
Jshint | codeissues | Javascript | On-premise / Cloud | - |
Eslint | codeissues | Javascript | On-premise / Cloud | - |
Tslint | codeissues | Typescript | On-premise / Cloud | - |
Pylint | codeissues | Python | On-premise / Cloud | - |
Gometalinter | codeissues | Go | On-premise / Cloud | - |
Phpcs | codeissues | PHP | On-premise / Cloud | - |
Phpmd | codeissues | PHP | On-premise / Cloud | - |
Sqlcheck | codeissues | SQL | On-premise / Cloud | - |
Gamma_Java | codeissues | Java | On-premise / Cloud | - |