Recommendation Engine with docker-compose

August 17, 2021

Uncategorized

No Comments

Overview:

In this section, we will see how to setup Recommendation Engine and Embold on same Linux host.
We will also see how an existing embold server can be integrated with RE .

Installation steps

Prerequisites:
- docker engine
- docker-compose, version 1.25.0 or up
- In order to install Embold RE with docker-compose, you need 3 images:
- bitnami/mongodb:4.2.21 (Available on docker hub as public image)
- bitnami/elasticsearch:7.17.5 (Available on docker hub as public image)
- embold/gammare-snapshot:<current ver> (Available on Embold private docker repo,or as a tar.gz file in your Embold account)
- If the host where you will install RE has internet access, you need to download and load only embold/gammare-snapshot: prior to running docker-compose

Image procurement and setup process
- Download Embold RE image (tar.gz) from your Embold account (Releases section : https://codequality.browserstack.com/self-managed/v1/portal/)
  Or, You can run below command to download
  wget https://v1.embold.io/nfs/embold_1.9.16.1/RE/embold_re_0.9.2-SNAPSHOT.tar.gz
- On the docker host machine that will run RE, run:
- docker load -i embold_re_0.9.2-SNAPSHOT.tar.gz
- If you need to download the mongodb and elasticsearch images as well, run the steps below: On some machine with internet access:
- docker pull bitnami/mongodb:4.2.21
- docker save -o mongodb.tar.gz bitnami/mongodb:4.2.21
- docker pull bitnami/elasticsearch:7.17.5
- docker save -o elasticsearch.tar.gz bitnami/elasticsearch:7.17.5
- On the docker host machine that will run RE, run:
- docker load -i mongodb.tar.gz
- docker load -i elasticsearch.tar.gz
- Prepare the docker-compose script, directories and environment variables
- Create an environment file (e.g. re_env) in a folder with these 2 variables in it:
- # Edit to your actual host paths!!
- GAMMA_RE_DATA=/some/path/on/host/gamma_re_data
- GAMMA_RE_LOGS=/some/log/path/gamma_re_logs
- IMPORTANT!!
  Create the directories /some/path/on/host/gamma_re_data/mongo_data/data and
- /some/path/on/host/gamma_re_data/elastic_data on the host
- Change recursive ownership of to the user/group 1001:1001 like below:
  chown -R 1001:1001 /some/path/on/host/gamma_re_data
- chown -R 1001:1001 /some/log/path/gamma_re_logs

Creation of docker-compose yaml

Now with all components in place, you can run docker-compose with this sample script i.e docker-compose.yaml file:

version: "3"

services:
       
      db:
       image: bitnami/mongodb:4.2.21
# Internal port: 27017
       volumes:
         - ${GAMMA_RE_DATA}/mongo_data:/bitnami/mongodb

      indexer:
       image: bitnami/elasticsearch:7.17.5
# Internal port: 9200
       volumes:
         - ${GAMMA_RE_DATA}/elastic_data:/bitnami/elasticsearch/data

      gammare:
       image: embold/gammare-snapshot:0.9.2-SNAPSHOT
       container_name: gammare
       environment:
         - ES_DB_URL=mongodb://db:27017/
         - ES_INDEXER_URL=http://indexer:9200
       ports:
         - "9090:9090" # Only port exposed to outside world
       volumes:
         - ${GAMMA_RE_DATA}:/opt/gammare_data
         - ${GAMMA_RE_LOGS}:/opt/gammare_logs

      gamma:
       image: embold/gamma:1.9.16.1
       container_name: gamma
       environment:
         - SCM_THREADS=1
         - EMB_SCM_TRUST_SERVER_CERT=true
         - EMB_USE_NATIVE_PYPARSER=TRUE
         - re_host=http://gammare:9090
         - enableREScan=true
         - polling_pr_cron_time="*/2 * * * *"
         - gamma_ui_public_host=http://localhost:3000
         - RISK_XMX=-Xmx1024m
         - ACCEPT_EULA=Y
         - ANALYSER_XMX=-Xmx3072m        
       ports:
         - "3000:3000"         
       volumes:
         - /home/gamma:/opt/gamma_data
         - /home/pg:/var/lib/postgresql
         - /home/logs:/opt/gamma/logs

Save above file as “docker-compose.yaml” in the same directory where you have created re_env file
Start the embold services with the following command from directory where re_env file is created:
docker-compose --env-file=./re_env up -d
If all goes well, the last few messages in $GAMMA_RE_LOGS/cmodsrv.log (GAMMA_RE_LOGS=/some/log/path/gamma_re_logs) should look like:
- cmoddb.py:27 INFO| Initialized CmodDb
- cmoddb.py:27 INFO| Initialized CmodDb
- cmoddb.py:266 INFO| Cleared 0 requests from build queue

To Activate BrowserStack Code Quality goto activation.

Integrate existing standalone embold docker with RE

NOTE : Before you jump to this section , complete all the above steps except execution of docker-compose file.

Assume below is the docker command you have used to run standalone embold docker.

docker run -m 10GB -d -p 3000:3000 --name EMBOLD -e gamma_ui_public_host=http://<IP/domain name>:<Port> -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx6072m -v /home/${USER}/embold/gamma_data:/opt/gamma_data -v /home/${USER}/embold/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/embold/logs:/opt/gamma/logs embold/gamma:$EMBOLD_VERSION

So In above docker-compose.yaml, you have to edit only gamma section,

In environment, add -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx6072m from the standalone docker command

In volumes, replace values and use -v /home/${USER}/embold/gamma_data:/opt/gamma_data -v /home/${USER}/embold/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/embold/logs:/opt/gamma/logs from the standalone docker command

So finally your docker-compose.yaml will be as below

version: "3"

services:
       
      db:
       image: bitnami/mongodb:4.2.21
# Internal port: 27017
       volumes:
         - ${GAMMA_RE_DATA}/mongo_data:/bitnami/mongodb

      indexer:
       image: bitnami/elasticsearch:7.17.5
# Internal port: 9200
       volumes:
         - ${GAMMA_RE_DATA}/elastic_data:/bitnami/elasticsearch/data

      gammare:
       image: embold/gammare-snapshot:0.9.2-SNAPSHOT
       container_name: gammare
       environment:
         - ES_DB_URL=mongodb://db:27017/
         - ES_INDEXER_URL=http://indexer:9200
       ports:
         - "9090:9090" # Only port exposed to outside world
       volumes:
         - ${GAMMA_RE_DATA}:/opt/gammare_data
         - ${GAMMA_RE_LOGS}:/opt/gammare_logs

      gamma:
       image: embold/gamma:1.9.16.1
       container_name: gamma
       environment:
         - re_host=http://gammare:9090
         - enableREScan=true
         - gamma_ui_public_host=http://<IP/domain name>:<Port>
         - RISK_XMX=-Xmx1024m
         - ACCEPT_EULA=Y
         - ANALYSER_XMX=-Xmx3072m        
       ports:
         - "3000:3000"         
       volumes:
         - /home/${USER}/embold/gamma_data:/opt/gamma_data
         - /home/${USER}/embold/gamma_psql_data:/var/lib/postgresql
         - /home/${USER}/embold/logs:/opt/gamma/logs

Save above file as “docker-compose.yaml” in the same directory where you have created re_env file

Start the embold services with the following command from directory where re_env file is created:

docker-compose --env-file=./re_env up -d

If all goes well, the last few messages in $GAMMA_RE_LOGS/cmodsrv.log (GAMMA_RE_LOGS=/some/log/path/gamma_re_logs) should look like:

cmoddb.py:27 INFO| Initialized CmodDb

cmoddb.py:266 INFO| Cleared 0 requests from build queue

For configuration of a repository with a Recommendation Engine for the beta version, check the link here.

recommendation engine Recommendation Engine docker deployment