Recommendation Engine with docker-compose

August 17, 2021

Uncategorized

No Comments

Overview:

In this section, we will see how to setup Recommendation Engine and Embold on same Linux host.
We will also see how an existing embold server can be integrated with RE .

Installation steps

Prerequisites:
- docker engine
- docker-compose, version 1.25.0 or up
- In order to install Embold RE with docker-compose, you need 3 images:
- bitnami/mongodb:4.2.21 (Available on docker hub as public image)
- bitnami/elasticsearch:7.17.5 (Available on docker hub as public image)
- embold/gammare-snapshot:<current ver> (Available on Embold private docker repo,or as a tar.gz file in your Embold account)
- If the host where you will install RE has internet access, you need to download and load only embold/gammare-snapshot: prior to running docker-compose

Image procurement and setup process
- Download Embold RE image (tar.gz) from your Embold account (Releases section : https://codequality.browserstack.com/self-managed/v1/portal/)
  Or, You can run below command to download
  wget https://v1.embold.io/nfs/embold_1.9.28.1/RE/embold_re_0.9.4-SNAPSHOT.tar.gz
- On the docker host machine that will run RE, run:
- docker load -i embold_re_0.9.4-SNAPSHOT.tar.gz
- If you need to download the mongodb and elasticsearch images as well, run the steps below: On some machine with internet access:
- docker pull bitnami/mongodb:4.2.21
- docker save -o mongodb.tar.gz bitnami/mongodb:4.2.21
- docker pull bitnami/elasticsearch:7.17.5
- docker save -o elasticsearch.tar.gz bitnami/elasticsearch:7.17.5
- On the docker host machine that will run RE, run:
- docker load -i mongodb.tar.gz
- docker load -i elasticsearch.tar.gz
- Prepare the docker-compose script, directories and environment variables
- Create an environment file (e.g. re_env) in a folder with these 2 variables in it:
- # Edit to your actual host paths!!
- GAMMA_RE_DATA=/home/${USER}/BrowserStackCodeQuality/gamma_re_data
- GAMMA_RE_LOGS=/home/${USER}/BrowserStackCodeQuality/log/path/gamma_re_logs
- IMPORTANT!!
  Create the directories /home/${USER}/BrowserStackCodeQuality/gamma_re_data/mongo_data/data and
- /home/${USER}/BrowserStackCodeQuality/gamma_re_data/elastic_data on the host
- Change recursive ownership of to the user/group 1001:1001 like below:
  chown -R 1001:1001 /home/${USER}/BrowserStackCodeQuality/gamma_re_data
- chown -R 1001:1001 /home/${USER}/BrowserStackCodeQuality/gamma_re_logs

Creation of docker-compose yaml

Now with all components in place, you can run docker-compose with this sample script i.e docker-compose.yaml file:

version: '2'

services:
  db:
    image: "postgres:16.3-alpine"
    container_name: BrowserStackCodeQuality-DB
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - /home/${USER}/BrowserStackCodeQuality/gamma_pg_data:/var/lib/postgresql/data

  indexer:
    image: bitnami/elasticsearch:7.17.5
    volumes:
      - ${GAMMA_RE_DATA}/elastic_data:/bitnami/elasticsearch/data

  gammare:
    image: embold/gammare:0.9.4
    container_name: gammare
    environment:
      - ES_DB_URL=mongodb://db_mongo:27017/
      - ES_INDEXER_URL=http://indexer:9200
    ports:
      - "9090:9090"
    volumes:
      - ${GAMMA_RE_DATA}:/opt/gammare_data
      - ${GAMMA_RE_LOGS}:/opt/gammare_logs

  db_mongo:
    image: bitnami/mongodb:4.2.21
    volumes:
      - ${GAMMA_RE_DATA}/mongo_data:/bitnami/mongodb
    ports:
      - "27017:27017"
    healthcheck:
      test: ["CMD", "mongo", "--eval", "db.runCommand({ connectionStatus: 1 })"]
      interval: 30s
      retries: 5
      timeout: 10s

  app:
    image: "embold/gamma:1.9.28.1"
    tty: true
    container_name: BrowserStackCodeQuality
    environment:
      - ACCEPT_EULA=Y
      - gamma_ui_public_host=http://localhost:3000
      - RISK_XMX=-Xmx1024m
      - ANALYSER_XMX=-Xmx6072m
      - PGHOST=3.72.41.7
      - PGPORT=5432
      - PGUSER=postgres
      - PGPASSWORD=postgres
      - GAMMA_DATABASE=gamma
      - ANALYTICS_DATABASE=corona
      - SCM_THREADS=1
      - EMB_SCM_TRUST_SERVER_CERT=true
      - EMB_USE_NATIVE_PYPARSER=TRUE
      - re_host=http://gammare:9090
      - enableREScan=true
      - polling_pr_cron_time="*/2 * * * *"
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 10G
    ports:
      - "3000:3000"
    volumes:
      - /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data
      - /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs
      - /home/${USER}/BrowserStackCodeQuality/gamma_data/tmp:/tmp

Save above file as “docker-compose.yaml” in the same directory where you have created re_env file
Start the embold services with the following command from directory where re_env file is created:
docker-compose --env-file=./re_env up -d
If all goes well, the last few messages in $GAMMA_RE_LOGS/cmodsrv.log (GAMMA_RE_LOGS=/home/${USER}/BrowserStackCodeQuality/gamma_re_logs/cmodsrv.log) should look like:
- cmoddb.py:27 INFO| Initialized CmodDb
- cmoddb.py:27 INFO| Initialized CmodDb
- cmoddb.py:266 INFO| Cleared 0 requests from build queue

To Activate BrowserStack Code Quality goto activation.

Integrate existing standalone embold docker with RE

NOTE : Before you jump to this section , complete all the above steps except execution of docker-compose file.

Assume below is the docker command you have used to run standalone embold docker.

docker run -m 10GB -d -p 3000:3000 --name EMBOLD -e gamma_ui_public_host=http://<IP/domain name>:<Port> -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx6072m -v /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data -v /home/${USER}/BrowserStackCodeQuality/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs embold/gamma:$EMBOLD_VERSION

So In above docker-compose.yaml, you have to edit only gamma section,

In environment, add -e RISK_XMX=-Xmx1024m -e ACCEPT_EULA=Y -e ANALYSER_XMX=-Xmx6072m from the standalone docker command

In volumes, replace values and use -v /home/${USER}/embold/gamma_data:/opt/gamma_data -v /home/${USER}/BrowserStackCodeQuality/gamma_psql_data:/var/lib/postgresql -v /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs from the standalone docker command

So finally your docker-compose.yaml will be as below

version: '2'

services:
  db:
    image: "postgres:16.3-alpine"
    container_name: BrowserStackCodeQuality-DB
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - /home/${USER}/BrowserStackCodeQuality/gamma_pg_data:/var/lib/postgresql/data

  indexer:
    image: bitnami/elasticsearch:7.17.5
    volumes:
      - ${GAMMA_RE_DATA}/elastic_data:/bitnami/elasticsearch/data

  gammare:
    image: embold/gammare:0.9.4
    container_name: gammare
    environment:
      - ES_DB_URL=mongodb://db_mongo:27017/
      - ES_INDEXER_URL=http://indexer:9200
    ports:
      - "9090:9090"
    volumes:
      - ${GAMMA_RE_DATA}:/opt/gammare_data
      - ${GAMMA_RE_LOGS}:/opt/gammare_logs

  db_mongo:
    image: bitnami/mongodb:4.2.21
    volumes:
      - ${GAMMA_RE_DATA}/mongo_data:/bitnami/mongodb
    ports:
      - "27017:27017"
    healthcheck:
      test: ["CMD", "mongo", "--eval", "db.runCommand({ connectionStatus: 1 })"]
      interval: 30s
      retries: 5
      timeout: 10s

  app:
    image: "embold/gamma:1.9.28.1"
    tty: true
    container_name: BrowserStackCodeQuality
    environment:
      - ACCEPT_EULA=Y
      - gamma_ui_public_host=http://localhost:3000
      - RISK_XMX=-Xmx1024m
      - ANALYSER_XMX=-Xmx6072m
      - PGHOST=3.72.41.7
      - PGPORT=5432
      - PGUSER=postgres
      - PGPASSWORD=postgres
      - GAMMA_DATABASE=gamma
      - ANALYTICS_DATABASE=corona
      - SCM_THREADS=1
      - EMB_SCM_TRUST_SERVER_CERT=true
      - EMB_USE_NATIVE_PYPARSER=TRUE
      - re_host=http://gammare:9090
      - enableREScan=true
      - polling_pr_cron_time="*/2 * * * *"
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 10G
    ports:
      - "3000:3000"
    volumes:
      - /home/${USER}/BrowserStackCodeQuality/gamma_data:/opt/gamma_data
      - /home/${USER}/BrowserStackCodeQuality/logs:/opt/gamma/logs
      - /home/${USER}/BrowserStackCodeQuality/gamma_data/tmp:/tmp

Save above file as “docker-compose.yaml” in the same directory where you have created re_env file

Start the embold services with the following command from directory where re_env file is created:

docker-compose --env-file=./re_env up -d

If all goes well, the last few messages in $GAMMA_RE_LOGS/cmodsrv.log (GAMMA_RE_LOGS=/home/${USER}/BrowserStackCodeQuality/log/path/gamma_re_logs/cmodsrv.log) should look like:

cmoddb.py:27 INFO| Initialized CmodDb

cmoddb.py:266 INFO| Cleared 0 requests from build queue

For configuration of a repository with a Recommendation Engine for the beta version, check the link here.

recommendation engine Recommendation Engine docker deployment