Upgrading

Upgrading Owl from 1.0.0 to 1.1.x

Upgrading owl is as easy as replacing a few files. Once the files are replaced owl leverages capabilities native to itself in order to determine if there has been any schema changes. Follow these steps below to perform an upgrade from 1.0.0 to 1.1.x.

Step 1: Stop services

Use the owlmanage.sh script to stop all the services. This script is located in the owl/bin directory and stop all processes using the command below

./owlmanage.sh stop

Check to make sure all the process are stopped using the “jps” command.

jps

command, output as follows.

19169 OServerMain

18711 HMaster

18823 HRegionServer

1600 QuorumPeerMain

19309 ZeppelinServer

20046 owl-webapp-1.0.0.jar

NOTE: all the processes above should have been stopped. If you still see these process below wait a few moments and try the jps command again. If the processes still show up then you can run

kill -15 <PROCESS ID>

Step 2: Replace files and update 1 file

Once the process are stopped extract the owl-packages-1.1.0.tar.gz and replace 3 of the following files, and while 1 file needs to be updated.

Files	1.0.0 version	1.1.x version
owl-core	owl-core-1.0.0-jar-with-dependencies.jar	owl-core-1.1.x-jar-with-dependencies.jar
owl-webapp	owl-webapp-1.0.0.jar	owl-webapp-1.1.x.jar
owlcheck	owlcheck	owlcheck
owlmanage.sh		owlmanage.sh (needs to be updated)

Step 3: Update owlmanage.sh script

Inside the owl/bin directory please find the owlmanage.sh script. Please update this file to reflect the new 1.1.x version. example:

vi owlmanage.sh

example screenshot using vi to edit the owlmanage.sh script

Copy and paste

:%s/1.0.0/1.1.x/g

right inside the vi edited owlmanage.sh script and hit enter. This string will substitute all 1.0.0 values found in the owlmanage.sh script with 1.1.x values globally in the entire file (replace the x with the appropriate version).

enter

:wq!

To write and quit the file once the modification has been made.

Step 4: Start owl and run maintenance

Once the new files have been replaced and the owlmanage.sh script has been updated. Use the owlmanage.sh script to start all the services. This script is located in the owl/bin directory and start all processes using the command below

./owlmanage.sh start

Check to make sure all the processes have started using the “jps” command.

jps

command, output as follows.

19169 OServerMain

18711 HMaster

18823 HRegionServer

1600 QuorumPeerMain

19309 ZeppelinServer

20046 owl-webapp-1.1.0.jar

inside the owl/bin directory run the following command.

java -cp owl-core-1.1.x-jar-with-dependencies.jar -Dlog4j.configuration=file:///<path to/log4j.properties> com.owl.core.maintenance.UpgradeTableExt "localhost" "2181" "/hbase" "_UPGRADE_2"

Java needs to be on the path for command to work (if not you will have to specify the full path to java). If you are not running the command in the bin directory you will have to specify the full path to the owl-core-1.1.0-jar-with-dependencies.jar file. In the above command you will have to adjust your log4j.properties file (owl maintains log4j.properties files in ../owl/config/). com.owl.maintenance.UpgradeTableExt is the class name inside the jar file that is going to perform the heavy lifting. The remaining parameters wrapped in double quotes are as follows:

“localhost” = server name / ip of hbase that is going to be upgraded

“2181” = the port zookeeper is listening on to establish a connection to hbase

“/hbase” = the directory maintained in zookeeper for Owl’s hbase

“_UPGRADE_2” = the extension that is added at the end of new tables that are going to be created.

When the process is executed it will create all the required tables in Hbase appending _UPGRADE_2 extension in order not to interfere with current tables from 1.0.0. Once all tables are created owl will compare the 1.0.0 tables with the 1.1.0 table and output a list of ALTER commands that will need to be run in sqlline.py (what columns in which tables changed -- if any). When completed the process will then delete all the tables with extension “_UPGRADE_2”. NOTE: for any net new tables that have been created owl-web and owl-core will automatically create them.

Sample output of the alter commands in the screenshot below.

Which is stating these are the ALTER TABLE commands that need to be executed within sqlline.py.

ALTER TABLE "DATASET_LEDGER" ADD "rc" BIGINT; ALTER TABLE "DATASET_LEDGER" ADD "peak" INTEGER; ALTER TABLE "DATA_ASSET" ADD "MON" INTEGER; ALTER TABLE "DATA_ASSET" ADD "TUE" INTEGER; ALTER TABLE "DATA_ASSET" ADD "WED" INTEGER; ALTER TABLE "DATA_ASSET" ADD "THU" INTEGER; ALTER TABLE "DATA_ASSET" ADD "FRI" INTEGER; ALTER TABLE "DATA_ASSET" ADD "SAT" INTEGER; ALTER TABLE "DATA_ASSET" ADD "SUN" INTEGER; ALTER TABLE "DATA_ASSET" ADD "TIME_ZONE" VARCHAR;

Step 5: Run alter commands

On the server running the owl-web application inside of phoenix/bin launch ./sqlline.py specifying the <zookeeper hostname>:2181. In the example below we are using standalone mode so every owl component is installed on the same host.

Now copy and paste all the ALTER TABLE commands in the above (or from your output) into sqlline.py. As show below.

Notice that the first ALTER TABLE command fails, this is to be expected as the column already exists inside this table as a column of type INT.

However, from version 1.0.0 to 1.1.x this column “rc” = which stands for row count changed from INT to BIGINT. In order to appropriately handle this change we have to do the following.

Step 7: update “rc” column from INT to BIGINT

Step 7a: Create temp table

First we have to create a temp table that mirrors the schema for DATASET_LEDGER which we will call DATASET_LEDGER_2. From within sqlline.py execute

CREATE TABLE IF NOT EXISTS DATASET_LEDGER_2 ("dataset" VARCHAR NOT NULL, "run_id" VARCHAR NOT NULL, "score" INTEGER, "rc" BIGINT, "pass_fail" INTEGER, "peak" INTEGER DEFAULT 1,"updt_ts" DATE CONSTRAINT DATASET_LEDGER_2_PK PRIMARY KEY ("dataset", "run_id"));

Step 7b: Copy dataset_ledger table to temp table “dataset_ledger_2”

UPSERT INTO DATASET_LEDGER_2("dataset","run_id","score","rc","pass_fail","peak","updt_ts") SELECT "dataset","run_id","score","rc","pass_fail","peak","updt_ts" FROM DATASET_LEDGER;

Step 7c: Validate the data copied the temp table “dataset_ledger_2”

SELECT count(*) FROM DATASET_LEDGER; SELECT count(*) FROM DATASET_LEDGER_2; SELECT * FROM DATASET_LEDGER limit 10; SELECT * FROM DATASET_LEDGER_2 limit 10;

Step 7d: Drop the “rc” table from dataset_ledger table

ALTER TABLE DATASET_LEDGER DROP COLUMN "rc";

Step 7e: Add the “rc” table back to dataset_ledger table as BIGINT

ALTER TABLE DATASET_LEDGER ADD "rc" BIGINT;

Step 7f: Copy data from temp table into original table

UPSERT INTO DATASET_LEDGER("dataset","run_id","score","rc","pass_fail","peak","updt_ts") SELECT "dataset","run_id","score","rc","pass_fail","peak","updt_ts" FROM DATASET_LEDGER_2;

Step 7g: Validate data copied back to original dataset_ledger table

SELECT count(*) FROM DATASET_LEDGER; SELECT count(*) FROM DATASET_LEDGER_2; SELECT * FROM DATASET_LEDGER limit 10; SELECT * FROM DATASET_LEDGER_2 limit 10;

Step 7f: Drop temp table

DROP TABLE DATASET_LEDGER_2;

The upgrade from 1.0.0 to 1.1.x has been completed. Exit ./sqlline.py by entering !quit <enter>.

Validate that you can run owlcheck scripts as you have been able to before.