Installing
Obtain owl-packages.tar.gz file
Extract the contents to a folder (this will become the -owlpackage argument).
Optional: See Appendix about encrypting the partition using LUKS prior to installing Owl.
chmod the setup.sh script to allow its execution.
The setup.sh script is the script that will drive the placement of the packages and has some arguments that can be passed to it:
Setup.sh arguments
-non-interactive skip asking to accept JAVA license agreement
-skipSpark skips the extraction of spark components
-stop do not automatically start up all components (orient,owl-web,zeppelin,postgres)
-port= set owlweb application to use defined port
-user= Optional Parameter (default will be current user) set the user to run owl as.
-owlbase= set base path to where you want owl installed
-owlpackage= Optional Parameter (default is current working directory) set owl package directory
-help display this help and exit -options= the different owl components to install (comma separated list) --- owlagent,owlweb,zeppelin,postgres,orient,spark
-stop do not automatically start up all components (orient,owl-web,zookeeper, hbase will not be started on completion of setup.sh script)
Example:
The Owl tar ball has been extracted to this folder on my EC2 Instance: /home/ec2-user/packages/
Owl will be running as the ec2-user
The owl-web application will run on port 9000
The base location for the setup.sh script to create the owl folder and place all content under owl will be: /home/ec2-user/
./setup.sh -port=9000 -user=ec2-user -owlbase=/home/ec2-user -owlpackage=/home/ec2-user/package
Example installing just the agent (perhaps on an Edge node of a hadoop cluster):
The Package has been extracted to this folder on my EC2 Instance: /home/ec2-user/packages/
Owl-agent will be running as the ec2-user
The base location for the setup.sh script to create the owl folder and place all packages under owl will be: /home/ec2-user/
./setup.sh -user=ec2-user -owlbase=/home/ec2-user -owlpackage=/home/ec2-user/package -options=owlagent
When installing different features questions will be asked
postgres = Postgres DBPassword needs to be supplied
orient = Orient DBPassword needs to be supplied
If postgres is not being installed (such as agent install only) postgres metastore server name needs to be supplied
Launching and Administering owl:
When the setup.sh script finishes by default software is automatically started. The setup.sh also creates the owlmanage.sh script which allows for stopping and starting of all owl services or some components of services.
The setup script will also generate an owl-env.sh script that will hold the main variables that are reused across components (see owl-env.sh under the config directory).
Owl Directory Structure after running Setup.sh
export ORIENTDB_HOST="localhost"
Configuration / ENV settings within owl-env.sh
Contents of the Owl-env.sh script and what is is used for.
OWL-ENV.SH Scripts | Meaning |
export ORIENTDB_PORT="2424" | Port that OrientDB is using |
export SPARK_CONF_DIR="/home/danielrice/owl/cdh-spark-conf" | directory on machine where the spark conf directory resides. |
export INSTALL_PATH="/home/danielrice/owl" | Installation directory of Owl |
export JAVA_HOME="/home/danielrice/jdk1.8.0_131" | Java Home for Owl to leverage |
export LOG_PATH="/home/danielrice/owl/log" | Log path |
export BASE_PATH="/home/danielrice" | The base location under which the owl director resides |
export SPARK_MAJOR_VERSION=2 | Spark Major version. Owl only supports 2+ version of spark |
export OWL_LIBS="/home/danielrice/owl/libs" | Lib Directory to inject in spark -submit jobs |
export USE_LIBS=0 #1 is on, 0 is off | use the lib directory or not. 0 is the default. |
export SPARKSUBMITEXE="spark-submit" | Spark submit executable command. CDH using spark2-submit as example |
export ext_pass_manage=0 #0 to disable 1 to enable | If using a password management system. You can enable for password to be pulled from it. |
export ext_pass_script="/opt/owl/bin/getpassword.sh" | Leverage password script to execute a get password script from the vault. |
TIMEOUT=900 #15 minutes in seconds | Owl-Web user time out limits |
PORT=9003 #owl-web port NUMBER | default port to use for owl-web |
SPRING_LOG_LEVEL=ERROR | logging level to be displayed in the owl-web.log |
SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver | driver class name for postgres metastore (used by web) |
export SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/postgres | JDBC connection string to Owl Postgres metastore |
export SPRING_DATASOURCE_USERNAME=danielrice | Owl Postgres username |
export SPRING_DATASOURCE_PASSWORD=3+017wfY1l1vmsvGYAyUcw5zGL | Owl Postgres password |
export DATASETS_PER_ROW=200000 | Delete datasets after this threshold is hit (must be greater than the default to change) |
export ROW_COUNT_THRESHOLD=300000 | Delete rows after this threshold is hit (must be greater than the default to change) |
export SERVER_HTTP_ENABLED=true | Enabling HTTP to owl web |
export OWL_ENC=OFF #JAVA for java encryption | Enable Encryption (NOTE need to add to owl.properties also). Has to be in form owl.enc=OFF within owl.properties file in this form owl.enc=JAVA. |
PGDBPATH=/home/danielrice/owl/owl-postgres/bin/data | Path for Postgres DB |
export RUNS_THRESHOLD=5000 | Delete runs after this threshold is hit (must be greater than the default to change) |
export HTTP_SECONDARY_PORT=9001 | Secondary HTTP port to use which is useful when SSL is enabled |
export SERVER_PORT=9000 | same as PORT |
export SERVER_HTTPS_ENABLED=true | enabling of SSL |
export SERVER_SSL_KEY_TYPE=PKCS12 | certificate trust store |
export SERVER_SSL_KEY_PASS=t2lMFWEHsQha3QaWnNaR8ALaFPH15Mh9 | certificate key password |
export SERVER_SSL_KEY_ALIAS=owl | certificate key alias |
export SERVER_REQUIRE_SSL=true | Override HTTP on and force HTTPS regardless of HTTP settings |
SECURITYFLAG=1 | Use keytabs/principal for all spark submit jobs |
KEYTABS | (NOT USED) Directory holding keytab files |
KRB5PATH | path to owl's krb5 |
JAASTEMPLATE | JAASTEMPLATE file used to more easily generate |
Configuration / owl.properties file
Example | Meaning |
key=XXXXXX | The license key |
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres | connection string to owl metastore (used by owl-core) |
spring.datasource.password=xxxxxx | password to owl metastore (used by owl-core) |
spring.datasource.username=xxxxxx | username to owl metastore (used by owl-core) |
spring.datasource.driver-class-name=com.owl.org.postgresql.Driver | shaded postgres driver class name |
orientdb.user=root | orient username |
orientdb.pass=xxxxx | orient password |
spring.agent.datasource.url | jdbc:postgresql://10.142.0.19:5432/owltrunk |
spring.agent.datasource.username | owluser |
spring.agent.datasource.password | owlpassword |
spring.agent.datasource.driver-class-name | org.postgresql.Driver |
Last updated
