Setting up and using Owl Agents
Agents allow for owlcheck jobs to be executed remotely
High Level Architecture of Owl Agent setup
The above image provides a high level depiction of what transpires when using agents within Owl. Job execution is driven by jobs that are written to an agent_q table inside the owl-postgres metastore (via the web / or rest-end point). The agents pull the table every 5 seconds to execute the jobs they are only responsibly for. When the agents picks up their jobs to execute it will launch the job either locally on the agent node itself, or if the agent is setup as an edge node of a cluster it will get launch on a cluster as a spark job. Depending on where the job launches the results of the owlcheck will write back to the metastore (owl-postgres) to reflect inside the owl web UI.
Setting Up an Owl Agent using setup.sh script
New installation can use the setup.sh script to create the connection immediately.
Example setup script to setup an agent = ./setup.sh -owlbase=/home/danielrice -options=owlagent -pgpassword=owlpassword -pgserver=cdh-edge-dan.us-east4-c.c.owl-hadoop-cdh.internal:5432/postgres
You can leave off the -pgpassword and -pgserver parameters but during install it will ask you for them as they are needed to establish a connection back to the owl-metastore.
Setting up an Owl Agent manually
Manual setup will require modification of the owl.properties
spring.datasource.url=jdbc:postgresql://cdh-edge-dan.us-east4-c.c.owl-hadoop-cdh.internal:5432/postgres spring.datasource.username=danielrice spring.datasource.password=JB+kqivzHUNEyyzyGcuVpuoPt2ZOx/kMQyCOG7vqlGF/8KKPEevT spring.datasource.driver-class-name=com.owl.org.postgresql.Driver spring.agent.datasource.url=jdbc:postgresql://cdh-edge-dan.us-east4-c.c.owl-hadoop-cdh.internal:5432/postgres spring.agent.datasource.username=danielrice spring.agent.datasource.password=JB+kqivzHUNEyyzyGcuVpuoPt2ZOx/kMQyCOG7vqlGF/8K spring.agent.datasource.driver-class-name=org.postgresql.Driver
In order to generate an encrypted password string use the ./owlmanage.sh encrypt=enterpassword which will generate the output for the encrypted plain text password entered. That encrypted string can be used for the above parameters so they are not stored in plain text.
Managing Agents
Once your agent is configured start the agent using the ./owlmanage.sh start=owlagent script
An administrator can see an agent register successfully by going to the "Admin Console" and clicking on the "Remote Agent" button
A status indicator will show green if the agent is healthy, the agent id, name, jobs the agent is currently executing along with the connections the agent is allowed to execute a job against.
The icon on the agent row that looks like a pencil and paper allows you to edit the default configuration for the agent so when you walk through the explorer page to launch an ad hoc job from the web. When you select a specific agent it will pre-populate the command with the proper default parameters that were set in this "Edit Agent" configuration.
Assign connections to an agent
In order to have an agent execute jobs in association with a particular DB connection you have to assign the datasource connections to the agents allowed to use that connection.
Setting up an HA Group
If you have multiple Agents you can establish them as an HA Group. When doing so make sure both Agents have the same connections established to them. Click on the "AGENT GROUPS (H/A)" Tab name your HA Group and add the Agents you'd like to participate as Group. NOTE: HA GROUPS will execute jobs in a round robin fashion.
When the Agents have been registered, associated with DB connections, users can now execute a job via the explorer page. See below.
Last updated
