Release Notes

2.4.2 = 10.2.2019

Features

• Bug Fixes

Change Log / Fixes

• OWL-732 Owl.owlCheck() fails if props.cardOff set to true

• OWL-743 Add test notebook on DataBricks

• OWL-730 Cover Dupes activity with test notebooks on DataBricks

• OWL-744 Cover Profile activity with test notebooks on DataBricks

• OWL-742 Add ATM test notebooks on DataBricks

• OWL-701 long running fpg - review fpg buckets - not adhering to 50k limit

• OWL-681 Catalog hangs from spark where 1 =0 wrapper

• OWL-431 Additional User Activity Logging

• OWL-626 Create DATA_PREVIEW role to control who can see stored customer data

• OWL-735 When filequery is enabled, data is cached but then dropped from memory before Profile

Known Issues

• Validate Source if Oracle is Target TO_DATE is applied and breaks -- work around remove TO_DATE from Source -- impala and certain databases that do not support TO_DATE

• Back run -br does not warn you that ${rd} does not exist on wizard

• ~| as a file delimiter does not work in the UI (works via OwlCheck)

• Estimate Job does not always take. You can only click it while on Config Tab first

• Patterns large data long compute cycle

• If Run Date doesn't match to where clause date you run the risk of misaligned data, most notably in outliers and FPG

• Job Scheduler start time is based on JVM start. If time is set to start job at 12:00:00 it may start at 12:00:59 if the JVM started on the 59th second.

• Agent Edit page doesn't allow for edit of script location or name.

• Creating a tenant will need an admin to login and sync schema after creating (should be done immediately after creation).

• Job Estimator will be enhanced with additional parameters to generate more accurate estimates

• postgres username cannot contain a "-" during setup

• password with a $ in it need to be escaped

2.4.0 = 9.25.2019

Features

• Use-ability / bug fixes

• Enhanced scheduler

• Wizard improvements parallel JDBC in wizard

Change Log / Fixes

• OWL-542 Wrong initial Score calculation

• OWL-621 DQ INBOX Refactor and LIST VIEW

• OWL-659 Modify Owl Web to store and handle Spark enabled Principals and JDBC enabled Principals separately

• OWL-351 Document DB Encryption Requirements Per Type

• OWL-549 Build OwlHub Metastore and DAOs

• OWL-585 MT - new install no tables and the export MULTITENANTMODE=TRUE doesn't create main tables

• OWL-629 MT Login Via AD

• OWL-510 DB Notebook issue - Dataset score shows double reduction in the UI.

• OWL-665 Web UI should expose shape sensitivity and column exclusion settings on the Shapes tab on Hoot page

• OWL-623 SENSITIVIY PopUp

• OWL-160 setup.sh better handling if packages are not included in the package directory

• OWL-664 Populate executor cores field using values returned by job estimator

• OWL-622 SEARCH for PROFILE and HOOT in 1

• OWL-548 POC Run multiple tenants by schema in parallel

• OWL-667 Implement Web Controller method to write Shape settings to metastore

• OWL-700 sensitivities reduced for nulls empties and mixed types

• OWL-624 REPORT Summary View for public roles

• OWL-666 Data Shapes Noise Reduction Phase 2

• OWL-630 JWT MT Testing multi login scenarios

• OWL-586 Cannot log into to MT enabled UI trying owlhub

• OWL-511 DB notebook error in UI. UI shows 1 column added no matter what dataset I run against

• OWL-708 Fix CMD wrap

• OWL-707 Validate CMD to throw error when no 'TO_DATE' for Oracle

• OWL-652 dlexc dlinc added for categorical outliers

• OWL-532 Documentation on SQL Editor

• OWL-263 Implement Kerb Principal and Keytab handling in Owlcheck

• OWL-572 AD Auth Providers Per Tenant

• OWL-658 Expand Owlcheck to handle multiple Principals/Keytabs for different purposes

• OWL-498 Test Case - Match All Hoot Components for FPG and Validate Source to Hoot BLL and Sub Scoring Components to Scores BLL

• OWL-691 Handle Web UI for endpoints that return a 403 for no Data Preview Access

• OWL-690 Refactor Hoot Page Permissions for DATA_PREVIEW Role

• OWL-657 Kerberos Connect-As for JDBC connections

• OWL-693 Upload File to Rule Truthset OWL-618 HOOT job ERROR MSG display

• OWL-689 opts doesn't seem to have pguser/pgpassword or host parameters

• OWL-656 Establish Secured HDP 3.1 cluster for testing

• OWL-613 datashapescore flag is not being honored

• OWL-698 dupe first character as 0 fixed

• OWL-620 Data Preview TYPES in header vs LABELS

• OWL-695 Validation for Truthset Forms

• OWL-694 Concat Columns for Categorical Outliers

• OWL-619 Catalog +/- to arrow

• OWL-696 Add run tab to Rule page to re-run owlcheck with rules

• OWL-563 MT Documentation

• OWL-565 Logging validation

• OWL-562 Data preview on hoot page with masking checkboxes in header

• OWL-647 Exclude columns from Shape discovery in Owl Core

• OWL-604 MT - trying to execute a job to a specific agent throws an error

• OWL-682 Datashape table is not being cleared when re-running a existing day

• OWL-530 Documentation on Rules - update the rules section of docs

• OWL-648 Auto configure Shapes Sensitivity

• OWL-411 UI display of estimation details on wizard

• OWL-410 Job resource requirement calculation

• OWL-684 Sample 10 examples of Shapes Data Preview Highlights

• OWL-610 Testing validate source from UI

• OWL-685 Bin Month By Month fix date parsing

• OWL-553 Stress Test Concurrent tenant logins

• OWL-552 "Web Scheduled Tasks by Schema (cache job schedule)"

• OWL-632 Refactor Outlier History and Preview

• OWL-555 Owl setup with option to enable multi tenant with app props

• OWL-554 Owlcheck web custom -h flag if MT enabled

• OWL-362 Documentation user setup on all three encryption options

• OWL-361 Create new table/DAO/Domain Object for Col Masking

• OWL-209 New log4j2.xml file changes the owlcheck parameters

• OWL-593 MT with the use of agents

• OWL-671 Misleading example/hint SQL statements on Rule -> FreeformSQL -> SQL Editor page

• OWL-715 "Hive emits columns formated table.colum breaks ValSrc auto mapping"

• OWL-557 JWT/Cookie CURL in multi tenant mode

• OWL-439 New dataset runs do not show schema labels

• OWL-633 explorer wizard <= replaces with >=

• OWL-515 regular -dl outlier (take outlierLimit) cap data preview inserts

• OWL-559 Default configuration of tenant admin on enable MT

• OWL-635 file explorer wizard (local file don't add yarn parameters)

• OWL-558 Alt login path if entering owl hub

• OWL-676 Inject Kerberos Principal and Keytab required to submit Owlcheck via agent configs

• OWL-678 Hive -jdbcprinc -jdbckeytab patch

• OWL-673 Shape DataPreview Batch Insert fails if preview row column value is NULL (DB NULL constraint)

• OWL-479 Run OwlCheck Cluster mode when Hive JDBC is the Source

• OWL-600 "Handle Tab and ""~|"" delimiter for file ingest in Owlcheck"

• OWL-674 MIN/MAX stats = NULL causes NPE in Histograms

Known Issues

• Validate Source if Oracle is Target TO_DATE is applied and breaks -- work around remove TO_DATE from Source -- impala and certain databases that do not support TO_DATE

• Back run -br does not warn you that ${rd} does not exist on wizard

• ~| as a file delimiter does not work in the UI (works via OwlCheck)

• Estimate Job does not always take. You can only click it while on Config Tab first

• Patterns large data long compute cycle

• If Run Date doesn't match to where clause date you run the risk of misaligned data, most notably in outliers and FPG

• Job Scheduler start time is based on JVM start. If time is set to start job at 12:00:00 it may start at 12:00:59 if the JVM started on the 59th second.

• Agent Edit page doesn't allow for edit of script location or name.

• Creating a tenant will need an admin to login and sync schema after creating (should be done immediately after creation).

• Job Estimator will be enhanced with additional parameters to generate more accurate estimates

• postgres username cannot contain a "-" during setup

• password with a $ in it need to be escaped

2.3.0 = 8.30.2019

Features

• Multi Tenancy

• Masking

• JWT CURL

Change Log / Fixes

• OWL-566 Outliers Performance Review

• OWL-522 Rules UI refactor

• OWL-528 Build Controller method that determines whether there is a minority set of data shapes and only returns those shapes

• OWL-605 profileRDD for comparison

• OWL-550 OwlHub Management GUI for Multi Tenancy

• OWL-524 behavior dimension to check for nulls

• OWL-567 Shapes Insert unique idx conflict fix

• OWL-603 Lookback query does not build correct bin boundaries

• OWL-526 Handle Data Shapes more efficiently both for user experience and backend processing

• OWL-590 Turn Outliers Preview Highlight back on with bulk write to DB

• OWL-587 Load Activity does not correctly build time binned historical queries for PG/Oracle/MSSQL/DB2

• OWL-608 validate source matches and counts

• OWL-589 Specifying a queue name in the explorer page on the config tab doesn't get added to the ""Run CMD"" owlcheck cli

• OWL-594 Installing just owl agent doesn't prompt for owl-postgres password

• OWL-595 Agent Configs number of cores doesn't get added to the owlcheck

• OWL-556 Customize Login page to select tenant before login

• OWL-592 Compute Stats in DQRowCheck routine to optimize Profile Activity

• OWL-614 Don't down score for Schema Evolution on first run

• OWL-591 MultiTenant (MT) creation of more then one tenant spawns duplicates

• OWL-598 Enable core to handle non-UTF-8 encoding when parsing target and source files

• OWL-600 "Handle Tab and ""~|"" delimiter for file ingest in Owlcheck"

• OWL-596 setup.sh when only installing Postgres misses creation of owl-env.sh script

• OWL-545 setup.sh script all parameterized for unattended installations of owl

• OWL-130 Agent Documentation

• OWL-607 dupe test for block index

• OWL-606 validate source matching values with count

• OWL-576 Integrate BitBucket with Jenkins to automatically build and test batches of commits

• OWL-134 MultiTenancy Documentation

• OWL-133 HA Documentation

Known Issues

• Job Scheduler start time is based on JVM start. If time is set to start job at 12:00:00 it may start at 12:00:59 if the JVM started on the 59th second.

• Agent Edit page doesn't allow for edit of script location or name.

• Creating a tenant will need an admin to login and sync schema after creating (should be done immediately after creation).

• Job Estimator will be enhanced with additional parameters to generate more accurate estimates

• postgres username cannot contain a "-" during setup

• password with a $ in it need to be escaped

2.2.1 = 8.23.2019

No Features

• Bug fixes only

Change Log/Fixes

• Performance related improvements.

• Bulk inserts into owl-postgres metastore

2.2.0 = 8.12.2019

Features

• Improvements in DataBricks integration

• SQL Editor

• Explorer Page usability enhancements

Change Log/Fix

• OWL-544 Datasets with a high number incidents of shape issues align all rows in preview

• OWL-541 Create Additional Estimate Button on Explorer to display row and col values in addition to run settings

• OWL-539 Implement Logic for runDate on owlCheck

• OWL-538 Performance tuning of Histogram String when low number of rows and high number of columns

• OWL-536 datashapes drill in with preview update

• OWL-535 template creation with $rd parameter support.

• OWL-534 Break Load().execute() into modular methods to better control loading and caching

• OWL-533 change setup.sh script to pull hostname and add to owl-env.sh in place of localhost

• OWL-527 Only look for data shapes on columns that have not been tagged as a semantic schema type

• OWL-523 Dupe Slider in Wizard

• OWL-521 VarSrc Lib in wizard owlcheck

• OWL-520 Add Data Preview Collapsible Area Above Agent Area on Explorer

• OWL-514 Owl Notebook API support for loading historical data

• OWL-512 master - UI explorer adding custom persist at runtime.

• OWL-504 256Bit Encryption on internal Enc Method

• OWL-503 log encryption for Spark UI

• OWL-502 Agent add exports in owl.properties specifically for agent due to shade of postgres

• OWL-501 Clean Activity fix (VizErrors.Clean() test case fix)

• OWL-500 Load Phase Build Historical breaks when dataCol predicate comes after where statement

• OWL-497 Alter table commands between versions of owl - (Postgres release)

• OWL-495 Databricks notebook execution have to manual update the application.properties file for it to write back to DB

• OWL-494 Hoot Page Always Show 'Processing' for Hourly and Minute Run Ids

• OWL-493 2019-06-18T11:08:00.000 0000 Date Parser Format Support

• OWL-488 Categorical Data Preview Inserts - Check for Accuracy

• OWL-484 Test Rules for Notebook API per OWL-472

• OWL-483 --Conf support at CmdLine

• OWL-481 Wizard Kerb Hive Support for OWL-479

• OWL-480 Kerb JDBC Connection for OWL-479

• OWL-479 Run OwlCheck Cluster mode when Hive JDBC is the Source

• OWL-466 UI Save enhanced notifications

• OWL-451 Run Tab enhancements In Wizard

• OWL-437 Hoot DataPreview Highlight and Shapes

• OWL-406 UI Global connection with customer password manager option

• OWL-390 more s3 test cases

• OWL-346 Hoot dynamic loading with UI improvements

• OWL-322 Truth Set Rules

• OWL-98 Install of components, interaction base on what is NOT installed.

• OWL-48 COLUMN SEARCHING IN WIZARD

Known Issues

• Job Estimator will be enhanced with additional parameters to generate more accurate estimates

• postgres username cannot contain a "-" during setup

• password with a $ in it need to be escaped

2.1 = 7.18.2019

Features

• Scheduler added

• Support for XML/ORC

• Disable/Enable Notebook & Orient features for the UI

• Admin Delegate sub-authorities for Managing User Access to Various Functions

• owlcheck support of --conf (in owl-env.sh)

• Connection Templates for common drivers

Change Log/Fix

• OWL-453 Reference Record for Record Added Dropped

• OWL-285 rule history -> removeAllRules,

• OWL-486 Dupes on an integer column causes dupes to NOT be found,

• OWL-460 owlcheck examples on gitbook,

• OWL-459 documentation: add information about the scheduler (screenshots) and how to use it.,

• OWL-457 backrun (-br) using cluster mode is failing do to what seems to be a spark context issue,

• OWL-446 Any time the timezone is updated it generates more behaviors,

• OWL-445 Setup.sh script to include all drivers at install time,

• OWL-426 OwlCheck cannot run in Cluster mode when Hive JDBC is the Source,

• OWL-423 Postgres Multiple Schema test within the same DB,

• OWL-387 If Alert mail server is not setup - alert user when the click alerts that no email server has been setup

• OWL-386 If Orient or Zeppelin are not installed we should just remove the buttons from the UI

• OWL-344 owlcheck support of --conf

• OWL-476 DataPreview Working in Hoot but not Rules/Profile

• OWL-468 Databricks - Classpath issues running notebooks

• OWL-467 Upgrade Web/Common to Spring 2

• OWL-465 Application Features Endpoint

• OWL-464 Controller Endpoint for Update Agent

• OWL-462 Edit Agent Modal

• OWL-461 Page Running status for Notebooks and Orient on Load

• OWL-455 Shapes Testing

• OWL-454 Hoot Actions Tab Updates

• OWL-452 Pattern - rebranding and usability in explorer

• OWL-450 Connection Templates For Common Drivers

• OWL-449 Scheduler - one click Job scheduling

• OWL-447 Back running - replays with templates

• OWL-441 Job Scheduling

• OWL-435 Shapes drill in preview does not scroll horizontal

• OWL-433 Batch insert for DAO DataPreview by Executor

• OWL-430 Change support email address

• OWL-400 Validate Passfail limit < 75 fails runs

• OWL-393 Application Status Endpoint

• OWL-25 Owlcheck on XML file format

• OWL-24 Owlcheck on ORC file format

• OWL-469 Agent New Default values

• OWL-456 JOB UserName

• OWL-349 Alert Form Validation

• OWL-487 Add Hive Warehouse Connector to support "Hive Native" on HDP3.x

• OWL-485 Support Owlcheck on XML files

• OWL-473 Outlier Binned Timeseries (HOUR/MIN) does not calculate correct time range (startdate time missing),

• OWL-472 Migrate Notebook API to use Load activity

• OWL-470 Use New DAO batch methods to better control Shapes Issues written to metastore,

Known Issues

• Job Estimator will be enhanced with additional parameters to generate more accurate estimates

• postgres username cannot contain a "-" during setup

• Agent logic has issues with shaded jar (com.owl.org.postgresql.Driver) that is rectified in next version OWL-501.

2.0 = 6.27.2019

Features

• Encryption at REST

• Job Status Page

• Rules Src Validate Function

• Explorer search and custom filters

• Profile and Hoot UI Enhancements

• New App props (see all)

  • Key 2.0 Additions for custom configuration of the meta-store DB in owl-env.sh:

    • SPRING_DATASOURCE_URL

    • SPRING_DATASOURCE_USERNAME

    • SPRING_DATASOURCE_PASSWORD

  • Matching configuration properties for owlcheck in owl.properties

    • spring.datasource.url

    • spring.datasource.password

    • spring.datasource.username

Change Log/Fix

• OWL-292 Create Agent Component on Wizard Page

• OWL-293 Refactor Owlcheck submission for agent/agent_group as agent and agent_group use Serial/Integer

• OWL-294 LOG Files CLUSTER Mode

• OWL-296 Spark options (bounds/colname/partition)

• OWL-298 Owl-env.sh script use Lib option

• OWL-303 Cloudera Test Env Setup

• OWL-304 Rules Refactor Speed Increase

• OWL-305 Rules Score Update

• OWL-306 Cloudera Env infrastructure support

• OWL-307 Rules HOOT shows (5) when nothing breached

• OWL-309 Job Status . FINISHED, RUN ALL in UI table

• OWL-310 Rule page to use a runId for toggling

• OWL-311 Profile Page to Use new Datapreview and Histogram expand fullscreen

• OWL-312 Current date selected in hoot page heatmap chart (top chart #2) not showing for long runId

• OWL-313 Profile Page Rework all sections

• OWL-314 null in filtergram check and replace with string 'null'

• OWL-315 cascading and repeating names in scorecard page

• OWL-317 Spark Tuning R&D

• OWL-323 Rules Current vs Last Run (t1) usability enhancement

• OWL-334 Rules Performance

• OWL-336 Rules UI Error Handling

• OWL-337 Rules CodeMirror Syntax Validation

• OWL-338 Item labels on behavior items

• OWL-340 Application User Logs

• OWL-341 AgentQ Table for issuing agent job requests

• OWL-342 Job Status Page Enhancements for all owlChecks

• OWL-345 Ability to change all default passwords (such as orient/postgres)

• OWL-350 HOOT TimeZone Edit button

• OWL-356 Security Configuration option for type of encryption to be used

• OWL-357 Configuration Setting for where to get public encryption key for encryption/decryption

• OWL-363 HOOT Processing outlier...dupe...

• OWL-364 Jenkins Build process / Test invoke / email on failures

• OWL-365 Filter options for JDBC query before expanding the DB

• OWL-366 owlcheck overwriting the semantic schema issue

• OWL-370 Dupe Detection runtime reduction

• OWL-371 Activity Decouple

• OWL-372 Hist refactor for performance

• OWL-373 WebApp Driver UnRegister Bad Driver

• OWL-375 Load Time Trigger

• OWL-378 Datashape Distribution turn back on, profile the performance issue

• OWL-379 Retrain Button

• OWL-380 RECORD Changes and SCHEMA Changes not showing in UI

• OWL-381 HOOT timezone says TRUE

• OWL-382 Item Label Outlier

• OWL-383 Install and test v2.0 of Owl with Postgres version 9.6.5

• OWL-384 Encrypt data within postgres Data_Preview table (decrypt automatically)

• OWL-385 dataset_field table encryption using java encryption key held in owl

• OWL-388 encryption of the dataset_field_value table

• OWL-389 dataset_hist table doing columns hist_values

• OWL-395 ItemLabel table encryption of column item_values

• OWL-396 Explorer RUN Tab UI

• OWL-397 Load Time Alert

• OWL-398 ReTrain Changes Score on behavior item for previous runs

• OWL-401 DuplicateLabels added anytime I down train an item

• OWL-402 observation table - encrypted (out_column, out_value, out_median)

• OWL-403 outlier table encryption at rest in postgres

• OWL-404 Once you label something as ignore it cannot be set back to - not be ignored

• OWL-413 Owl web app fails to restart with error below

• OWL-414 -h command doesn't work with owl-core if not using the default schema name in Postgres

• OWL-416 Hive JDBC connection in Explorer page

• OWL-417 Remove HBase Phoenix

• OWL-418 Data Cleanup Thread

• OWL-419 PG Connections encryption pwd

• OWL-420 PG Credentials in props and Batch Insert for Executor Insert Model

• OWL-421 Oracle mod to filtergram needs wrapper for rownum where clause limit

• OWL-422 Filtergrams with Oracle not working (SQL Exception: Unable to execute sql for this dataset)

• OWL-424 Add agent start/stop to owlmanage.sh

• OWL-425 val src observations inserts

• OWL-427 Audit Trail Timestamp conversion from epoch long to a readable date

• OWL-428 Performance improvements

• OWL-434 Cleanup/Constraint options for insert Data Preview

• OWL-444 DevOps - Setup script doesn't interrogate user for orient password at setup

• OWL-466 UI Save with no success/failure response back to user

• OWL-471 Ensure valsrcinc/valsrcexc columns are applied to schema checks and counts

Known Issues

• Default notification email "To" email address is not customizable