Data source support

The Continuous Compliance service supports profiling, masking, and tokenizing a variety of different data sources including distributed databases, mainframes, PaaS databases, and files. At a high level, Continuous Compliance breaks up support for data sources into two categories:

• Delphix connectors: These are data sources that the Delphix Engine can connect to directly using built-in connectors that have been optimized to perform masking, profiling, and tokenization. Delphix Connectors are available as Standard Connectors and Select Connectors. Standard Connectors are bundled with the Continuous Compliance Engine. Select Connectors are an add-on to the Delphix engine and require a separate installation and configuration process.

• FEML sources: FEML (File Extract Mask and Load) is a method used to mask and tokenize data sources that do not have dedicated Delphix Connectors. FEML uses existing APIs from data sources to extract the data to a file, masks the file, and then uses APIs to load the masked file back into the database.

Standard connectors

The Delphix Engine has standard masking connectors for the following data sources:

• Distributed database: Db2 LUW, Oracle, MS SQL, MySQL, SAP ASE (Sybase), PostgreSQL, MariaDB

• Files: Fixed Width, Delimited, XML

For a detailed view of all the versions, features, etc. Delphix supports each data source - see the sections below.

Select and Premium connectors

The Delphix Engine has Select masking connectors for the following data sources:

• Distributed database: Salesforce (Compliance Accelerator documentation), CockroachDB, and SAP HANA 2.0 (SAP Compliance documentation)

• Mainframe/Midrange: Db2 Z/OS, Db2 iSeries, Mainframe data sets.

For a detailed view of all the versions, features, etc. Delphix supports each data source - see the sections below.

Database Masking applies only to physical tables within databases. It does not support masking for other types of database objects such as views, materialized views, federated systems, or proxy objects. Ensure that all data requiring masking resides within physical tables to utilize the masking functionality effectively.

Db2 LUW connector

Introduction

Db2 for Linux, UNIX, and Windows is a database server product developed by IBM. Sometimes called Db2 LUW for brevity, it is part of the Db2 family of database products. Db2 LUW is the "Common Server" product member of the Db2 family, designed to run on the most popular operating systems. By contrast, all other Db2 products are specific to a single platform.

Support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings page.

2. Restart the Compliance engine.

3. Create a Db2 connector in Continuous Compliance with the relevant parameters. Upload a properties file for the connector with the following:
   sslConnection = True

Oracle connector

Introduction

Oracle Database (commonly referred to as Oracle RDBMS or simply as Oracle) is a multi-model database management system produced and marketed by Oracle Corporation.

Support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Compliance engine.

3. Create an Oracle connector in Continuous Compliance with JDBC URL, similar to below examples, depending upon the Oracle database and certificate configuration.
   Note: SSL connection with BASIC Oracle connector is not supported.

   CODE

   jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=servername)(PORT=2484))(CONNECT_DATA=(SERVICE_NAME=servicename)))
   jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=servername)(PORT=2484))(CONNECT_DATA=(SID=SID_NAME)))
   jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=servername)(PORT=2484))(CONNECT_DATA=(SID=SID_NAME))(SECURITY=(SSL_SERVER_CERT_DN="CN=<certificate cn_name>")))

Microsoft SQL Server connector

Introduction

Microsoft SQL Server is a relational database management system developed by Microsoft. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network (including the Internet).

Support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Compliance engine.

3. Create a Microsoft SQL Server connector in Continuous Compliance with the relevant parameters. Upload a properties file for the connector with the following:

   CODE

   encrypt = true 
   trustServerCertificate = false / true <—---- Depending upon whether to directly accept the database certificate without checking certificate common-name
   hostNameInCertificate = 10-110-229-143.qa-ad.delphix.com   <—---- This property is only required in case trustServerCertificate is set to false

Google Cloud SQL IAM Authorization setup

To authorize connections from Continuous Compliance to a Google Cloud SQL Microsoft SQL Server instance, do the following:

1. Provision a Google Compute Engine running Continuous Compliance. In the compute engine’s settings, enable Cloud SQL in the Identity and API access section.

2. Create a built-in Microsoft SQL Server connector with the following settings:

   1. Advanced Connector with JDBC URL:

      CODE

      jdbc:sqlserver://localhost;databaseName=<your-database>;encrypt=false

   2. Upload a property file with the following:

      CODE

      socketFactoryClass=com.google.cloud.sql.sqlserver.SocketFactory
      socketFactoryConstructorArg=<connection name of the SQL Server instance from the Google Cloud web console>

PostgreSQL connector

Introduction

PostgreSQL, often simply Postgres, is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards compliance. PostgreSQL is developed by the PostgreSQL Global Development Group, a diverse group of many companies and individual contributors. It is free and open-source, released under the terms of the PostgreSQL License, a permissive software license.

Support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Compliance engine.

3. Create a PostgreSQL connector in Continuous Compliance with the relevant parameters. Upload a properties file for the connector with the following:

   CODE

   ssl=true
   sslmode=verify-full
   sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory

   Note: To use the verify-full setting (highest security), the PostgreSQL database certificate’s Common Name (CN) field must match the hostname. To check the CN value in the certificate: openssl x509 -in server.crt -text -noout

Google Cloud SQL IAM Authorization setup

To authorize connections from Continuous Compliance to a Google Cloud SQL PostgreSQL instance, do the following:

1. Provision a Google Compute Engine running Continuous Compliance. In the compute engine’s settings, enable Cloud SQL in the Identity and API access section.

2. Create a built-in PostgreSQL connector with the following settings:

   1. Host: 127.0.0.1

   2. Port: 12345

   3. Upload a property file with the following:

      CODE

      cloudSqlInstance=<connection name of the PostgreSQL instance from the Google Cloud web console>
      socketFactory=com.google.cloud.sql.postgres.SocketFactory

Yugabyte connector

Introduction

YugabyteDB, often simply referred to as Yugabyte, is a distributed SQL database management system with a focus on scalability, performance, and high availability. It is designed to be compatible with PostgreSQL and offers distributed ACID transactions, automatic sharding, and fault tolerance. Yugabyte is developed by Yugabyte Inc., with contributions from various companies and individual contributors. It is free and open-source software, released under the YugabyteDB License.

Support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Delphix Continuous Compliance Engine.

3. Create a YugabyteDB connector in the Delphix Continuous Compliance Engine with the relevant parameters. Upload the properties file for the connector with the following:

ssl=true
sslmode=verify-full
sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory

To use the verify-full setting (highest security), the Yugabyte database certificate’s Common Name (CN) field must match the hostname. To check the CN value in the certificate, run the following command: openssl x509 -in server.crt -text -noout.

CockroachDB connector

Introduction

CockroachDB is a distributed SQL database management system with focus on scalability, performance, and high availability. It is designed to be compatible with PostgreSQL and offers distributed ACID transactions, automatic sharding, and fault tolerance. CoackroachDB is an open-source software developed by Cockroach Labs.

Support matrix

Identity column support is not applicable for CockroachDB because CockroachDB will automatically create a primary key column with the name ‘rowid’ If you don't define a primary key at table creation time

Triggers are not a supported feature for CockroachDB, request to add this feature is tracked under https://github.com/cockroachdb/cockroach/issues/28296

Primary Key and Unique constraints can’t be dropped because of the issues tracked under https://github.com/cockroachdb/cockroach/issues/48026?version=v23.1 and https://github.com/cockroachdb/cockroach/issues/42840?version=v23.1

Note: The Continuous Compliance Engine requires the multiple_active_portals_enabled feature to be enabled in order to successfully mask data in CockroachDB. If job fails with the following error during masking, it indicates that the multiple_active_portals_enabled  is set to False causing the follow error:

CODE

Caused by: org.postgresql.util.PSQLException: ERROR: unimplemented: multiple active portals is in preview, please set session variable multiple_active_portals_enabled to true to enable them.

To resolve this, enable multiple active portals using one of the following methods:- Set the multiple_active_portals_enabled session variable to True. Refer to the CockroachDB documentation CockroachDB documentation for more details.

• Alternatively, create a custom property file with the following content and add it to your CockroachDB connector, following the instructions in the Create, test, edit, or delete a connector page:

CODE

options=-c multiple_active_portals_enabled=true

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Delphix Continuous Compliance Engine.

3. Create a CockroachDB connector in the Delphix Continuous Compliance Engine with the relevant parameters. Upload the properties file for the connector with the following:

ssl=true
sslmode=verify-full
sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory

To use the verify-full setting (highest security), the Cockroach database certificate’s Common Name (CN) field must match the hostname. To check the CN value in the certificate, run the following command: openssl x509 -in server.crt -text -noout.

MySQL / MariaDB connector

Introduction

MySQL is an open-source relational database management system (RDBMS). MySQL was owned and sponsored by a single for-profit firm, the Swedish company MySQL AB. MySQL is now owned by Oracle Corporation.

MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. Development is led by some of the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle Corporation.

A MySQL Connector may be used to connect to either a MySQL or MariaDB database instance.

MySQL support matrix

NOTE: AWS Aurora MySQL 5.7 is not supported

Google Cloud SQL IAM Authorization setup

To authorize connections from Continuous Compliance to a Google Cloud SQL MySQL instance, do the following:

1. Provision a Google Compute Engine running Continuous Compliance. In the compute engine’s settings, enable Cloud SQL in the Identity and API access section.

2. Create a built-in MySQL connector with the following settings:

   1. Host: igoreme

   2. Port: 123

   3. Upload a property file with the following:

      CODE

      socketFactoryClass=com.google.cloud.sql.mariadb.SocketFactory
      socketFactoryConstructorArg=<connection name of the SQL Server instance from the Goog

MariaDB support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Compliance engine.

3. Create a MySQL/MariaDB connector in Continuous Compliance with the relevant parameters. Upload a properties file for the connector with the following:

   CODE

   useSSL=True
   trustServerCertificate=false
   keyStore=file:/var/delphix/server/etc/.truststore
   keyStoreType=JKS

SAP ASE (Sybase) connector

Introduction

SAP ASE (Adaptive Server Enterprise), originally known as Sybase SQL Server, and also commonly known as Sybase DB or Sybase ASE, is a relational model database server product for businesses developed by Sybase Corporation which became part of SAP AG.

Support matrix

TLS/SSL setup

1. Add the database’s certificate in the setup application using instructions in the Adding a certificate section, in the TrustStore settings article.

2. Restart the Compliance engine.

3. Create a Sybase connector in Continuous Compliance with the relevant parameters. Upload a properties file for the connector with the following contents:

   CODE

   ENABLE_SSL=true

Db2 z/OS and iSeries connectors

Introduction

Db2 for z/OS and iSeries are relational database management systems that run on IBM Z (mainframe) and IBM Power Systems.

Support matrix

TLS/SSL setup

See the instructions in the Db2 LUW connector section

Files connector

Introduction

Data stored in a variety of different formats may be masked using the same algorithms available for other data sources.

Support matrix

Mainframe data set connector

Introduction

In addition to databases and files, the Continuous Compliance Engine can process data stored in Mainframe data sets commonly found on the IBM z/OS operating system. For more information on data sets, see this IBM knowledge center article.

Support matrix

The Continuous Compliance Engine requires that data be encoded in EBCDIC rather than something like ASCII or UTF-8. EBCDIC is the encoding traditionally used on Mainframes.

On-The-Fly masking jobs

Continuous Compliance supports On-The-Fly (OTF) masking jobs where the data is read from a source location and written to a different target location. Only certain combinations of connector types are supported for OTF jobs.

OTF jobs with connectors of the same type are supported. For example, masking data from an Oracle source database to an Oracle target database is supported if both are using the built-in Oracle connector. OTF jobs using Extended Connectors are supported if both the source and target are using the same Extended Driver (the same uploaded JDBC driver). Additionally, OTF jobs with a relational database source and a delimited file target are supported. The following data sources are supported as source connectors for OTF jobs with delimited file targets.

• Oracle

• Db2

• MS SQL

• PostgreSQL

• MySQL / MariaDB

• SAP ASE (Sybase)

• Connectors created as Extended Connectors.

For masking flat files (e.g. XML, delimited, etc) in an on-the-fly masking job, it is no longer required to copy or create empty files on the target. If the file name pattern does not match any file on the source, the execution will reported as success, although no file is masked.

No other combinations of connector types are supported. For example, an Oracle source with a PostgreSQL target, or an MS SQL source with a fixed-width file target, are unsupported.

File masking jobs

Understanding in-place file masking

When performing in-place masking (IP) on files, the system uses a temporary hidden file to ensure the integrity and security of the masking process. Here's how the process works:

• Initial Read and Masking: The system first reads the original file and applies the masking algorithms to its columns based on the inventory configuration.

• Temporary File Creation: The masked data is then written to an intermediary temporary file with the extension *.msk. This temporary file is created on the target connector location. It is hidden and serves as a secure placeholder for the modified data.

• Final Replacement: Once the masking process is completed and the data has been safely written to the temporary file, the data in the temporary (*.msk) file is read, and its content is written over the original file. This ensures the original file is not modified until the masking process is entirely successful, preventing data corruption or loss. This also ensures that the user/group permissions stay the same.

Advantages of Using a Temporary File

Using an intermediary temporary file during the masking process offers a safeguard against failures. If any issue arises during the process, the original file remains unchanged and intact. It is also a requirement to keep the user and group permissions the same on the source file.

Understanding on-the-fly file masking

On-the-fly (OTF) file masking is used when data is read from one (source) file and the masked data is written to another (target) file. Here’s how the OTF masking process works:

• Initial Read and Masking: The system reads the source file and applies the masking algorithms to the data. The masking is performed "on the fly" as the data is being read.

• Writing the Masked Data: The masked data is directly written to the target file. This process eliminates the need to create an intermediary temporary file.

Advantages of OTF file masking:

OTF file masking is significantly faster than in-place masking because it does not require an intermediary temporary file.