Managing rule sets
This section describes how rule sets can be created, edited, and removed.
The rule set screen
To access the rule sets linked to a specific environment, simply click on the “Rule Sets” tab from any location within that environment. This action will display the rule sets listing screen. If you haven't created any rule sets yet, the list will be empty.
The rule sets screen contains the following information and actions:
Rule Set ID — The numeric ID of the rule set used to refer to the rule set from the Masking API.
Name — The name of the rule set.
Connector Name — The name of the connector.
Source Type — The specific type of ruleset. One of Database, Fixed Width, Delimited, JSON, XML, or DataSet
Metadata — The type of rule set (Database type, File, or Mainframe).
Refresh/Save — Refresh status of the rule set. Only applies to Database rule sets. See here for more details.
Actions — List of actions
Edit — Edit the rule set. See more details below.
Duplicate — Copy the rule set. See more details below.
Delete — Delete the rule set. See more details below.
Refresh — Refresh the rule set. Only applies to Database rule sets. See here for more details.
The rule sets on the screen can be filtered or sorted by the various informational fields by clicking on the respective field. More information on grid filtering and sorting can be found here.
Creating a new rule set
To create a new rule set, first select an Environment by clicking on its name. Then, navigate to the Rule Set tab and click on the "Add Rule Set" button located in the upper right-hand corner.
A dialog box prompts you to select the type of rule set you wish to create. You'll find four options to choose from: Database, Semi-Structured, Flat Files, and Copybook. Simply click on the option that corresponds to your choice, and then proceed by clicking on "Continue".
An Add Rule set wizard appears comprising three steps, depending on the rule set type selection made earlier to start the creation of a new rule set. See "The Add/Edit rule set wizard" for a complete description of this screen and additional options, which will guide you through the process of creating a new rule set.
Editing/Modifying a rule set
To edit a rule set, navigate to the Rule Set tab of an environment, Click on the (…) in the Actions column for the rule set, and select the Edit option from the menu.
An Edit Rule set wizard similar to the Add ruleset wizard appears, with data pre-filled depending on the corresponding rule set being edited. See "The Add/Edit rule set wizard" for a complete description of this screen and additional options, modify the rule set as desired using the steps provided here.
The Add/Edit rule set wizard
Step 1: Details
Start by providing a name for the rule set you wish to create.
Select an appropriate connector from the dropdown menu.
The dropdown displays connectors that have already been created. If you haven't created any connectors yet, the dropdown will be empty, and you will not be able to proceed with rule set creation.
The connectors shown in this dropdown are filtered based on the rule set type chosen in the previous dialog.
Additionally, you can quickly filter the connector list to select the desired connector by typing into the input box.
Click on "Next" to proceed.
During the editing of a rule set, all input fields in this step will be non-editable and displayed in read-only mode. Click "Next" to proceed to adding, removing, or modify database tables, files, or file patterns within the rule set.
Step 2: Data Tables / Files
When initially creating a ruleset, this list will be empty. You can select data tables or files that are accessible by the chosen connector to add to this ruleset. When editing the ruleset, this grid will display the tables or file names already included in the ruleset being modified.
Database Rule Set:
The “Data Tables” grid contains the following information,
Table Name — The name of the table in the rule set.
Logical Key — A logical key is a unique, non-null value that identifies a row in the database.
Filter — A WHERE clause to filter or subset the data.
Custom SQL — Customized SQL SELECT query for the table.
While editing the ruleset, a read-only column named "Status" is shown to indicate whether the table is newly added, modified, or removed. The status column will be blank for unmodified items.
You can filter and sort the grid based on the "Table Name" column. For more details on grid filtering and sorting, refer to the information here.
Adding tables to a database rule set.
Click on the "Add Tables" button to include new tables in the rule set being created or modified.
A dialog box titled "Add Database Table" will appear, listing all the table names for the selected connector. From this list, you can select multiple tables and click "Save" to include them in the rule set being created or modified.
The table names can be filtered either by using a substring or a starts with condition. For more details on grid filtering and sorting, refer to the information here.
All the tables can also be selected/deselected by using the checkbox given before the header Table Name, refer to the information here.
Flat Files, Semi-Structured or Copybook Rule Set :
The “Data Files” step contains the following information and actions, depending on the connector type selected.
Fixed-Width : File Name or Pattern, Format File, End of Record, Mask Whole File, Is Pattern.
Delimited : File Name or Pattern, Format File, End of Record, Delimiter, Text Enclosure, Enclosure Escape Strategy, Escape Enclosure Escape Character, Is Pattern.
XML & JSON : File Name or Pattern, Format File, Is Pattern.
Copybook : File Name or Pattern, Format File, Variable Length, Is Pattern.
Properties of a Data File :
File Name or Pattern — The name of the file or pattern in the rule set.
Format File — Name of the file format created specific to the file type.
End of Record — The string of characters that delineates the end-of-record for a file. Note that, for linux this is '\n', and for windows it is '\r\n'.
Mask Whole File — This flag indicates whether the file is to be read as whole or line-by-line. Applicable only for Fixed-Width file type.
Is Pattern — This flag indicates whether or not this file name represents a regular expression.
Delimiter — The delimiter for a delimited file. This field should be left blank for other file types.
Text Enclosure — The text enclosure for the file.
Enclosure Escape Strategy — The character used to escape a literal enclosure character within an enclosed value. By default, this is equal to the enclosure value itself, so doubling the enclosure character escape it.
Escape Enclosure Escape Character — This flag indicates whether the enclosure escape character also escapes itself. For example, if the enclosure escape character is *, then the sequence ** would be treated as a single * character, rather than an escape.
Variable Length — The record format type for the mainframe data set.
While editing the ruleset, a read-only column named "Status" is shown to indicate whether the file is newly added, modified, or removed. The status column will be blank for unmodified items.
You can filter and sort the "File Name or Pattern" column. For more details on grid filtering and sorting, refer to the information here.
Adding files or file patterns to a Flat File, Semi-Structured or Copybook Rule Set.
Click on "Add File Name" to include new file names in the rule set being created or modified.
A dialog box titled "Add File Name” will appear, the “File Name” dropdown lists all the files accessible by the selected connector. You can choose multiple files to include in the rule set.
Click on "Add File Pattern" to input one or more regular expression patterns for matching the data files accessible by the selected connector.
A dialog box titled “Add File Pattern“ will appear, in “Regular Expression Pattern“ field you can specify multiple regular expression patterns. After typing a regular expression click on the option Add item shown in the dropdown or press ENTER key to include the regular expression as a input. To remove a pattern you've added, simply click on the :cancel: icon next to each corresponding regular expression entry.
File pattern syntax
Expressions are case sensitive. A file pattern uses the regular expression syntax defined by the Java Pattern class. The syntax is documented here. For example, the pattern .*\.txt will match any file with a .txt extension, such as example.txt.
Provide all required properties for the files or patterns as necessary. Please note that the fields displayed here may vary depending on the selected connector types. These properties will be applied to all selected files or the multiple regular expressions provided.
Selecting a file format from the drop-down menu is essential. Be sure to choose the correct format that aligns with the data in the selected files or regular expression patterns.
For Mainframe you will see a checkbox for a variable length for dataset where each record within the dataset can have a different length.
For all other file types, select the end-of-record to let the application know whether the file is in Windows/DOS format (CR+LF) or Linux format (LF).
If the file is delimited, you will find a field available to input the delimiter.
if the file is fixed-width type you'll find a checkbox labeled "Mask Whole File" available. Further information about this feature is available here.
Click on the Save button to add all the files names or file patterns selected above to the Data Files grid.
The records are not added to the rule set until the Save button is clicked.
Reset All records
"Reset All" button will restore the rule set to its original state. When editing a rule set, this action reverts any changes since the last save. When creating a new rule set, the rule set is reset to an empty state.
Once you've completed adding, modifying or removing files/tables from the ruleset, click on "Next" to continue.
Step 3: Summary
Review the rule set configuration. Click Back to correct errors or Save to continue.
The "Save" button will remain inactive when no changes have been made to the edited rule set or when creating a new rule set without any tables or files included.
Changing the connector selection will reset all modifications made to the rule set.
Removing a table or file from the ruleset
To remove a table or file from a rule set:
Click the Actions (…) drop-down to the right of a rule set on the rule set screen, and then select the Edit option.
In the rule set wizard's second step (either "Data Tables" or “Data Files”), you can select one or more tables or files either by clicking on the checkboxes or by clicking anywhere on the rows.
Click on the “Remove Selected” button, located on top of the grid.
A confirmation dialog will pop up, displaying the count of tables or files being removed from the ruleset.
Click on Confirm.
Any newly added tables or files will be removed from the grid. However, tables or files already saved to the rule set, their corresponding rows will be updated with the status "Removed".
Click on Next to verify the summary, and then click on Save button.
Select all option is not available for the edit database rule set wizard.
Clicking the Confirm button in the dialog will not immediately delete the tables or files from the rule set. The changes will only take effect after clicking the Save button in the rule set wizard.
Modifying tables properties in a rule set
Logical key
A logical key is a unique, non-null value that identifies a row in the database.
If your table has no primary keys defined in the database, and you are using an In-Place masking strategy, you must specify an existing column or columns to be a logical key. This logical key does not change the target database; it only provides information to Delphix. For multiple columns, separate each column using a comma.
If no primary key is defined and a logical key is not defined an identity column will be created.
To enter a logical key:
In the rule set wizard's "Data Tables" step, click on one or more checkboxes or click anywhere on the rows to select the desired tables.
Click on the “Edit Logical Key” button, located on top of the grid.
Enter the logical key value for the tables in the Logical Key field.
Click on Save.
The logical key cannot be more than 1024 characters in length.
Clicking the Save button in the dialog will not immediately update the table properties. The changes will only take effect after clicking the Save button in the rule set wizard.
Edit filter
Use this function to specify a filter to run on the data before loading it to the target database.
To add a filter to a database rule set table or edit a filter:
In the rule set wizard's "Data Tables" step, click on one or more checkboxes or click anywhere on the rows to select the desired tables.
Click on the “Edit Custom SQL or Filters” button, located on top of the grid.
Ensure the Filter radio button is selected
Enter the where condition for this table in the text input area.
Click on Save.
Be sure to specify column name with table name prefix, for example, customer.cust_id < 1000.
Clicking the Save button in the dialog will not immediately update the table properties. The changes will only take effect after clicking the Save button in the rule set wizard.
Custom SQL
Use this function to supply a customized SQL SELECT Query for the table. Typically, this query will include a WHERE clause to filter or subset the data.
The custom SQL must contain the primary key column (or columns if the table uses a composite primary key) and all columns that will be masked.
To add or edit SQL code:
In the rule set wizard's "Data Tables" step, select a checkbox or click anywhere on the row to select the desired table.
Click on the “Edit Custom SQL or Filters” button, located on top of the grid.
Ensure the Custom SQL radio button is selected
Enter the custom SQL code for this table in the text input area or click on “Generate Custom SQL” to automatically suggest a custom SQL for you.
Click on Save.
You cannot update custom SQL for multiple tables simultaneously. When multiple tables are selected, the Custom SQL radio button will be disabled.
The "Generate Custom SQL" option will be available only for the tables that have already been added to the rule set.
Clicking the Save button in the dialog will not immediately update the table properties. The changes will only take effect after clicking the Save button in the rule set wizard.
Delphix will run the query to subset the table based on the SQL you specify.
A table can have either Custom SQL or Filter at a given time. Adding a filter to the table will remove the table's existing custom SQL, and vice versa.
Control character support for delimited files
The user can specify control character as a delimiter/end of record from UI/API.
Control Character
There are two ways to specify control characters,
From UI click on the CTRL button to open a virtual keyboard, where you can choose the desired control character.
From UI or API manually input the control character in the input fields. Ensure it's in the format $[hex value of the control character], such as $[01] for ^A.
The control character value supports the UTF-8 character set. Avoid using '^t' or '\t'. Instead, utilize the CTRL button and select the appropriate control character.
Control character as a delimiter
In order to use control character as a delimiter, the user needs to click on CTRL button inside delimiter input text or manually enter the control character value.
Control character as an end of record
In order to use control character as an end of record, the user needs to click on CTRL button inside custom end of record input text or manually enter the control character value.
Control character as a value
Control characters are supported as values in a delimited file. No special configuration is necessary. Simply configure the delimited file format as usual.
The user doesn’t need to configure anything extra if the control character is only part of the value and not being used as a delimiter or end of record. However, the user needs to define delimiter/end of record as per the requirement.
Define enclosure escaping strategy for delimited files
The user can configure the enclosure escape character from the UI/API to escape the enclosure. To configure the enclosure escape character from the UI, user needs to select the "Enclosure Escaping Strategy" dropdown value as per below options on the edit rule set popup window,
Double enclosure
Double enclosure option will set the escape character value same as enclosure value. For example, if the enclosure escape character is " then escape character value will be " as well.
Custom
By selecting custom option user can specify any single character as an enclosure escape character except the "escape sequences" and "control characters".
Default Enclosure Escape Character
The default value for "Enclosure Escaping Strategy" is "Double Enclosure".
Escape "enclosure escape character"
Selecting this checkbox indicates whether the enclosure escape character also escapes itself. For example, if the enclosure escape character is " then the sequence "" would be treated as a single " character, rather than an escape.
Configure enclosure escape character for a large rule set
To configure the enclosure escape character for a large rule set, you can use this API Script.
Refreshing a rule set
Refreshing a rule set will result in the columns in the tables in the rule set being rescanned. As a result, the inventory associated with the rule set will also be refreshed, but any pre-existing algorithm assignments will be retained.
To refresh a rule set:
Click the Actions (…) drop-down to the right of a rule set on the rule set screen, and then select the Refresh option.
The Refresh/Save icon on the grid will turn to an hourglass as the associated tables are rescanned.
After the refresh is complete, the Refresh/Save icon will return to the circular arrow.
Copying a rule set
If you copy a rule set, the inventory associated with that rule set will also be copied. Also, any filter conditions defined for that rule set will be copied.
To copy a rule set:
Click the Actions (…) drop-down to the right of a rule set on the rule set screen, and then select the Duplicate option.
The Duplicate rule set window appears.
Enter a Name for the new rule set.
Click Save.
Modify the rule set as you want, using the procedures described here.
Deleting a rule set
If you delete a rule set, the inventory associated with that rule set will also be deleted. Also, any filter conditions defined for that rule set will be deleted.
To delete a rule set, Click the Actions (…) drop-down to the right of a rule set on the rule set screen, and then select the Delete option.