Managing inventories

An inventory describes all of the data present in a particular rule set and defines the methods that will be used to secure it. Inventories typically include a table or file name, column/field name, data classification, and the chosen algorithm.

The inventory screen

From anywhere within an environment, click on the Rule Sets tab to see the rule set screen. This displays the list of rule sets for the environment. Click on the Rule Set name to access the given rule set’s inventory screen.

The rows on the screen can be filtered or sorted by the various informational fields by clicking on the respective field. More information on grid filtering and sorting can be found here.

Sorting is applied within each group when rows are grouped by Record types or Table name.
The Mainframe grid does not have sorting enabled because the memory is shared across record types and dataset masking is sensitive to the order of the field in the Copybook format.
To show Masked column on the grid which shows information if the particular column/field is assigned with an algorithm, hover on the Algorithm column, click on the ꠵ icon appearing on the right side corner of the dialogue, and choose the Not blank option from the drop down.

Assigning algorithms

To set criteria for sensitive columns or fields:

Click the Actions (…) button to the right of the corresponding column or field, then select the Edit option for the Edit Field/Column dialogue to appear.

From the Inventory screen, the Edit Field/Column dialogue allows you to edit properties only under the Masking section.
To edit the Formatting properties of a file field, go to the Settings → Data Formats and edit the corresponding format. You can also navigate to Settings → Data Formats from Inventory screen using the Edit File Format button present at the right corner, just above the grid.

From the Domain drop-down list, select the appropriate sensitive data element type.
The Continuous Compliance Engine defaults to a Masking Algorithm, as specified in the Settings screen. If necessary, you can override the default algorithm.
1. To select a different masking algorithm, choose one from the Algorithm drop-down list. For detailed descriptions of these algorithms, see the Out-of-the-box algorithm instances article.

If you select a DATESHIFT algorithm and are not masking a datetime or timestamp column, you must specify a Date Format (this field only appears if a DATESHIFT algorithm is selected from the Masking Algorithm dropdown). The default format is YYYY-MM-DD in the legacy UI.

A dropdown provides the capability to add a new date format or select from the existing list in the dropdown. Click on the or icon next to the dropdown for more suggestions on valid formats.

Choose Enable Automatic Updates:
1. Check (Enabled)
  The default setting. A profiling job can determine or update whether to mask a column.
2. Uncheck (Disabled)
  The user decides whether to mask/unmask a column. The user's choice overrides the profiling job.
You can add/remove notes in the Notes text field.
Once complete, click Save, which must be done for any edits to take effect.

Managing database inventory settings

Database inventory screen lists Columns from all the tables in the rule set, the number beside each table name in parentheses is the total number of columns in that particular table.
If a database column is a Primary Key, Foreign Key, or Index, it will be indicated below the column name.
Metadata for the database column appears under the Data Type column including its Length mentioned in parentheses. This information is read-only.
By default, only Table Name, Column Name, Data Type, Domain, Algorithm, and File Format columns will be displayed in the database inventory screen.
If the Inventory Approval Workflow is enabled for the environment, a Status button appears just above the grid if any changes like Add/Edit/Delete/Import inventory are done to any column properties.

Managing a fixed-width or delimited file inventory settings

Search/select a file or file format under the File Format dropdown to create or edit the inventory of sensitive data. The Record Types and Fields for that specific file will appear in the grid below.
Fields are listed in groups of Record Types, which can be collapsed and expanded by clicking on the down arrow icon ⌄ beside each Record Type Name. Users can also Expand All and Collapse All using the Actions button in the right corner, just above the grid.
The count next to the Record Type in parentheses shows the total number of fields in that record.

To add a new Field, Record Type to the inventory, or to manage qualifiers or headers & footers of the selected file format, click on the Edit File Format button in the right corner. Navigate to the selected Settings Format screen and use Add Field.

Managing a JSON file inventory settings

Search/select a file or file format under the File Format dropdown to create or edit the inventory of sensitive data.

To add a new JSON path to the inventory, click on Edit File Format button in the right corner that will navigate to the respective selected Edit Formats screen, where you can use the Add Field button.
Profiling is not supported on the JSON file rule sets.

Managing an XML file inventory settings

Search/select a file or file format under the File Format dropdown to create or edit the inventory of sensitive data.
The fields are displayed with XPath in a flat-grid structure and are, by default, sorted by XPath.

XML attributes can be identified by “@“ in the XPath.

Managing Mainframe inventory settings

Search/select a file or file format under the File Format dropdown to create or edit the inventory of sensitive data.
Fields are listed in groups by a parent field which can be collapsed and expanded by clicking on the down arrow icon ⌄ beside the field name. Users can also Expand All and Collapse All using the Actions button in the right corner, just above the grid.
The count next to the Field name in parentheses shows the total number of children fields.
If a field is type of REDEFINED or REDEF then it will be indicated below the field name.

To edit Redefine conditions, click on the Edit Mainframe Format button at the right corner above the grid to navigate to the respective selected Edit Formats screen, where you can edit Redefine conditions. For more information on Redefine condition see the Managing File Formats article.

Masking a node with Level 88 children is not supported. A Level 88 node refers to the condition name in the Copybook format. If we allow masking a node with Level 88 children, the dataset file generated after masking will no longer be compatible with it’s Copybook format.

Importing and exporting an inventory

Importing and exporting an inventory is CSV format is only supported for database, delimited, fixed-width, and XML inventories.

To export an inventory

Click on the Actions button in the right corner just above the grid, then choose Export Rule Set from the options.
The Export Ruleset window appears with the name of the currently selected Rule Set and provide a corresponding .csv File Name to export.
Click Export.

When the export operation is complete, a .csv file with the name provided earlier will start to be Downloaded on the browser.

To import an inventory

Click on the Actions button in the right corner just above the grid, then choose Import Rule Set from the options.
The Import Rule Set window appears with the name of the currently selected Rule Set.
Click Choose File to browse for the exported comma-separated (.csv) file.
Click Import.

When the import operation is complete, the inventory you imported appears in the Rule Set list for this environment.

Only one rule set can be imported at a time.
The format of an imported .csv file must exactly match the format of the exported inventory. If you plan to import an inventory, you should export it first and then update the exported file as needed before importing it.
After importing the inventory to a 10.0.0.0 version or above Compliance Engine from older versions, rule set refresh is mandatory when the inventory has any document store type assignments, or the user needs to perform document store type masking on the columns from the imported inventory.

Document Store Type masking

This feature provides the ability to mask structured documents that are stored in database columns and delimited files. This is done by marking a column/delimited field as Structured and assigning a respective Document Store Type and File Format to it.

With the release of version 10.0.0.0 of the Continuous Compliance Engine, the document store type masking will support automatic datatype identification. This will be done by using the JDBC SQL Type associated with columns. String and BLOB types will be supported for document store type masking.

With version 10.0.0.0 release

In the case of existing rule sets, a rule set refresh is mandatory before using Document Store Type masking.
Masking jobs having rule sets with Document Store Type assignments will need mandatory rule set refresh. Without rule set refresh job will not be allowed to run.
Masking jobs having rule sets without document store type assignments will not need rule set refresh.
Rule set refresh is not required for newly created rule sets.

For Database columns
- The database column type should be from one of the following JDBC SQL Types: CHAR, NCHAR, VARCHAR, NVARCHAR, CLOB, NCLOB, LONGVARCHAR, LONGNVARCHAR, BLOB, SQLXML.
- BLOB type will not be supported for MySQL databases.
- SQLXML type will be only supported for Oracle databases.
- The file format must be either XML or JSON.
For Delimited file-fields
- Document store type masking for delimited field is supported when JSON or XML data is enclosed by Enclosure and has enclosure escaping strategy as Double Enclosure.
- Only the double enclosure escaping strategy is supported. Custom enclosure escaping strategy and "enclosure escape character" functionality are not supported for delimited fields with structured data.
- More details on how to assign enclosure to delimited file-rule set can be found here.

Database columns with a supported data type or a Delimited file field provide a setting called Data Model, which can be configured as either Plain or Structured.

As shown in the image below, columns with Plain selected as the Data Model can be masked as a single value by assigning a Domain and Algorithm.

When the Structured value is selected for the Data Model, a Document Store Type and File Format can be assigned as shown in the image below.

The image below shows the Inventory screen for the database rule set with a structured column. To quickly access an assigned File Format from this screen (books.xml in this example), click on the Actions button (...) and select the Edit File Format option from the dropdown.

Users can assign algorithms to the fields inside document store selected file format by either clicking on the Actions button (...) and selecting the Edit File Format option in the grid or by going to Settings > Data Format and click on the assigned File Format name.

Multi-column algorithm support for document store type masking

With the release of version 10.0.0.0, Multi-column algorithms will be supported for JSON and XML document store type masking with limited buffer-data size.

Buffer size (in bytes) will be using calculated using the below formula:

((Max_memory_of_Job/No_of_streams_for_job)*CharStreamingBufferLimitRate)/100

The default values will be used when the maximum memory and number of the stream for the job are not defined.
Buffer-data size is configurable via the application setting CharStreamingBufferLimitRate under Mask group settings. For adjusting CharStreamingBufferLimitRate, refer to the Masking API client.

The fields having multi-column assignments should not exceed the limit of buffer data size. In case of exceeding the limit of buffer data size, the job will fail. Users can configure buffer size by adjusting CharStreamingBufferLimitRate to avoid exceeding the buffer data size issue.

Multi-column algorithm with JSON file format

Multi-column algorithm is supported for JSON files and JSON in Document Store Type masking.
Multi-column algorithm is not supported for JSON fields where,
- JSON field is an array.
- JSON fields are part of different arrays.
- JSON fields are on different levels having one or more fields from JSON arrays.

Multi-column algorithm assignment for JSON fields will be validated at the time of assignment. If any of the above combinations are found while assigning a multi-column algorithm, that assignment will not be allowed.

Below is a sample JSON file format with valid and invalid multi-column assignment examples.

Assigning a multi-column algorithm to an invalid combination of JSON fields will produce an error that shows JSON paths.

Multi-column algorithm with XML file format

In the case of XML document store type masking, multi-column algorithm assignment to XML elements will not be validated at the time of assignment. XML can be difficult to find out if an element is a type of an array or a single element until the whole data is read. Here, the masking job will fail immediately when any of the invalid multi-column assignments are found while running the job. Make sure the algorithm assignment should follow the below rules.

Multi-column algorithm for XML file masking is not supported.
Multi-column algorithm assignment to XML attributes is not supported.
Multi-column algorithm is not supported for XML elements where,
- The element is a type of array.
- Elements are part of different arrays.
- Elements are on different levels having one or more elements of type array.

Below is a sample XML file format with valid and invalid multi-column assignment examples.

Inventory Approval Workflow (database rule sets only)

This feature is only available for database rule sets.

When enabled, this feature requires a user (the “approver”) to approve a rule set’s inventory settings before a masking job for that rule set can be executed.

A database masking job will only be allowed to run if its rule set is in the Approved state. If the database rule sets state is either New, Submitted, or Rejected, and the user tries to run the masking job, then the user will receive the following error message: “Attempt to execute job while approval workflow is enabled and the rule set is not approved.”

Enabling Inventory Approval Workflow for an environment

Users can enable the inventory approval workflow for any environment by selecting the checkbox “Enable Approval Workflow (database rule sets only)” at the bottom of the Add, Edit, or Copy environment dialogue boxes.

Workflow stages

NEW
When a user updates any column properties for a database rule set, the approval workflow status will be reset to NEW for that particular rule set. A masking job will not be able to run on any rule set in this status. Users will have to submit these inventories for approval by clicking on the Submit button which appears just above the grid.
SUBMITTED
Once the user modifies any properties in the database inventory and submits it for approval, an Admin user or any user whose role has the Inventory Approval privilege enabled will be able to approve/reject these changes by clicking on the Approve/Reject buttons appearing on the top of the grid. The Approve and Reject buttons will be hidden for the users without this privilege.
APPROVED
If the database rule set status is Approved, masking jobs using this rule set may be executed.
REJECTED
If the database rule set status is Rejected, the user will have to re-modify the inventory properties set to the database columns and submit the inventory again for approval.