Job monitoring

This section describes how users can monitor the progress and completion state of a job.

Monitoring jobs

Once a job has been created and started, you can monitor its progress by navigating to the Monitor > Job Executions or by clicking on the name of the job on any screen. The Job Executions page shows you a list of executed jobs, their progress as well and their current status. To get even more detail on the progress of an individual job, click on the Job Name.

Jobs Queued refers to the number of jobs in the Jobs Queue. When the limit is reached for running the maximum number of jobs and any job is triggered then it moves to Jobs Queue.

Click on View Logs from the Action (…) button to the right of the corresponding execution, if you want to view the logs of completed execution.

The executions on the screen can be filtered or sorted by the various informational fields by clicking on the respective fields. More information on grid filtering and sorting can be found here.

Sortable and filterable columns are Environment, Job Name, Status, Submit Time, Start Time, and Type.

Event status

The following table lists the states of a job/event.

Job status icon	Status	Description
	Cancelled	It appears when a user cancel a running task/execution.
	Failed	It appears when there is an error in the execution of an event in the task/execution.
	Queued	It means the events of the task/execution are yet to start.
	Running	It appears when the event is in progress.
	Succeeded	It means the event is completed successfully. It appears when the event is successful.
	Skipped	It appears when the event is skipped and process moved to the next event of the task/execution.
	Warning (Non-Conformant)	It appears when an event of the task/execution is successful but with warning.

Monitoring a single job

In addition to viewing high-level stats about the status/progress of all your jobs, you can also deep dive into each job to get more details by clicking on the job name from the Job executions grid.

Execution Summary

This section displays more granular information about execution including; Environment name, Connector name, Job Name, Previous run time, Number of tables/files in the job, Number of tables/files masked, Number of tables/files to be masked, Type of job, Rows remaining to mask, Rows masked, Number of streams, etc.

Events

This section displays the Start time, Run Time, and End time of the job and also the sequence in which the events are executed.

The events are executed in the following sequence:

1. Init Execution: The execution has begun.

2. Collecting Job Configurations: Collect the Job details stored in the MDS/DB.

3. Preparing Execution: Create transformation XML for a kettle that includes, pre-script, post-script, create and drop identity XML.

4. Execute Pre-Execution Custom Driver Task: Execute custom drive pre-execution tasks.

5. Start Execution: Starts the masking Job

6. Pre SQL Script: execution prescript if available

7. Post-SQL Script: execution postscript if available

8. Execute Post Execution Custom Driver Task: Execute post-SQL operations from the custom driver

9. Collect Job Information: Collect and store execution information in the database.

10. Execution Finished: The execution is finished and removed from monitoring

Execution Components

In addition to seeing this information about each job, you can look into the status/progress of each table/file defined in the job. Each table/file will be displayed under Execution Components. It includes information such as Time taken for processing, Rows Masked per Minute, Rows Masked, Rows Remaining, Number of columns/fields having Non-conformant data, etc.

Users can maximize this section by using the icon on the top right corner of this section.

It also gives the option for viewing logs for individual tables/files.

• On hovering on the column name, the information about column is displayed in the tooltip.

• The execution components grid supports sorting and filtering on ID, Table Name, Status and Rows Masked columns.

• This grid doesn’t support displaying filter chips after the filter is applied and filters are not persisted in cache.

• More information on grid filtering and sorting can be found here.

Report and Logs

Using the Report and Logs button users can retrieve Job and Inventory Reports in PDF formats. From this screen, the user can also view logs for this execution by clicking on “Execution Log“.

Profile Results

This section comes for Profiling jobs. It displays the sensitive data findings on a table-column by table-column or file-field by file-field basis.

• Profile results grid supports sorting and filtering on all the columns.

• This grid doesn’t support displaying filter chips after the filter is applied and filters are not persisted in cache.

• More information on grid filtering and sorting can be found here.

Displaying non-conformant data

When a masking job encounters non-conformant data, the job will either Fail or Succeed with a warning, depending on how the algorithms associated with the ruleset for the job are configured. As depicted in the screenshot, the non-conformant data can be accessed via the Execution Components on the Monitor page for the job, which can be accessed by clicking on the Job name from the Monitor Grid. In the Execution Summary section, a summary of the Tables with Nonconforming Data and Columns with Nonconforming Data is reported. Further details on the non-conformant data encountered can be accessed by clicking the “View Logs“ on each table or file listed in the Execution Components grid.

The non-conformant data events are displayed followed by the masking log for the table or file. If there were no non-conformant data events, "No Events to Display" is displayed, otherwise, for each type of non-conformant data, a row will be displayed with the following information:

• Event type: Either JOB_ABORTED or UNMASKED_DATA if the job was not aborted.

• Cause: Always PATTERN_MATCH_FAILURE.

• Approximate Row Count: Approximate number of rows with non-conformant data (at least within an order of magnitude).

• Description: Details the name of the column or field with non-conformant data

Interpreting samples of non-conformant data patterns

Each character in the non-conformant data is sampled per its Unicode Character Property.

• N for digits

• L for letters

• M for marks

• P for punctuation

• S for symbols

• Z for separator

• O for other

• U for unknown

Tracking Non-conformant Data

Please note that actual personal data is never displayed, only the samples (a.k.a. patterns) of non-conformant data are displayed on this page

Using the database-specific SQL query, it is possible to locate data corresponding to the non-conformant data sample. The table and column names can be found on the table report. In the example above, the table name is "testdata_EXCEPTION" and the column name is "EXCEPTION_00".

Note

The pattern might be not an exact representation of the data in the field, but a part of the data. For instance, white spaces at the beginning or at the end of the data might be truncated.

Oracle example

Below are the Oracle character classes, used in the regular expression:

For the LLLLL sample in the example above, the Oracle DB SQL query would look like this:

SELECT EXCEPTION_00 FROM testdata_EXCEPTION WHERE regexp_like(EXCEPTION_00, '[[:alpha:]]{5}');

For the LLLLZLLLZLLLL sample, the Oracle DB SQL query would look like:

SELECT EXCEPTION_00 FROM testdata_EXCEPTION WHERE regexp_like(EXCEPTION_00, '[[:alpha:]]{4}[[:space:]][[:alpha:]]{3}[[:space:]][[:alpha:]]{4}');

Limitation for the multi-column extensible algorithm

If a Non-conformant data pattern is encountered - it is displayed for all the masked columns of the MC Algorithm, not only for the column where that event has occurred. In that case, the manual analysis of the error message will be required to find the actual column(s) with the Non-conformant data.