Cleanse and Govern

Data Governance 

Data Governance (DG) is the overall management of the availability, usability, integrity and security of data used in an enterprise. 

Data Governance forms the basis for company-wide data management and makes the efficient use of trustworthy data possible. The efficient management of data is an important task that requires centralized control mechanisms. MDO helps to achieve this by its enriched set of data governing mechanism. You can use and apply the functionalities of MDO in two possible ways with some different set of rules. These are: 

  • Active Governance: Works on the data which is entered in the system after applying the rule. 
  • Passive Governance: Works on the data which is already stored in the system. 

Data Workflow 

MDO has a well-defined governing process for data which is known as Data Workflow. Data Workflow provides customers with the ability to customize the flow of data within an organization to meet their requirement. It gives an organization the option to determine who owns and is responsible for the data at every step of the workflow. 

Data Workflow provides the customer with a range of functionalities to implement and customize the workflow according to an organization’s need. For examples an organization can define: 

  1. The responsible person at each step. 
  2. What a person can see and have access to change. 
  3. On which basis the request should go to which person. 
  4. Step by step assignment 
  5. Which user(s) have the authority to approve or reject a request; if approved it goes to the next approver or step, or by rejection it goes back to the previous step, etc. 

Note: Users can configure these features in a module by navigating to the Advance -> workflow section and for other rules by going into the Business rule -> schema. 

Below is the list of the rules/features which works on different type of governance. 

These rules can be found and implemented by going into: 

  1. Log in with administrator credentials. 
  2. Click on the Settings icon at top right corner on the Home page. 
  3. From the left menu select Modules > Select the module for which the rules need to be defined.  
  4. Click on Business Rules. Select the rule you want to apply, along with some additional features that will help to manage the data easily. E.g., Defaults, number settings, etc. 

Below is an overview of the type of rules that can be implemented at different levels of governance. 

Rule Name  Active Governance  Passive Governance 
Dependency  Yes, via custom dependency configuration.  Yes 
API    Yes 
Duplicacy Check  Yes  Yes 
Metadata Rule  Yes, via Metadata Conf.  Yes 
Missing Rule  Yes, via User Defined Rule eg. Template  Yes 
User Define  Yes, via User Defined JavaScript  Yes 
Regex  Yes, via User Defined JavaScript  Yes 
Look Up Table    Yes 
Query Rule    Yes 

 


Data Cleansing
 

Data cleansing is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database. It refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data. The actual process of data cleansing may involve validating and correcting values against a known list of entities. 

In MDO, customers have a variety of options/features by which they can ensure that their data is transformed in the way they want. 

The features provided by MDO data cleansing include: 

  • Validity 
  • Accuracy 
  • Completeness 
  • Consistency 
  • Uniformity 

Customers can also perform the following actions using MDO: 

  • Data auditing 
  • Workflow specification 
  • Workflow execution 
  • Post-processing and controlling 
  • Parsing 
  • Data transformation 
  • Duplicate elimination 

The data can be cleaned and transformed in the standard form by using the enriched set of data cleansing rules provided in MDO. Some examples of these rules include: 

  • Defining the length or type of data that needs to be inserted by the user while defining the fields. 
  • Writing custom codes to validate or restrict the data.  
  • Use predefined field types to validate data such as contact number, email id, etc. 

To access the list of predefined rules provided in MDO, follow these steps: 

  1. Log in with administrator credentials. 
  2. Click on the Settings icon at top right corner on the Home page. 
  3. From the left menu select Modules > Select the module for which the rules need to be defined. 
  4. Click on Business Rules -> Schema.  
  5. Click Add New to Create a new rule. Fill in the basic details as required.  
  6. Select the created schema click Add Business RuleSelect the specific rule you want to add. 
  7. Add the details and business logic and click Activate. 
  8. The rule will be assigned. 

The standard schema rules provided are:  

  • Dependency: This rule establishes a dependency relationship between the source and target fields. 
  • API: This rule is used to define any complex custom logic to validate/transform target fields using the API framework. 
  • Duplicacy Check: This rule helps to identify the duplicate records based on matching field combinations. 
  • Metadata Rule: This rule is to check metadata properties like data type, length, predefined formats etc. 
  • Missing Rule: This rule helps identify the fields which do not contain any values. 
  • Regex: This rule helps to check the accuracy of field value through regular expression (Regex). 
  • User Defined Rule: Define readable conditions to validate/transform the field values. 
  • Lookup Table: This rule helps to validate/transform target values based on mapping fields defined in the Lookup tables. 
  • Query Rule: This rule helps to validate values based on mapping fields defined in the Query. 

All the reports on data quality and validation can be seen on the data quality workbench provided by MDO. 

It provides a deep insight of the gap that exist in the current data verse what is expected as per the defined rules. It has an enriched and interactive UI with lots of additional features to analyze data.