Data Quality

Data Quality includes the following business rule types: 

  • Missing rule -This rule helps identify the fields which do not contain any values.  
  • Metadata rule -This rule is to check metadata properties like data type, length, predefined formats etc.  
  • User-defined rule -This rule permits user to define readable conditions using in built string, math and special operators. Multiple conditions can be grouped together using the AND/OR blocks in the rule. 
  • Regex rule -This rule helps to check the accuracy of a field value through regular expression (Regex). Standard functions like email, phone number validations etc. are available within the regex rule for users to select and skip the process of building the regex expression from scratch. 
  • Lookup rule– The Lookup Table rule is used to validate or transform target values based on the source values defined in the table. This rule is used in data migration and transformation projects.   

Note- You can apply all these rules together in a single schema. While running data quality rules, you cannot apply duplicate rules and MRO rules to the same schema.   

Data quality view- 

It is used to display the result set of business rules such as missing rule, metadata rule, user-defined rule, and regex rule. The result screen for this view produces a spread-sheet-like table with records falling into falling in to or success tab based on the validations applied in the business rule. Any view produced by a business rule in MDO is a spreadsheet-like table that allows the user to work with data the way they would work with data in Excel or Google sheets. That is why the user should be able to have 3 options: 

DQ view basically covers all the business rules which have the records processed in error or success tab based on BR validations. 

We cannot configure duplicate check and classification check rules with DQ rules. 

List of Data Quality rules: 

  • Missing rule 
  • Metadata rule 
  • Regex rule 
  • User-defined rule 
  • Lookup rule 
  • Export/import/integrate the view with Microsoft Excel; 
  • Export/import/integrate the view with Google Sheets; 
  • Work with data directly in DIW (MDO) as a default/fallback option; 

Nested/flat structure (DQ view) 

Users should be able to view the data of the interconnected data subsets. A data set in MDO can have the following types of data subsets: 

Header data – group of fields that store data 

Grid data – group of fields displayed in a tabular format. used for storing multiple values for each field as a new line item. 

Data subset – links to data set using a primary key 

Grid data – group of fields displayed in a tabular format. used for storing multiple values for each field as a new line item. 

Data sub-subset – links to dataset & subset using a primary key 

Header data – group of fields that store data 

Grid data – group of fields displayed in a tabular format. used for storing multiple values for each field as a new line item. 

In MDO, one data set can have multiple subsets underneath it, linked via a set of primary key(s). As the parent dataset fields can be used to derive data decisions, the data subset view must have the data subset fields appended towards the right of the parent dataset fields. The parent dataset repeats itself with every related child dataset. 

In Data quality, the user has the flexibility to navigate through structural hierarchies in datasets.