Microsoft Purview Data Quality overview (Preview)

Artículo
04/09/2024

Microsoft Purview Data Quality is a comprehensive solution that empowers business domain and data owners to assess and oversee the quality of their data ecosystem, facilitating targeted actions for improvement. In today's AI-driven landscape, the reliability of data directly impacts the accuracy of AI-driven insights and recommendations. Without trustworthy data, there's a risk of eroding trust in AI systems and hindering their adoption.

Poor data quality or incompatible data structures can hamper business processes and decision-making capabilities. Microsoft Purview Data Quality addresses these challenges by offering users the ability to evaluate data quality using no-code/low-code rules, including out-of-the-box (OOB) rules and AI-generated rules. These rules are applied at the column level and aggregated to provide scores at the levels of data assets, data products, and business domains, ensuring end-to-end visibility of data quality within each domain.

Microsoft Purview Data Quality also incorporates AI-powered data profiling capabilities, recommending columns for profiling while allowing human intervention to refine these recommendations. This iterative process not only enhances the accuracy of data profiling but also contributes to the continuous improvement of the underlying AI models.

By applying Microsoft Purview Data Quality, organizations can effectively measure, monitor, and enhance the quality of their data assets, bolstering the reliability of AI-driven insights and fostering confidence in AI-based decision-making processes.

Data quality life cycle

Assign users(s) data quality steward permissions in your data catalog to use all data quality features.
Register and scan a data source in your Microsoft Purview Data Map.
Add your data asset to a data product
Set up a data source connection to prepare your source for data quality assessment.
Configure and run data profiling for an asset in your data source.
1. When profiling is complete, browse the results for each column in the data asset to understand your data's current structure and state.
Set up data quality rules based on the profiling results, and apply them to your data asset.
Configure and run a data quality scan on a data product to assess the quality of all supported assets in the data product.
Review your scan results to evaluate your data product's current data quality.
Repeat steps 5-8 periodically over your data asset's life cycle to ensure it's maintaining quality.
Continually monitor your data quality
1. Review data quality actions to identify and resolve problems.
2. Set data quality notifications to alert you to quality issues.

Supported Data Quality Regions

Microsoft Purview Data Quality is currently supported in the following regions:

Australia East
Canada Central
Brazil South
Central India
Central US
East US
East US 2
France Central
Japan East
North Europe
South Central US
South East Asia
UAE North
UK South
West Central US
West Europe
West US
West US 2

Supported data source types

Azure Data Lake Storage (ADLS Gen2)
- File Types: Delta format
Azure SQL Database
Fabric Lakehouse (delta table)

Currently, Microsoft Purview can only run data quality scans using Managed Identity as authentication option.

Data quality features

Data source connection configuration
- Configure connection to allow Purview DQ SaaS application to have read access to data for Quality scanning and profiling.
- MS Purview uses Managed Identity as an authentication option
Data profiling
- AI enabled data profiling experience
- Industry standard statistical snapshot (distribution, min, max, standard deviation, uniqueness, completeness, duplicate, …)
- Drill down column level profiling measures.
Data quality rules
- Out of box rules to measure six industry standards Data quality dimensions (completeness, consistency, conformity, accuracy, freshness, and uniqueness)
- Custom rules creation features include number of out of the box functions and expression values.
- Auto generated rules with AI integrated experience
Data quality scanning
- Select and assign rules to columns for data quality scan.
- Apply Data freshness rule in the entity / table level to measure the data freshness SLA.
- Scheduling data quality scanning job for time period (hourly, daily, weekly, monthly, etc.)
Data quality job monitoring
- Enable monitoring data quality job status (active, completed, failed, etc.)
- Enable browsing the DQ scanning history.
Data quality scoring
- Data Quality score in rule level (what is the quality score for a rule that applied to a column)
- Data Quality score for Data assets, Data Products, and Business Domains (one business domain can have many data products, one data product can have many data assets, one data asset can have many data columns)
Data quality for critical data elements (CDEs)
- This is one of the key features of Purview Data Quality, it's ability to apply data quality rules to the logical construct of CDEs, which then propagate down to the physical data elements that comprise them. By defining data quality rules at the level of CDEs, organizations can establish specific criteria and thresholds that CDEs must meet to maintain their quality
Data quality alerts
- Configure alerts to notify data owners, data stewards if data quality threshold missed the expectation.
- configure email alias or distribution group to send the notification about data quality issues.
Data quality actions
- Actions center for DQ with actions to address DQ anomaly states including diagnostic queries for DQ steward to zero in on the specific data to fix for each anomaly state.
Data quality managed virtual network
- A virtual network managed by data quality that connects with private endpoints to your Azure data sources.

Next steps

Assign users(s) data quality steward permissions in your data catalog to use all data quality features.
Set up a data source connection to prepare your source for data quality assessment.
Configure and run data profiling for an asset in your data source.