Automating Data Quality Assurance Using Machine Learning in ETL Pipelines

Raghavender Maddali

doi:10.5281/zenodo.15107533

Automating Data Quality Assurance Using Machine Learning in ETL Pipelines

Author(s)	Raghavender Maddali
Country	India
Abstract	The Rising data processing pipeline complexity and size necessitate the use of secure automated means to ensure data quality during Extract, Transform, Load (ETL) processes. This article presents a machine learning framework that is capable of automating data quality assurance in ETL pipelines using anomaly detection, predictive modeling, and self-healing. The framework leverages machine learning algorithms to detect inconsistencies, predict potential data errors, and autonomously correct issues, enhancing data integrity, accuracy, and consistency in real-time processing. By continuously monitoring data quality, the system reduces manual intervention, minimizes operational costs, and improves decision-making reliability. The framework is adaptable across various industries, including healthcare, finance, and manufacturing, where high-quality data is essential for business intelligence and analytics. Experimental findings reveal that it successfully minimizes data errors and strengthens the ETL pipeline significantly. The suggested approach offers a scalable and smart solution to ensure high-quality data in a changing environment of data.
Keywords	ETL automation, data quality assurance, machine learning, anomaly detection, predictive modeling, self-healing mechanisms, real-time data processing, data integrity.
Field	Computer > Data / Information
Published In	Volume 2, Issue 6, June 2021
Published On	2021-06-08
Cite This	Automating Data Quality Assurance Using Machine Learning in ETL Pipelines - Raghavender Maddali - IJLRP Volume 2, Issue 6, June 2021. DOI 10.5281/zenodo.15107533
DOI	https://doi.org/10.5281/zenodo.15107533
Short DOI	https://doi.org/g8986j

View / Download PDF File

About IJLRP Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijlrp.com

International Journal of Leading Research Publication

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Automating Data Quality Assurance Using Machine Learning in ETL Pipelines

Share this