Topic 2

Lesson 2. Data Formats, Structures, and Preparation

Objective: formation of students' complex of knowledge, analytical abilities and practical engineering skills in the field of choosing data storage formats, implementing ETL architectural processes, data normalization and cleaning (Data Cleaning), as well as automation of engineering scenarios for preparing information using the No-Code / Low-Code tool Power Query. The student must master the classification of data formats (structured, semi-structured, unstructured), deeply understand the physical, syntactic and comparative specifics of CSV, XLSX, JSON, XML formats, as well as grasp the fundamental concept of ETL (Extract, Transform, Load) / ELT and the GIGO rule (Garbage In, Garbage Out). During the training, special attention is paid to the practical mastering of methods for detecting and eliminating data defects (UTF-8 encoding, gap processing, deduplication, text noise cleaning), mastering the principles of Tidy Data normalization and atomicity of values, as well as studying the procedures for transforming table forms (Pivoting and Unpivoting). An important practical result is the development of the ability to design reproducible deterministic data processing algorithms in the Power Query environment: configure connectors to heterogeneous sources, perform merges (Merge / Append), combine files from folders, and read/edit automatically generated code in the declarative functional language M (Advanced Editor) to create automated data preparation pipelines that are updated in one click without damaging the original source files.