The art of data engineering

Talk video

Talk presentation

As the data space has increased, data engineering has emerged as a separate and related role that works together with data scientists. Usually, data scientists focus on finding new insights from a data set, while data engineers are concerned with the production readiness of that data.

In this talk, I’ll show you how to gather and collect the huge amount of data, store it, do batch processing or real-time processing on it, and how to build a data pipeline using Airflow for processing billions of records per table.

Also, we will discuss what is big data, and why it’s important to be able to process it so quick.