The art of data engineering
As the data space has increased, data engineering has emerged as a separate and related role that works together with data scientists. Usually, data scientists focus on finding new insights from a data set, while data engineers are concerned with the production readiness of that data.
In this talk, I’ll show you how to gather and collect the huge amount of data, store it, do batch processing or real-time processing on it, and how to build a data pipeline using Airflow for processing billions of records per table.
Also, we will discuss what is big data, and why it’s important to be able to process it so quick.
- Python developer in the day, Go developer (gopher) under the hood. Big fan of full-text search and graph databases
- Top 3% of Freelance Talent (Toptal is an exclusive network of the top freelance software developers, designers, and finance experts in the world)
- Contributed in different python/go open source projects: pyhelm, aiohttp-swagger, mezzanine; chalice, requests, aiohttp tutorial; sendgrid-python and sendgrid-django; OpenAPI v3 specification, fix Go docs
- Blogger, Twitter, LinkedIn, GitHub