DataExpert-io/data-engineer-handbook: This is a repo with links to everything you'd ever want to learn about data engineering

This is a repo with links to everything you'd ever want to learn about data engineering - DataExpert-io/data-engineer-handbook

Summary

The Data Engineering Handbook is a curated directory of learning resources, technical references, tools, communities, and career guidance for data engineers. Rather than providing a runnable data platform, it helps engineers find material for building pipelines, evaluating technologies, and developing data-engineering skills.

What it is useful for

The handbook is useful when an engineer needs direction across the data stack. Its resource sections cover orchestration tools such as Airflow, Dagster, Prefect, and Kestra; cloud and lakehouse platforms such as Databricks, Delta Lake, Apache Iceberg, and Apache Polaris; and warehouses including Snowflake and Firebolt.

It also points to specific engineering concerns:

• Data integration: Airbyte, Fivetran, dlt, Meltano, and Sling.
• Data quality: dbt, Great Expectations, Soda, DQOps, and Coalesce.
• Analytics and visualization: Metabase, Tableau, Power BI, Apache Superset, and Evidence.
• Modern OLAP and real-time systems: Apache Druid, ClickHouse, Apache Pinot, DuckDB, RisingWave, and Streamdal.
• Data lineage: OpenLineage.
• Background reading: company engineering blogs, whitepapers, podcasts, and books.

Engineers can also use the interviews, books, and communities sections to prepare for job interviews or find ongoing discussion. The projects.md section provides hands-on examples, while the beginner and intermediate bootcamp folders include introductions and software checklists.

How engineers can use it

Start with the repository’s linked 2024 breaking into data engineering roadmap if the topic is new to you. For structured course material, open the relevant beginner-bootcamp/introduction.md or intermediate-bootcamp/introduction.md, then consult its matching software.md file. The README does not document an installation or runtime setup for the handbook itself, because it is primarily a reference collection; use its included instructions for any external course or project.

For real work, treat the handbook as a research index. If designing a pipeline, compare the orchestration, integration, storage, and quality sections against your requirements. For example, an engineer evaluating a lakehouse workflow can review Delta Lake and Apache Iceberg entries, then use the company blogs and whitepapers for deeper reading. If preparing for interviews, combine the interviews page with a relevant project and the recommended books.

One limitation is that some links are tied to dated or time-sensitive material, including the 2024 roadmap and scheduled bootcamps. Check that external resources are still current before building a learning plan around them.