Analytics Engineering, Orchestration, and Graphs: Insights from Trial and Error

Here is the translated blog post in English:

Analytics Engineering, Orchestration, and Graphs: Insights from Trial and Error

Today, I’d like to talk about Analytics Engineering, Orchestration, and Graphs. After going through many trial-and-error experiences, I wanted to take the time to organize my thoughts and perhaps gain some insights in the process.

Why Analytics Engineering?

The reason I became interested in Analytics Engineering is quite simple: I wanted to address two key needs — customers and graphs. As many of you may know, it is rare to load raw data directly into a graph database. Instead, customers often have to process data from OLTP systems before transforming it into a graph format. I see this as one of the main barriers to adopting graph databases, so I started exploring ways to automate or semi-automate this process to reduce the burden.

Of course, this challenge is less significant for naturally occurring network/graph data that doesn’t require separate relationship modeling.

Now, let’s take a closer look at the process. The key challenge is discovering relationships in tables and expressing them in a graph format. This is where analytics comes into play. A crucial tool that helps with this is dbt (Data Build Tool).