Data Engineering Agent is Buddy's AI digital employee in the "Data Development" mode. You only need to describe your requirements in natural language, and Agent automatically completes the entire data engineering pipeline, from requirement understanding to task deployment.
AI-Driven Full Process
Traditional data engineering requires engineers to manually sort out requirements, write code line by line, configure tasks one by one, and troubleshoot issues manually, making the entire process time-consuming, labor-intensive, slow to respond, and error-prone. Data Engineering Agent transforms all of this into an AI-driven automated pipeline:
Traditional approach | Buddy Data Engineering Agent |
Manually sort out requirements | Understand requirements with AI |
Manual coding | Generate data warehouse solution |
Configure tasks one by one | Automatically generate code |
Manual Ops troubleshooting | Intelligently orchestrate workflows |
Time-consuming, labor-intensive, error-prone, and inefficient | Efficient, intelligent, fast-responding, and highly accurate |
Data Engineering Agent covers six core capabilities, forming a complete closed loop:
Step | Capability | Description |
1 | Generate data warehouse solution from requirements | Understand business requirements and automatically generate a data warehouse solution that fits the business |
2 | Intelligent data ingestion | Automatically identify data sources, intelligently map fields, and quickly complete data ingestion |
3 | Code generation | Automatically generate high-quality SQL / script code based on the solution and rules |
4 | Automated workflow development | Automatically orchestrate task dependencies and scheduling, and generate a complete workflow with one click |
5 | Data warehouse structure analysis | Automatically parse the existing data warehouse structure, identify layering relationships and modeling issues, and provide optimization suggestions. |
6 | Task diagnosis and repair | Monitor tasks in real time, intelligently diagnose issues, and provide repair suggestions. |
The entire process is connected end to end by AI. You only need to confirm at key nodes, without manually switching between multiple modules.
Differences from Data Analysis Agent
Buddy switches between Data Development mode (Data Engineering Agent) and Data Analysis mode through the "Scenario Mode" switcher at the top. The artifacts, knowledge bases, and output scopes produced by the two modes are isolated from each other:
Dimension | Data Engineering Agent (This Document) | Data Analysis Agent |
Primary audience | Data Engineers / Data Developers | Data Analysts / Business Users |
Typical artifacts | SQL files, Python files, notebooks, workflows, and data ingestion tasks | Markdown reports, HTML dashboards, query result tables, charts |
Knowledge base ownership | Workspace-level knowledge base | Analysis space-level knowledge base (bound to the analysis space) |
Prerequisites
Before using Data Engineering Agent, ensure the following:
Buddy has been enabled for the workspace.
The current account has the required permissions on the resources involved in operations within the target Workspace, including data sources, Catalog, Studio folders, compute resources, and workflows. Agent verifies the actual permissions of this account when executing deployment operations, and provides an insufficient permission prompt if the account lacks the required permissions.
Scenario 1: Intelligent Data Ingestion
Feature Description
Intelligent Data Ingestion supports creating real-time or offline synchronization tasks through natural language descriptions, covering the following scenarios:
Real-time whole-database multi-table ingestion: synchronizes an entire database (or a set of tables matched by a regular expression) to Catalog in real time.
Real-time sharded database and table ingestion: merges tables with the same name from multiple sharded databases into a single target table.
Offline single-table ingestion: synchronizes a single source table to a target table on a daily / hourly schedule.
Offline batch single-table ingestion: creates offline synchronization tasks for multiple source tables at once in batches.
Buddy automatically identifies the ingestion method, generates field mappings, recommends the write mode and scheduling time, and waits for your confirmation at key nodes before deployment. You only need to provide the data source name, source database/table scope, and target Catalog. The remaining configurations, including task name, synchronization policy, incremental field, write mode, and field mapping, are intelligently recommended by AI.
Usage Methods
Describe your requirements in natural language in the Buddy input box. You only need to provide the data source name, source database/table scope, and target Catalog:
Input Example | Effect |
Sync the orders table in the production MySQL environment to lakehouse_catalog on a daily basis. | Create an offline single-table synchronization task. |
Synchronize all tables starting with order_ in mysql_prod_01 to ods_catalog in real time. | Create a real-time whole-database multi-table synchronization task. |
Merge and synchronize the order tables from the 8 sharded databases order_db_00 to order_db_07 into order_all. | Create a real-time sharded database and table synchronization task. |
Sync all tables in the ods schema to lakehouse_catalog on a daily basis. | Create an offline batch single-table synchronization task. |
Scenario 2: Code Assistance
Feature Description
Code Assistant supports performing full code lifecycle operations within Studio through natural language:
Code Generation: generates SQL / Python / Notebook files from natural language descriptions and automatically inserts them into Studio.
Code Explanation: parses the current code using natural language to explain its core logic and business meaning.
Code Correction and Dialect Conversion: automatically corrects syntax errors and converts dialect differences such as :param ↔ ${param} and dbutils → dlcutils.
Trial Run + Automatic Error Fixing: automatically performs a trial run after the script is written. If the run fails, it analyzes the error logs and attempts to fix and retry.
Buddy deeply integrates with the Studio file tree, editor, and run results panel. When a file is modified, a Diff is displayed, and you can choose to accept or reject it.
Usage Methods
Describe your requirements in natural language in the Buddy or Studio sidebar:
Input Example | Effect |
Write a SQL statement to calculate the GMV by day and channel from ods_orders. | Generate SQL File. |
Create a Python file to read the dws_user_daily table by using dlcutils. | Generate Python File. |
Explain what this code does. | Code Explanation |
Convert this Databricks SQL to DataBuddy dialect. | Dialect Conversion |
Run this SQL. | Trial run with automatic repair and retry upon failure. |
Scenario 3: Workflow Orchestration
Feature Description
Workflow orchestration supports assembling scripts in Studio into workflows through natural language:
Create Task from Code: creates a task node based on SQL / Python / Notebook files.
Automatic DAG Generation: parses the FROM / JOIN relationships in code and automatically generates task dependencies and the DAG topology.
DAG Validation: detects circular dependencies, identifies unreasonable dependencies, and checks for scheduling conflicts.
Scheduling Configuration: recommends scheduling cycles, scheduling times, dependency policies, and retry policies based on business time requirements.
Workflow CRUD: modify nodes, configurations, and dependencies using natural language.
Usage Methods
Describe your orchestration requirements in natural language in the Buddy input box:
Input Example | Effect |
String dwd_orders.sql, dws_order_daily.sql, and ads_channel_gmv.sql into a workflow and schedule it at 03:00 daily. | Create workflow + configure scheduling. |
Add a push task after ads_channel_gmv. | Add node + configure dependency. |
Change the scheduling of the entire workflow to once every hour. | Modify scheduling cycle. |
Scenario 4: Intelligent Diagnosis
Feature Description
Intelligent diagnosis covers anomaly locating and repair suggestions across the entire data engineering pipeline, covering the following scopes:
Workflow Task Diagnosis: when a task in a workflow encounters failure / waiting for resources / upstream failure, the system automatically performs root cause analysis, impact assessment, and failure link tracing, and provides repair suggestions (rerun / terminate / rerun after modification).
In-Studio Code Run Diagnosis: analyzes error logs and provides fix suggestions when running SQL / Python / Notebook in Studio fails.
Real-Time Ingestion Task Diagnosis: locates root causes and provides handling suggestions when real-time data synchronization tasks encounter issues such as latency, interrupted streams, or field anomalies.
Offline Ingestion Task Diagnosis: locates root causes and provides handling suggestions when offline synchronization tasks encounter failures, timeouts, or DDL changes.
Usage Methods
In the workflow run details, hover over a failed node and click AI Diagnosis to trigger it. Alternatively, click AI Diagnosis in the Studio run panel, or describe the issue in natural language in Buddy:
Input Example | Effect |
The xx workflow task failed. Help me identify the cause. | Root cause analysis + impact scope assessment |
The SQL for the xx task failed to run. Help me check it. | Code execution diagnostics in Studio |
The xx real-time sync task is severely delayed. Help me troubleshoot it. | Real-time data ingestion task diagnostics |
The xx offline sync task reported an error. Help me identify the cause. | Offline data ingestion task diagnostics |
End-to-End Scenario Example: Building a Data Warehouse from Scratch
The following complete scenario demonstrates how Data Engineering Agent starts from a single requirement and completes the entire data warehouse construction process end to end.
Input example
I am a data engineer. The business team has added an external PostgreSQL data source and wants to build a business analysis data warehouse based on this data.
Business users want to view sales performance, seasonal sales trends, and product category analysis on a daily basis, including metrics such as GMV, order volume, revenue, average order value, category sales proportion, and top-selling category rankings.
The data source name is XX, and the source tables are located under the XX schema. I want to synchronize all source tables under this schema to the XX catalog, and complete data warehouse design and construction based on these source tables. The data is updated at 10:00 every morning.
Agent Automatic Execution Process
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Data Source │ │ View Tables │ │ Build Plan │ │ Data Ingestion│ │ Code Development │ │ Workflow │
│ Connect │ ─▶ │ Structure │ ─▶ │ Generate │ ─▶ │ Configure │ ─▶ │ Generate │ ─▶ │ Orchestrate │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
Connect target Identify target ODS/DWD/ Create sync task ETL Dependency DAG
Data source Under schema DWS/ADS Code generation + Scheduling configuration (daily at 10:00)
All table structures Layering plan Code validation & alarm configuration, and so on