Product Architecture
Download
Focus Mode
Font Size
Tencent Cloud DataBuddy adopts an integrated product architecture featuring Agent-native design, Data + AI integration, and full-modality support. The entire product integrates three layers from top to bottom: the business delivery layer (Agent + visual analytics), the capability platform layer (data assets, semantics, and governance), and the engine and storage layer. It is designed around a full-cycle closed loop for enterprise data, covering the stages from "ingestion → warehousing → semanticization → consumption → governance".
Architecture Overview
Note:
The diagram above illustrates DataBuddy's end-to-end core workflow: multimodal data ingestion into the data lakehouse → full-modality asset management → unified semantic construction → AI-Ready data knowledge → consumption via Agent or visual analytics. This entire process is powered by the DataBuddy Harness framework for foundational Agent capabilities, safeguarded horizontally by the Data Governance Agent, and provided with a unified catalog view by the Unified Metadata TC Catalog (Tencent Cloud Catalog, Tencent Cloud Unified Metadata Catalog). It ultimately runs on a managed data lakehouse and compute engine.
I. Data Source Access Scope
Tencent Cloud DataBuddy supports the ingestion of various data sources, both internal and external to the enterprise, covering the following types:
Data Warehouses
On-premises Systems
SaaS Applications
Machine & Application Logs
Application Events
Mobile & IoT Data
Note:
It supports a wide range of data sources, both domestic and international, including relational databases, data warehouses, lakehouses, NoSQL, KV, message queues, streams, files, COS, APIs, and multimodal data. It offers five major ingestion solutions: offline synchronization, real-time synchronization, file-to-Table, file-to-Volume, and batch-to-Volume.
II. Business Delivery Layer: Agent-Native, Oriented Toward "Conversation as Delivery"
Tencent Cloud DataBuddy follows the product logic of "Agent-first, visualization-assisted": code, tasks, semantic models, and dashboards generated by the Agent are all persisted in the unified metadata TC Catalog, where they can be further orchestrated programmatically or graphically via GUI or Notebook.
Three Built-In Agents
Scenario | Delivery Agent | Key Capabilities |
Data engineering | Data Engineering Agent | Data development, PL (Pipeline, data pipeline) workflow construction and configuration, intelligent Ops and system self-healing |
Data analysis | Data Analytics Agent | Natural language data query, anomaly detection, root cause analysis, analysis reports, intelligent visual reports |
Data governance | Data Governance Agent | AI-based automated governance of defined quality rules, data warehouse quality diagnosis, metadata completion, asset inventory, and AI-based identification of sensitive fields |
Visual Analysis (DashBoard)
DashBoard serves as the dedicated consumption entry point for data analytics, sourcing data directly from AI-Ready Data Knowledge:
It hosts dashboards, reports, and exploration results generated by the Agent.
It enables business users to self-service create and adjust dashboards on a graphical interface.
It enables dual-channel collaboration with the Analysis Agent: after the Agent automatically generates a dashboard, you can perform secondary editing within DashBoard.
III. Tencent Cloud DataBuddy Harness System: Agent-Native Foundation
The Harness framework serves as the core foundation for the three major Agents to stably achieve "conversation as delivery". Spanning the three business lines of engineering, analytics, and governance, it uniformly provides the following capabilities:
Knowledge Base: Enterprise-level knowledge injection, encompassing business ontologies, data assets, SOPs (Standard Operating Procedures), and more.
Context: Multi-turn conversation context management and cross-session state persistence.
Memory: Persistent long-term memory for the Agent, accumulating user preferences and historical decisions.
Big Data Domain Skills and Tools: It comes with built-in domain-specific skills and toolkits for scenarios such as data engineering, semantic modeling, governance, and analytics.
Agent Security: It provides bidirectional Guardrails (input-side prompt injection detection + output-side result protection), strong OBO (On Behalf Of) permission binding, high-risk SQL interception, and intent verification, covering all risks in the OWASP LLM Top 10.
Agent Evaluation: Multi-dimensional evaluation covering Skill and Tool invocation trace tracking, task success rate, Token consumption, and semantic fidelity.
IV. Capability Platform Layer
4.1 Multimodal Data Ingestion into Lakehouses and Warehouses
Tencent Cloud DataBuddy performs multi-layer warehouse construction within the lakehouse, forming a standard data warehouse layered structure:
ODS (Operational Data Store): It hosts the raw details after they are ingested into the data lake.
DWD (Data Warehouse Detail): Data cleansing and standardization are performed at this layer.
DWS (Data Warehouse Summary): It constructs topic-based summary wide tables.
DIM (Dimension): It extracts common dimensions for reuse by upper layers.
ADS (Application Data Store): It directly serves analytics, reporting, and Agent consumption.
Multimodal Support: Structured data is stored in Tables, while unstructured and semi-structured data (images, videos, documents, audio, logs) is stored in Volumes. A single warehouse construction and layering framework simultaneously covers both.
AI-Powered: The Data Engineering Agent supports natural language-based creation of data ingestion and warehouse construction tasks, AI-powered intelligent Q&A, and AI-driven root cause analysis for task exceptions.
4.2 Full-Modal Asset Management
Tencent Cloud DataBuddy provides a unified multimodal asset view through the TC Catalog, enabling structured, unstructured, model, and external image data to share a common set of metadata, permissions, and lineage.
Asset type | Description |
Table Catalog | Tables, views, and functions (structured data) |
Volume Catalog | Logs, images, PDFs, audio, video, and other unstructured data |
Model Catalog | Machine learning models, version management, and lineage |
External Catalog | Mirror of external data sources (queryable without ETL) |
4.3 Unified Semantic Construction (Unity Semantics)
The Unified Semantic Layer is key to making DataBuddy "AI-Ready" - it eliminates definitional ambiguities through standardized ontology modeling and a system of metrics and dimensions.
Ontology Modeling (Business Ontology Modeling): It standardizes entities, attributes, and business logic to construct a business ontology model.
Metrics and Dimensions: The enterprise-grade "Semantic Knowledge Graph" unifies metric definitions and dimension definitions.
Relationship: It encompasses cross-domain entity associations and attribute mappings.
Semantic Governance: It works in conjunction with the Data Governance Agent to achieve definition verification and semantic iterability.
4.4 AI-Ready Data Knowledge
The outputs of semantic construction - ontology models, metrics, dimensions, semantic models, and datasets - are packaged as AI-Ready data knowledge, serving as the shared data provisioning layer for both Agents and visual analytics.
Provide data to the Data Analytics Agent: This drives conversational consumption such as natural language querying, intelligent reporting, and attribution analysis.
Provide data to the DashBoard: This serves as the semantic source for visual analytics, preventing charts from becoming disconnected from actual business definitions.
4.5 Data Governance Agent
The Data Governance Agent operates horizontally across three key capability domains: multimodal data ingestion into data lakes/warehouses, full-modal asset management, and unified semantic construction. It achieves the transition from "manual governance to AI-powered intelligent guardianship" through agentification. Core features:
AI-Driven Governance for Quality Rule Definition: Users define rules using natural language, and the Agent then completes the rule details and automatically associates them with the data pipeline.
Data Warehouse Quality Diagnosis: It supports three modes (Snapshot, Time Series, Inference) and utilizes drift metrics such as KS, PSI, Wasserstein, Chi-Square, and JS Divergence.
Metadata Completion: It automatically completes descriptions at the Table and Column levels, business definitions, and Owner information.
Asset Inventory and AI-Powered Sensitive Field Identification: It identifies sensitive fields such as PII (Personally Identifiable Information), financial, and identity data in batches, triggers data classification and masking, and performs an inventory of all assets.
4.6 Unified Metadata TC Catalog
TC Catalog serves as the "metadata hub" for the entire platform: Table Catalog, Volume Catalog, Model Catalog, and External Catalog share a unified set of catalogs, permissions, lineage, and Tags, supporting Agents and visual analytics upstream and connecting to engines and storage downstream.
V. Engine and Storage Layer
5.1 Computing Engine
Engine | Location | Typical Scenario |
Batch computing engine | Parallel processing of large-scale data | Batch ETL, data warehouse construction, model training, large-scale data processing |
OLAP (Online Analytical Processing) analytical engine (Planned) | Interactive sub-second response | Interactive analysis, report acceleration, DashBoard data provisioning |
5.2 Lakehouse Storage TCLake
Managed Storage + Read/Write Request Billing: Customers are exempt from Ops.
Full-Modal: It simultaneously supports both Tables (structured) and Volumes (images, videos, documents, audio, logs).
Single Dataset, Shared by Multiple Engines: Batch and OLAP engines read from and write to the same source, eliminating redundant data movement.
5.3 Customer External Data Sources
Analyze customer's existing data sources (such as MySQL, PostgreSQL) without data movement by leveraging External Catalog and federated query capabilities to achieve analysis without data movement.
VI. Core Architectural Features
1. Agent-Native: The capabilities in knowledge, context, memory, Skill, security, and evaluation provided by the three built-in Agents + Harness framework make "conversation as delivery" the default delivery mode.
2. Data + AI Integration: The data platform and AI platform share a unified set of metadata, lineage, permissions, scheduling, and storage, eliminating the migration and governance costs associated with maintaining "two separate systems".
3. Full-Modal Support: It provides a unified framework for Tables, Volumes, Models, and External sources, covering structured, semi-structured, and unstructured data, Model assets, and external images.
4. Unified Semantic-Driven: Unifying business definitions through ontology modeling and a metrics/dimensions system is the key foundation for the stable delivery of Agents and Dashboards.
5. AI-Ready Data Knowledge Sharing: Semantic outputs are packaged into a data provisioning layer, which concurrently serves both Agents and visual analytics, eliminating "analysis definition drift".
6. Integrated Governance: The Data Governance Agent spans across data lake/warehouse ingestion, asset management, and semantic construction. By leveraging the Agent security capabilities of Harness, it ensures that deep AI involvement in enterprise data remains controllable.
7. Open Connectivity: It can both host and ingest various external data sources, and perform federated analysis on customers' existing MySQL, PostgreSQL, and other databases without data movement via External Catalog.
Help and Support
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback