DataFusion & SQL Analytics

Flow-Like embeds Apache DataFusion as a query layer. A workflow creates a session, registers one or more sources as named tables, then runs SQL across those tables.

Session model

Step	Purpose
Create session	Allocate one query context for the run
Register sources	Give files, databases, or lake tables stable SQL names
Inspect	List tables and confirm schemas
Query	Filter, join, aggregate, or window the registered data
Deliver	Send the result to another node, table, chart, file, or model tool

Start with Create DataFusion Session and pass the same session value to every registration and query node that should share tables.

Register files and in-memory data

Source	Node
CSV file	Mount CSV
JSON or NDJSON file	Mount JSON
Parquet file	Mount Parquet
Lance table	Register Lance Table
CSVTable value already in the workflow	Register Table

Choose a SQL-safe table name and keep it stable across the query. Validate file schemas before assuming a column type.

Register databases

The generated catalog includes:

Database	Node
PostgreSQL	Register PostgreSQL
MySQL	Register MySQL
SQLite	Register SQLite
DuckDB	Register DuckDB
ClickHouse	Register ClickHouse
Oracle	Register Oracle
BigQuery	Register BigQuery
FlightSQL	Register FlightSQL
Athena	Register Athena Table

Store credentials in secrets or provider connections. Use a read-only account for analytical workflows unless the board explicitly requires writes elsewhere.

Register data-lake tables

Format	Nodes
Delta Lake	Register Delta Table, Delta Table Info, Delta Time Travel
Apache Iceberg	Register Iceberg Table, Iceberg Table Info, Iceberg Time Travel
Hive-partitioned Parquet	Register Hive Parquet
Partitioned JSON	Register Partitioned JSON

Time-travel nodes are useful for reproducible analysis. Record the selected table version or snapshot with the analysis result.

For Athena results stored in S3, Mount Athena S3 Results can make the result available to the session.

Inspect the session

Use List Tables to confirm registration and Describe Table to inspect the schema.

Inspecting first is especially important for agent-driven analysis and sources whose schema can evolve. Do not let a model guess table or column names when the workflow can retrieve them.

Execute SQL

Structured workflow output

SQL Query returns:

• a CSVTable for analytics and visualization;
• an array of row objects for workflow iteration;
• the row count.

Use it when downstream nodes need structured values.

Agent-readable output

Execute SQL returns a Markdown table, a CSVTable, and the row count. Its formatted text output is convenient for a controlled data-analysis tool, but large results should remain in structured storage rather than being copied into a model context.

SQL examples

Aggregate by period

SELECT
  DATE_TRUNC('month', order_date) AS month,
  SUM(revenue) AS revenue
FROM orders
WHERE order_date >= DATE '2026-01-01'
GROUP BY DATE_TRUNC('month', order_date)
ORDER BY month;

Join registered sources

SELECT
  o.order_id,
  c.customer_name,
  o.revenue
FROM orders AS o
JOIN customers AS c
  ON o.customer_id = c.customer_id
WHERE o.status = 'complete';

Window calculation

SELECT
  order_date,
  revenue,
  SUM(revenue) OVER (
    ORDER BY order_date
    ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
  ) AS seven_row_revenue
FROM daily_sales;

Common table expression

WITH monthly_sales AS (
  SELECT
    DATE_TRUNC('month', order_date) AS month,
    SUM(revenue) AS revenue
  FROM orders
  GROUP BY DATE_TRUNC('month', order_date)
)
SELECT *
FROM monthly_sales
ORDER BY month;

Dynamic query text should be constructed only from strictly parsed or allow-listed values. The SQL Query node accepts a query string; do not concatenate arbitrary user input into it.

Time-series helper nodes

The catalog includes workflow-oriented helpers for common time operations:

• Time Bin Aggregation
• Date Truncate Aggregation
• Window Aggregation
• Time Range Filter
• DateTime to SQL Timestamp

Use SQL when the calculation is already clear there. Use helper nodes when their typed inputs make a reusable board easier to configure safely.

Write results

Write Delta Table writes a result into Delta Lake. For other destinations, pass the CSVTable or row output to the corresponding file, database, API, or A2UI node.

Before publishing a derived table, record its source window, query or board version, and row count.

Performance guidance

• Filter early and select only required columns.
• Prefer Parquet or a lake table for repeated analytical scans.
• Aggregate before sending data to an A2UI page or model.
• Add LIMIT while exploring an unfamiliar table.
• Avoid per-row workflow loops for operations SQL can perform as a set.
• Inspect whether source filters are pushed down before assuming a federated query is cheap.
• Separate a fast summary query from slower drill-down queries.

Troubleshooting

Symptom	Check
Table not found	Same session value, registration execution path, exact table name
Column not found	Describe Table output, casing, schema evolution
Query is slow	Selected columns, filters, join size, source pushdown
Memory pressure	Result size, early aggregation, Parquet, batch boundaries
Unexpected duplicate rows	Join keys and source grain
Agent produces invalid SQL	List/describe tools, read-only tool, row limits, retry policy

Next steps

• Data loading and storage
• Data visualization
• AI-powered analysis
• Data pipelines