Data Loading & Storage

Choose the reader from the source format and expected data size. Preserve the source path and schema alongside the loaded data so downstream analysis remains traceable.

Choose a loading path

Input	Recommended starting point
Small text file	Read to String
Large CSV	Buffered CSV Reader
Queryable CSV, JSON, or Parquet	Mount it in a DataFusion session
Excel workbook	Cell, worksheet, or table extraction nodes
JSON payload	Parse with a schema; repair only when the source is expected to be imperfect
App file	Use the app storage path abstraction
Local structured records	Open the local database and insert or upsert
External database or lake	Register it with DataFusion
Provider file store	Use the provider connection and its typed file nodes

CSV files

Use Buffered CSV Reader when the file should be processed in bounded batches. Validate the header before accepting the first batch and keep a rejection path for malformed rows.

For SQL analysis, use Mount CSV to register the file in a DataFusion session.

Check:

delimiter and quoting rules;
text encoding;
whether headers are present and unique;
decimal, date, and timezone conventions;
empty-string versus null behavior;
expected row count and key uniqueness.

Excel files

Need	Node
Read or write one cell	Excel Read Cell, Excel Write Cell
Discover worksheets	Get Sheet Names
Extract predictable tables	Extract Tables (Excel)
Extract irregular tables	Extract Tables AI (Excel)

Inspect sheet names first, then select the intended sheet and validate its expected columns. AI extraction is useful for unusual layouts, but it should still be followed by type, row-count, and business-rule checks.

JSON

Use Parse JSON with Schema when the expected shape is known. The schema makes required fields and types explicit and avoids spreading defensive field checks throughout the board.

Repair Parse JSON is for input that may be almost, but not quite, valid JSON. Do not use repair to hide a broken contract from a system you control; fix the producer or reject the payload.

Parquet and analytical files

Mount Parquet registers a Parquet file for SQL queries without converting it to rows first. Parquet is a good fit for repeated analytical scans because it is columnar and carries a schema.

SELECT
  region,
  SUM(revenue) AS revenue
FROM analytics
WHERE event_date >= DATE '2026-01-01'
GROUP BY region
ORDER BY revenue DESC;

Use Mount JSON for JSON or NDJSON and Mount CSV for delimited files.

App storage and paths

App storage gives workflows a provider-independent path to files owned by the app. See App storage for how files are organized and accessed.

The file catalog provides distinct path constructors:

Source	Node
App storage directory	Storage Dir
Explicit raw path	Raw Path
Convert a raw path into a Flow-Like path	From Raw Path
Convert a local path value	Local Path to Path

Use Path Exists? before an optional read, and List Paths to enumerate a directory. Avoid relying on machine-specific paths in a board intended to run on different backends.

Local database

Open Database opens the app-local database. Choose a write node by volume and retry behavior:

Behavior	Node
Insert one record	Insert
Insert a collection	Batch Insert
Insert a CSV table	Batch Insert (CSV)
Retry-safe single write	Upsert
Retry-safe collection write	Batch Upsert

Build a stable key before using upsert. After a large write, Flush Database can make the persistence boundary explicit. Use Build Index for fields that support repeated filters or searches, then measure whether the index improves the intended workload.

Query local records with (SQL) Filter Database. Keep result limits and selected fields bounded for interactive workflows.

External sources

DataFusion can register PostgreSQL, MySQL, SQLite, DuckDB, ClickHouse, Oracle, BigQuery, Athena, FlightSQL, and other cataloged sources. Browse DataFusion databases and DataFusion lakes for the current set.

Provider nodes cover service-specific file and data operations. The catalog includes AWS, Azure, GCP, and Cloudflare provider builders plus typed integrations for services such as Microsoft 365, Google Workspace, GitHub, Notion, Atlassian, and Databricks.

Keep credentials in secrets or provider connections, not in path strings or examples.

Writing files

Use:

Write String for text;
Write Bytes for binary content;
format-specific writers when the destination has a structured contract.

Write to a temporary or versioned path first when replacing an important artifact. Confirm the result before moving consumers to the new file.

Loading checklist

Reader matches the actual format and expected size
Source path, version, and ingestion time are retained
Header or schema is validated before processing
Large files are streamed, mounted, or batched
Invalid rows have a rejection reason and source reference
Destination writes are retry-safe
Credentials come from secrets or provider connections
Machine-specific paths are avoided in portable boards
Row counts and key uniqueness are checked

Troubleshooting

Symptom	Check
File not found	Path constructor, storage scope, permissions, execution backend
Out of memory	Buffered reader, DataFusion mount, batch size, selected columns
CSV rows shift columns	Delimiter, quoting, embedded newlines, encoding
Spreadsheet table is missing	Worksheet selection, merged cells, table boundaries
JSON fields disappear	Schema optionality, field names, repair behavior
Duplicate database records	Stable key, upsert choice, checkpoint timing