Data Loading & Storage

Choose the reader from the source format and expected data size. Preserve the source path and schema alongside the loaded data so downstream analysis remains traceable.

Choose a loading path

Input	Recommended starting point
Small text file	Read to String
Large CSV	Buffered CSV Reader
Queryable CSV, JSON, or Parquet	Mount it in a DataFusion session
Excel workbook	Cell, worksheet, or table extraction nodes
JSON payload	Parse with a schema; repair only when the source is expected to be imperfect
App file	Use the app storage path abstraction
Local structured records	Open the local database and insert or upsert
External database or lake	Register it with DataFusion
Provider file store	Use the provider connection and its typed file nodes

CSV files

Use Buffered CSV Reader when the file should be processed in bounded batches. Validate the header before accepting the first batch and keep a rejection path for malformed rows.

For SQL analysis, use Mount CSV to register the file in a DataFusion session.

Check:

delimiter and quoting rules;
text encoding;
whether headers are present and unique;
decimal, date, and timezone conventions;
empty-string versus null behavior;
expected row count and key uniqueness.

Excel files

Need	Node
Read or write one cell	Excel Read Cell, Excel Write Cell
Discover worksheets	Get Sheet Names
Extract predictable tables	Extract Tables (Excel)
Extract irregular tables	Extract Tables AI (Excel)

Inspect sheet names first, then select the intended sheet and validate its expected columns. AI extraction is useful for unusual layouts, but it should still be followed by type, row-count, and business-rule checks.

JSON

Use Parse JSON with Schema when the expected shape is known. The schema makes required fields and types explicit and avoids spreading defensive field checks throughout the board.

Repair Parse JSON is for input that may be almost, but not quite, valid JSON. Do not use repair to hide a broken contract from a system you control; fix the producer or reject the payload.

Parquet and analytical files

Mount Parquet registers a Parquet file for SQL queries without converting it to rows first. Parquet is a good fit for repeated analytical scans because it is columnar and carries a schema.

SELECT
  region,
  SUM(revenue) AS revenue
FROM analytics
WHERE event_date >= DATE '2026-01-01'
GROUP BY region
ORDER BY revenue DESC;

Use Mount JSON for JSON or NDJSON and Mount CSV for delimited files.

App storage and paths

App storage gives workflows a provider-independent path to files owned by the app. See App storage for how files are organized and accessed.

The file catalog provides distinct path constructors:

Source	Node
App storage directory	Storage Dir
Explicit raw path	Raw Path
Convert a raw path into a Flow-Like path	From Raw Path
Convert a local path value	Local Path to Path

Use Path Exists? before an optional read, and List Paths to enumerate a directory. Avoid relying on machine-specific paths in a board intended to run on different backends.

Local database

Open Database opens the app-local database. Choose a write node by volume and retry behavior:

Behavior	Node
Insert one record	Insert
Insert a collection	Batch Insert
Insert a CSV table	Batch Insert (CSV)
Retry-safe single write	Upsert
Retry-safe collection write	Batch Upsert

Build a stable key before using upsert. After a large write, Flush Database can make the persistence boundary explicit. Use Build Index for fields that support repeated filters or searches. Choosing a Lance index explains the current Flow-Like choices, the latest stable upstream options, and when a legacy vector index needs a one-time rebuild for cosine distance.

Query local records with (SQL) Filter Database. Keep result limits and selected fields bounded for interactive workflows.

Column types from the first write

A table without a declared schema takes its schema from the insert or upsert that creates it. Most columns follow their JSON shape. These top-level columns are typed from their non-null values instead:

Values in the first write	Column type
Strings that all parse as RFC3339, as a Date pin produces	`Timestamp(Millisecond, "UTC")`, see Dates
Valid GeoJSON geometries, as a Geometry pin produces, including mixed kinds	WKB geometry with WGS 84 metadata, see Geometry
Arrays of integers 0–255, at least one non-empty, as a Byte array pin produces	`Binary`
Arrays in a column named `vector`	Fixed-size `Float32` vector, sized by the first non-empty array

A single value that does not fit, such as a Feature wrapper or the integer 300, leaves the column to its JSON shape. Reads return geometry as GeoJSON and Binary as an array of numbers.

After creation, the stored schema is authoritative. Later writes must fit it, so a Binary column rejects values outside 0–255, and existing tables keep their column types. Declared schemas have no integer-list type; nest a list of small integers inside an object when it must stay a list. To change an inferred type, recreate the table or create it with a declared schema in Data Studio.

Branches, versions, and snapshots

Open Database and Open Remote Database accept a branch and a revision: Latest, Version, or Tag. Latest opens the current branch for writes when permissions allow. Version and Tag open read-only snapshots. Version numbers belong to a branch; tags resolve to a branch and version when opened. A missing branch, version, or tag raises an error.

Use Checkout Database to open another reference without changing a connection already used elsewhere in the flow. Snapshot Database flushes pending writes and returns a frozen handle. Give the snapshot a retention tag when a model or experiment needs that data after version cleanup. Get Database Reference and Flush Database expose the committed version. Save that reference with the app, storage scope, selected columns, filter, and split definition when recording training provenance.

Task	Nodes
Inspect history	List Database Versions, List Database Branches, List Database Tags
Run an experiment on separate data	Create Database Branch, Checkout Database
Name a retained snapshot	Create Database Tag, Move Database Tag, Delete Database Tag
Restore historical contents	Restore Database Version
Compare two views	Compare Database Views
Create another table sharing source data	Clone Database
Remove a branch	Delete Database Branch
Remove old unprotected versions	Cleanup Database Versions

Restore Database Version writes a new version on the snapshot’s branch. Compare Database Views requires a unique, non-null string or integer key and returns full counts with a bounded changed-row preview. Clone Database shares source files and retains a source tag; it is not an independent backup. Remove the clone before removing its source protection.

Read-only snapshot handles also apply to existing filters, vector searches, schema reads, and DataFusion mounts. Writes through these handles fail before queuing rows. The Predict node accepts an optional Output Database and a Key Column so a frozen input can write full rows with predictions to a separate writable branch or table.

Data Studio’s native table viewer provides the same history and reference controls. To delete rows, enter a filter, preview its matches, and confirm the table name. The filter runs again when deletion executes, so concurrent writes may change the number removed. Drop Table removes the whole table, including every branch, and prunes graph references. Use row deletion or Purge Database when the schema and history should remain available. The multi-source Query Workbench continues to use its own source selection.

External sources

DataFusion can register PostgreSQL, MySQL, SQLite, DuckDB, ClickHouse, Oracle, BigQuery, Athena, FlightSQL, and other cataloged sources. Browse DataFusion databases and DataFusion lakes for the current set.

Provider nodes cover service-specific file and data operations. The catalog includes AWS, Azure, GCP, and Cloudflare provider builders plus typed integrations for services such as Microsoft 365, Google Workspace, GitHub, Notion, Atlassian, and Databricks.

Keep credentials in secrets or provider connections, not in path strings or examples.

Writing files

Use:

Write String for text;
Write Bytes for binary content;
format-specific writers when the destination has a structured contract.

Write to a temporary or versioned path first when replacing an important artifact. Confirm the result before moving consumers to the new file.

Loading checklist

Reader matches the actual format and expected size
Source path, version, and ingestion time are retained
Header or schema is validated before processing
Large files are streamed, mounted, or batched
Invalid rows have a rejection reason and source reference
Destination writes are retry-safe
Credentials come from secrets or provider connections
Machine-specific paths are avoided in portable boards
Row counts and key uniqueness are checked

Troubleshooting

Symptom	Check
File not found	Path constructor, storage scope, permissions, execution backend
Out of memory	Buffered reader, DataFusion mount, batch size, selected columns
CSV rows shift columns	Delimiter, quoting, embedded newlines, encoding
Spreadsheet table is missing	Worksheet selection, merged cells, table boundaries
JSON fields disappear	Schema optionality, field names, repair behavior
Duplicate database records	Stable key, upsert choice, checkpoint timing
A database column has an unexpected type	Values in the write that created the table, declared schema