Fit TF-IDF Vectorizer Node

AI/ML/Preprocessing

Learn a vocabulary from a text column and turn documents into numeric vectors weighted by term frequency times inverse document frequency. Feed the fitted vectorizer to Apply Transform to vectorize a column, then train a classifier such as Multinomial Naive Bayes on the result. Tokenization always uses the built-in regex tokenizer, because a custom tokenizer function cannot be persisted and would make the saved model unloadable.

fit_tfidf_vectorizerml

Inputs10

Outputs3

Security exposure6/10

Packageml

Ratings

Scores range from 0 to 10. Higher values mean more impact, exposure, or operational weight.

SecurityAttack surface and exposure impact.

6/10Medium

PrivacyPotential sensitivity of processed data.

5/10Medium

PerformanceRuntime or resource pressure.

7/10High

GovernancePolicy, audit, or compliance impact.

7/10High

ReliabilityOperational stability considerations.

8/10High

CostExternal or compute cost impact.

8/10High

Input Pins

Input

Execution

exec_in

Execution trigger that begins vectorizer fitting

Data Source

String

source

Choose which backend supplies the documents

Default Database

Database

IDF Method

String

method

Weighting formula. Smooth: log((1+n)/(1+df))+1, never divides by zero. Non-Smooth: log(n/df)+1, sharper but requires every term to appear at least once. Textbook: log(n/(1+df)), which discounts terms appearing in nearly every document down to a negative weight, so it cannot feed Multinomial Naive Bayes.

Default Smooth

SmoothNon-SmoothTextbook

Min N-Gram

Integer

n_gram_min

Smallest number of adjacent tokens forming a vocabulary entry (1 = single words)

Default 1

Range 1 to 10

Max N-Gram

Integer

n_gram_max

Largest number of adjacent tokens forming a vocabulary entry. Must not be smaller than Min N-Gram.

Default 1

Range 1 to 10

Lowercase

Boolean

convert_to_lowercase

Lowercase every document before tokenizing, so casing variants collapse into one vocabulary entry

Default true

Max Features

Integer

max_features

Keep only the most frequent N vocabulary entries, which caps the width of the produced vectors. 0 keeps all of them.

Default 0

Range 0 to 1000000

Min Document Frequency

Float

min_document_frequency

Drop terms appearing in a smaller share of documents than this (0-1). Useful to remove typos and one-off tokens.

Default 0

Range 0 to 1

Max Document Frequency

Float

max_document_frequency

Drop terms appearing in a larger share of documents than this (0-1). Useful to remove boilerplate that carries no signal.

Default 1

Range 0 to 1

Stopwords

String

stopwords

Comma separated words to exclude from the vocabulary, e.g. `the, and, of`. Leave empty to keep every term.

Output Pins

Done

Execution

exec_out

Activated once the vectorizer is fitted

Model

Struct

model

Thread-safe handle to the fitted TF-IDF vectorizer, for use with Apply Transform

Schema enforced

Vocabulary

String Array

vocabulary

Learned vocabulary entries, in the same order as the columns of the produced vectors

Node Info

Internal name: fit_tfidf_vectorizer
Category: AI/ML/Preprocessing