Documentation

A bit-packed validity bitmap. Bit i = 1 means row i is valid (not null).

Type-erased column GADT. Pattern-matching on the constructor recovers the representation; nullability is an optional bit-packed Bitmap (Nothing = no nulls, Just bm = bit i set iff row i is valid).

A mutable companion struct to dataframe columns.

Used mostly as an intermediate structure for I/O.

Test whether row i is valid (not null) in a bitmap.

Build a fully-valid bitmap for n rows (all bits set).

Build a bitmap from a VU.Vector Word8 validity vector (1 = valid, 0 = null), as produced by Arrow / Parquet decoders.

Build a bitmap from a list of null-row indices. nullIdxs are the positions that are NULL.

Slice a bitmap for rows [start .. start+len-1].

Concatenate two bitmaps covering n1 and n2 rows respectively.

Combine two bitmaps with AND (both must be valid for result to be valid).

Materialize a nullable column from VB.Vector (Maybe a); picks UnboxedColumn when a is unboxable, else BoxedColumn. Always attaches a bitmap so the column reads as nullable even with no Nothing values.

Materialize a nullable UnboxedColumn to VB.Vector (Maybe a) using runST. Always attaches a bitmap so the column is recognized as nullable even when no Nothing values are present (preserves the Maybe type marker).

Whether row i is null, respecting the bitmap.

Return the 'Maybe Bitmap' from a column.

Decode a PackedText into a BoxedColumn Text (bit-identical to materializing at freeze). Identity on every other column.

Whether a column is a PackedText.

Whether a column is a MergedColumn.

MergedColumn defers element construction, so forcing must still surface the one deferred error — a row null on both sides — inside strict IO/executor boundaries. O(rows) bitmap walk, no allocation; both-null needs a bitmap on each side, so anything else passes immediately.

A wrapper around the type-erased Column carrying a phantom element type, used to type-check expressions. The phantom is not guaranteed to match the underlying vector's type.

Gets the underlying value from a TypedColumn.

Gets the underlying vector from a TypedColumn.

Checks if a column contains missing values (has a bitmap).

Checks if a column contains only missing values.

Checks if a column contains numeric values.

Whether the column stores element type a. For nullable columns, also True when a = Maybe b and the column stores b internally.

An internal/debugging function to get the column type of a column.

An internal/debugging function to get the type stored in the outermost vector of a column.

Force evaluation of all elements in a column. Replacement for the removed instance NFData Column; used by the IO and lazy-executor strict paths.

Compare two nullable boxed columns element by element, skipping null slots. Uses a manual loop to avoid stream fusion forcing null-slot error thunks.

Byte-slice equality of two packed-text columns, skipping null slots (a null compares equal only to a null), mirroring eqBoxedCols.

A class for converting a vector to a column of the appropriate type. Given each Rep we tell the toColumnRep function which Column type to pick.

Constraint synonym for what we can put into columns.

O(n) Convert a vector to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> import qualified Data.Vector as V
> fromVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert an unboxed vector to a column. This avoids the extra conversion if you already have the data in an unboxed vector.

Examples:

> import qualified Data.Vector.Unboxed as V
> fromUnboxedVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert a list to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> fromList [(1 :: Int), 2, 3, 4]
[1,2,3,4]

O(n) Create a column of random elements within a range.

Takes a random number generator, a length, and a lower and upper bound for the random values.

Examples:

> import System.Random (mkStdGen)
> mkRandom (mkStdGen 42) 4 0 10
[4,2,6,5]

An internal function to map a function over the values of a column.

Applies a function that returns an unboxed result to an unboxed vector, storing the result in a column.

O(1) Gets the number of elements in the column.

O(n) Gets the number of non-null elements in the column.

O(n) Takes the first n values of a column.

O(n) Takes the last n values of a column.

O(n) Takes n values after a given column index.

O(n) Selects the elements at a given set of indices. Does not change the order.

Like atIndicesStable but treats negative indices as null. Keeps the index vector fully unboxed (no VB.Vector (Maybe Int)).

Internal helper to get indices in a boxed vector.

Internal helper to get indices in an unboxed vector.

Fold (right) column with index.

O(n) Seedless fold over groups using the first element of each group as seed. Like foldDirectGroups but for the case where no initial accumulator is available.

O(n) fold over groups by scanning the column linearly (rowToGroup[i] = group of row i). Random writes hit the small per-group accumulator array; when acc is unboxable that array is unboxed, avoiding pointer indirection.

An internal, column version of zip.

Merge two columns using These. O(1): the sides are kept in their native representation and These values materialize on element access.

Decode a MergedColumn into the eager BoxedColumn (These a b) form.

The eager element-wise merge (These per row, boxed). Bitmaps are honored for every representation pair: a null side yields This/That, both-null is an error (the join kernels never produce such a row).

An internal, column version of zipWith.

Freeze a mutable column into an Either Text a column: every recorded null position becomes Left rawText (preserving the original input), every other position becomes Right v. Used by CSV readers under EitherRead mode.

Promote a non-nullable column to a nullable one (add an all-valid bitmap). No-op when already nullable.

Fills the end of a column, up to n, with null rows. Does nothing if column has length >= n.

Fills the beginning of a column, up to n, with null rows. Does nothing if column has length >= n.

Concatenates two columns. Returns Nothing if the columns are of different types.

Like concatColumns but also combines columns of different types by wrapping values in Either (e.g. [1,2] and ["a","b"] become [Left 1, Left 2, Right "a", Right "b"]).

O(n) Concatenate a list of same-type columns in a single allocation. All columns must have the same constructor and element type (as they will within a single Parquet column). Calls error on mismatch.

Allocate a mutable column of size n matching the constructor/type of the given column.

Copy a column chunk into a mutable column starting at offset off.

Freeze a mutable column into an immutable column.

O(n) Converts a column to a list. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toList Int column
[1,2,3,4]
> toList Double column
exception: ...

Type-safe conversion of a column to a vector of element type a (specify via type application); Left TypeMismatchException when the column's type differs.

>>> toVector @Int @VU.Vector column
Right (unboxed vector of Ints)

>>> toVector @Text @VB.Vector column
Right (boxed vector of Text)

Convert a column to an unboxed Double vector, coercing numeric types (realToFrac for floats, fromIntegral for integrals; nulls become NaN). Left TypeMismatchException when the column is not numeric.

Convert a column to an unboxed Float vector, coercing numeric types (nulls become NaN); Left TypeMismatchException when not numeric. Converting from Double may lose precision.

Convert a column to an unboxed Int vector, coercing numeric types (floats are rounded via banker's rounding); Left TypeMismatchException when the column is not numeric. Does not support nullable columns.