Documentation

Our representation of a column is a GADT that can store data based on the underlying data.

This allows us to pattern match on data kinds and limit some operations to only some kinds of vectors. E.g. operations for missing data only happen in an OptionalColumn.

A TypedColumn is a wrapper around our type-erased column. It is used to type check expressions on columns.

Gets the underlying value from a TypedColumn.

Gets the underlying vector from a TypedColumn.

Checks if a column contains missing values.

Checks if a column contains only missing values.

Checks if a column contains numeric values.

Checks if a column is of a given type values.

An internal/debugging function to get the column type of a column.

An internal/debugging function to get the type stored in the outermost vector of a column.

A class for converting a vector to a column of the appropriate type. Given each Rep we tell the toColumnRep function which Column type to pick.

Constraint synonym for what we can put into columns.

O(n) Convert a vector to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> import qualified Data.Vector as V
> fromVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert an unboxed vector to a column. This avoids the extra conversion if you already have the data in an unboxed vector.

Examples:

> import qualified Data.Vector.Unboxed as V
> fromUnboxedVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert a list to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> fromList [(1 :: Int), 2, 3, 4]
[1,2,3,4]

An internal function to map a function over the values of a column.

Applies a function that returns an unboxed result to an unboxed vector, storing the result in a column.

O(1) Gets the number of elements in the column.

O(n) Gets the number of elements in the column.

O(n) Takes the first n values of a column.

O(n) Takes the last n values of a column.

O(n) Takes n values after a given column index.

O(n) Selects the elements at a given set of indices. May change the order.

O(n) Selects the elements at a given set of indices. Does not change the order.

Like atIndicesStable but treats negative indices as null, producing an OptionalColumn. Keeps the index vector fully unboxed (no VB.Vector (Maybe Int)).

Internal helper to get indices in a boxed vector.

Internal helper to get indices in an unboxed vector.

An internal function that returns a vector of how indexes change after a column is sorted.

Fold (right) column with index.

O(n) Fold a column over groups without materialising a sorted copy. Instead of backpermuting the column (expensive: O(n) allocation + random reads), this iterates valueIndices sequentially and accesses the column at each index. Avoids one 10M-element allocation per column per aggregation expression.

O(n) Seedless fold over groups using the first element of each group as seed. Like foldDirectGroups but for the case where no initial accumulator is available.

O(n) fold over groups by scanning the column LINEARLY. rowToGroup[i] = group index for row i. Avoids random column reads; random writes go to the accumulator array which is small (nGroups entries) and typically cache-resident. When acc is unboxable, uses an unboxed mutable vector for the accumulator array, eliminating pointer indirection on every read/write.

An internal, column version of zip.

Merge two columns using These.

An internal, column version of zipWith.

Fills the end of a column, up to n, with Nothing. Does nothing if column has length greater than n.

Fills the beginning of a column, up to n, with Nothing. Does nothing if column has length greater than n.

Concatenates two columns. Returns Nothing if the columns are of different types.

Concatenates two columns.

Works similar to concatColumns, but unlike that function, it will also combine columns of different types by wrapping the values in an Either.

E.g. combining Column containing [1,2] with Column containing ["a","b"] will result in a Column containing [Left 1, Left 2, Right "a", Right "b"].

O(n) Concatenate a list of same-type columns in a single allocation. All columns must have the same constructor and element type (as they will within a single Parquet column). Calls error on mismatch.

Allocate a mutable column of size n matching the constructor/type of the given column.

Copy a column chunk into a mutable column starting at offset off.

Freeze a mutable column into an immutable column.

O(n) Converts a column to a list. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toList Int column
[1,2,3,4]
> toList Double column
exception: ...

Converts a column to a vector of a specific type.

This is a type-safe conversion that requires the column's element type to exactly match the requested type. You must specify the desired type via type applications.

Type Parameters

Examples

>>> toVector @Int @VU.Vector column
Right (unboxed vector of Ints)

>>> toVector @Text @VB.Vector column
Right (boxed vector of Text)

Returns

See also

Converts a column to an unboxed vector of Double values.

This function performs intelligent type coercion for numeric types:

• If the column is already Double, returns it directly
• If the column contains other floating-point types, converts via realToFrac
• If the column contains integral types, converts via fromIntegral (beware of overflow if the type is Integer).

Optional column handling

Returns

Converts a column to an unboxed vector of Float values.

This function performs intelligent type coercion for numeric types:

• If the column is already Float, returns it directly
• If the column contains other floating-point types, converts via realToFrac
• If the column contains integral types, converts via fromIntegral
• If the column is boxed Integer, converts via fromIntegral (beware of overflow for 64-bit integers and Integer)

Optional column handling

Returns

Precision warning

Converts a column to an unboxed vector of Int values.

This function performs intelligent type coercion for numeric types:

• If the column is already Int, returns it directly
• If the column contains floating-point types, rounds via round and converts
• If the column contains other integral types, converts via fromIntegral
• If the column is boxed Integer, converts via fromIntegral

Returns

Note

Rounding behavior