O(1) Creates an empty dataframe

Checks if the dataframe is empty (has no columns).

Returns True if the dataframe has no columns, False otherwise. Note that a dataframe with columns but no rows is not considered null.

A record that contains information about how and what rows are grouped in the dataframe. This can only be used with aggregate.

Retrieves a column as an unboxed vector of Double values.

Returns Left with a DataFrameException if the column cannot be converted to doubles. This may occur if the column contains non-numeric data.

Retrieves a column as an unboxed vector of Float values.

Returns Left with a DataFrameException if the column cannot be converted to floats. This may occur if the column contains non-numeric data.

Retrieves a column as an unboxed vector of Int values.

Returns Left with a DataFrameException if the column cannot be converted to ints. This may occur if the column contains non-numeric data or values outside the Int range.

Get a specific column as a vector.

You must specify the type via type applications.

Examples

>>> columnAsVector @Int "age" df
[25, 30, 35, ...]

>>> columnAsVector @Text "name" df
["Alice", "Bob", "Charlie", ...]

Throws

Returns a dataframe as a two dimensional vector of doubles.

Converts all columns in the dataframe to double vectors and transposes them into a row-major matrix representation.

This is useful for handing data over into ML systems.

Returns Left with an error if any column cannot be converted to doubles.

Returns a dataframe as a two dimensional vector of floats.

Converts all columns in the dataframe to float vectors and transposes them into a row-major matrix representation.

This is useful for handing data over into ML systems.

Returns Left with an error if any column cannot be converted to floats.

Returns a dataframe as a two dimensional vector of ints.

Converts all columns in the dataframe to int vectors and transposes them into a row-major matrix representation.

This is useful for handing data over into ML systems.

Returns Left with an error if any column cannot be converted to ints.

For showing the dataframe as markdown in notebooks.

O(n) Convert a list to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> fromList [(1 :: Int), 2, 3, 4]
[1,2,3,4]

O(n) Converts a column to a list. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toList Int column
[1,2,3,4]
> toList Double column
exception: ...

Our representation of a column is a GADT that can store data based on the underlying data.

This allows us to pattern match on data kinds and limit some operations to only some kinds of vectors. E.g. operations for missing data only happen in an OptionalColumn.

O(n) Convert an unboxed vector to a column. This avoids the extra conversion if you already have the data in an unboxed vector.

Examples:

> import qualified Data.Vector.Unboxed as V
> fromUnboxedVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert a vector to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> import qualified Data.Vector as V
> fromVector (VB.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

Checks if a column is of a given type values.

Checks if a column contains missing values.

Checks if a column contains numeric values.

Converts a column to a vector of a specific type.

This is a type-safe conversion that requires the column's element type to exactly match the requested type. You must specify the desired type via type applications.

Type Parameters

Examples

>>> toVector @Int @VU.Vector column
Right (unboxed vector of Ints)

>>> toVector @Text @VB.Vector column
Right (boxed vector of Text)

Returns

See also

Unwraps a value from an Any type.

Wraps a value into an Any type. This helps up represent rows as heterogenous lists.

Converts the entire dataframe to a list of rows.

Each row contains all columns in the dataframe, ordered by their column indices. The rows are returned in their natural order (from index 0 to n-1).

Examples

>>> toRowList df
[Row {name = "Alice", age = 25, ...}, Row {name = "Bob", age = 30, ...}, ...]

Performance note

Converts the dataframe to a vector of rows with only the specified columns.

Each row will contain only the columns named in the names parameter. This is useful when you only need a subset of columns or want to control the column order in the resulting rows.

Parameters

Examples

>>> toRowVector ["name", "age"] df
Vector of rows with only name and age fields

>>> toRowVector [] df  -- Empty column list
Vector of empty rows (one per dataframe row)

A left fold for dataframes that takes the dataframe as the last object. This makes it easier to chain operations.

Example

ghci> D.fold (const id) [1..5] df

-----------------
index |  0  |  1
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13
3     | 4   | 14
4     | 5   | 15
5     | 6   | 16
6     | 7   | 17
7     | 8   | 18
8     | 9   | 19
9     | 10  | 20

O(n) Renames a single column.

Example

ghci> import qualified Data.Vector as V

ghci> df = insertVector "numbers" (V.fromList [1..10]) D.empty

ghci> D.rename "numbers" "others" df

--------------
index | others
------|-------
 Int  |  Int
------|-------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(1) Get DataFrame dimensions i.e. (rows, columns)

Example

ghci> D.dimensions df

(100, 3)

O(k) Get column names of the DataFrame in order of insertion.

Example

ghci> D.columnNames df

["col_a", "col_b", "col_c"]

Adds a vector to the dataframe. If the vector has less elements than the dataframe and the dataframe is not empty the vector is converted to type `Maybe a` filled with Nothing to match the size of the dataframe. Similarly, if the vector has more elements than what's currently in the dataframe, the other columns in the dataframe are change to `Maybe Type` and filled with Nothing.

Example

ghci> import qualified Data.Vector as V

ghci> D.insertVector "numbers" (V.fromList [1..10]) D.empty

---------------
index | numbers
------|--------
 Int  |   Int
------|--------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(n) Add a column to the dataframe.

Example

ghci> D.insertColumn "numbers" (D.fromList [1..10]) D.empty

---------------
index | numbers
------|--------
 Int  |   Int
------|--------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(k) Add a column to the dataframe providing a default. This constructs a new vector and also may convert it to an unboxed vector if necessary. Since columns are usually large the runtime is dominated by the length of the list, k.

O(n) Adds an unboxed vector to the dataframe.

Same as insertVector but takes an unboxed vector. If you insert a vector of numbers through insertVector it will either way be converted into an unboxed vector so this function saves that extra work/conversion.

O(n) Clones a column and places it under a new name in the dataframe.

Example

ghci> import qualified Data.Vector as V

ghci> df = insertVector "numbers" (V.fromList [1..10]) D.empty

ghci> D.cloneColumn "numbers" "others" df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 1
1     | 2       | 2
2     | 3       | 3
3     | 4       | 4
4     | 5       | 5
5     | 6       | 6
6     | 7       | 7
7     | 8       | 8
8     | 9       | 9
9     | 10      | 10

O(n) Renames many columns.

Example

ghci> import qualified Data.Vector as V

ghci> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)

ghci> df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 11
1     | 2       | 12
2     | 3       | 13
3     | 4       | 14
4     | 5       | 15
5     | 6       | 16
6     | 7       | 17
7     | 8       | 18
8     | 9       | 19
9     | 10      | 20

ghci> D.renameMany [("numbers", "first_10"), ("others", "next_10")] df

--------------------------
index | first_10 | next_10
------|----------|--------
 Int  |   Int    |   Int
------|----------|--------
0     | 1        | 11
1     | 2        | 12
2     | 3        | 13
3     | 4        | 14
4     | 5        | 15
5     | 6        | 16
6     | 7        | 17
7     | 8        | 18
8     | 9        | 19
9     | 10       | 20

O(n * k ^ 2) Returns the number of non-null columns in the dataframe and the type associated with each column.

Example

ghci> import qualified Data.Vector as V

ghci> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)

ghci> D.describeColumns df

-----------------------------------------------------------------------------------------------------
index | Column Name | # Non-null Values | # Null Values | # Partially parsed | # Unique Values | Type
------|-------------|-------------------|---------------|--------------------|-----------------|-----
 Int  |    Text     |        Int        |      Int      |        Int         |       Int       | Text
------|-------------|-------------------|---------------|--------------------|-----------------|-----
0     | others      | 10                | 0             | 0                  | 10              | Int
1     | numbers     | 10                | 0             | 0                  | 10              | Int

Creates a dataframe from a list of tuples with name and column.

Example

ghci> df = D.fromNamedColumns [("numbers", D.fromList [1..10]), ("others", D.fromList [11..20])]

ghci> df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 11
1     | 2       | 12
2     | 3       | 13
3     | 4       | 14
4     | 5       | 15
5     | 6       | 16
6     | 7       | 17
7     | 8       | 18
8     | 9       | 19
9     | 10      | 20

Create a dataframe from a list of columns. The column names are "0", "1"... etc. Useful for quick exploration but you should probably always rename the columns after or drop the ones you don't want.

Example

ghci> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

ghci> df

-----------------
index |  0  |  1
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13
3     | 4   | 14
4     | 5   | 15
5     | 6   | 16
6     | 7   | 17
7     | 8   | 18
8     | 9   | 19
9     | 10  | 20

Create a dataframe from a list of column names and rows.

Example

ghci> df = D.fromRows [A, B] [[D.toAny 1, D.toAny 11], [D.toAny 2, D.toAny 12], [D.toAny 3, D.toAny 13]]

ghci> df

-----------------
index |  A  |  B
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13

O (k * n) Counts the occurences of each value in a given column.

Example

ghci> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

ghci> D.valueCounts @Int "0" df

[(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1)]

Construct a SchemaType for the given a.

Examples

>>> :set -XTypeApplications
>>> schemaType @T.Text == schemaType @T.Text
True

>>> show (schemaType @Double)
"Double"

CSV read parameters.

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsv "./data/taxi.csv"

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsvWithOpts "./data/taxi.csv" (D.defaultReadOptions { dateFormat = "%d%-m%-Y" })

Read text file with specified delimiter into a dataframe.

Example

ghci> D.readSeparated ';' D.defaultReadOptions "./data/taxi.txt"

Read TSV (tab separated) file from path and load it into a dataframe.

Example

ghci> D.readTsv "./data/taxi.tsv"

Read a parquet file from path and load it into a dataframe.

Example

ghci> D.readParquet "./data/mtcars.parquet"

O(k * n) Take a range of rows of a DataFrame.

O(k * n) Take the first n rows of a DataFrame.

O(n * k) Filter rows by a given condition.

filter "x" even df

O(k * n) Drop the first n rows of a DataFrame.

O(n) Selects a number of columns in a given dataframe.

select ["name", "age"] df

O(n) select columns by column predicate name.

Criteria for selecting columns whose indices are in the given (inclusive) range.

selectBy [byIndexRange (0, 5)] df

Criteria for selecting a column by name.

selectBy [byName "Age"] df

equivalent to:

select ["Age"] df

Criteria for selecting columns whose name satisfies given predicate.

selectBy [byNameProperty (T.isPrefixOf "weight")] df

Criteria for selecting columns whose names are in the given lexicographic range (inclusive).

selectBy [byNameRange ("a", "c")] df

Criteria for selecting columns whose property satisfies given predicate.

selectBy [byProperty isNumeric] df

O(k) cuts the dataframe in a cube of size (a, b) where a is the length and b is the width.

cube (10, 5) df

O(k * n) Drop the last n rows of a DataFrame.

O(n) inverse of select

exclude ["Name"] df

O(n * k) removes all rows with Nothing from the dataframe.

filterAllJust df

O(k) a version of filter where the predicate comes first.

filterBy even "x" df

O(k) removes all rows with Nothing in a given column from the dataframe.

filterJust "col" df

O(k) returns all rows with Nothing in a give column.

filterNothing "col" df

O(k) filters the dataframe with a boolean expression.

filterWhere (F.col @Int x + F.col y F.> 5) df

Creates n folds of a dataframe.

Example

ghci> import System.Random
ghci> D.kFolds (mkStdGen 137) 5 df

Split a dataset into two. The first in the tuple gets a sample of p (0 <= p <= 1) and the second gets (1 - p). This is useful for creating test and train splits.

Example

ghci> import System.Random
ghci> D.randomSplit (mkStdGen 137) 0.9 df

Sample a dataframe. The double parameter must be between 0 and 1 (inclusive).

Example

ghci> import System.Random
ghci> D.sample (mkStdGen 137) 0.1 df

O(k * n) Take the last n rows of a DataFrame.

O(k * n) groups the dataframe by the given rows aggregating the remaining rows into vector that should be reduced later.

Aggregate a grouped dataframe using the expressions given. All ungrouped columns will be dropped.

Filter out all non-unique values in a dataframe.

Calculates the sum of a given column as a standalone value.

Calculates the Pearson's correlation coefficient between two given columns as a standalone value.

Show a frequency table for a categorical feaure.

Examples:

ghci> df <- D.readCsv "./data/housing.csv"

ghci> D.frequencies "ocean_proximity" df

----------------------------------------------------------------------------
index |   Statistic    | <1H OCEAN | INLAND | ISLAND | NEAR BAY | NEAR OCEAN
------|----------------|-----------|--------|--------|----------|-----------
 Int  |      Text      |    Any    |  Any   |  Any   |   Any    |    Any
------|----------------|-----------|--------|--------|----------|-----------
0     | Count          | 9136      | 6551   | 5      | 2290     | 2658
1     | Percentage (%) | 44.26%    | 31.74% | 0.02%  | 11.09%   | 12.88%

Calculates the inter-quartile range of a given column as a standalone value.

Calculates the mean of a given column as a standalone value.

Calculates the median of a given column as a standalone value.

Calculates the skewness of a given column as a standalone value.

Calculates the standard deviation of a given column as a standalone value.

Descriptive statistics of the numeric columns.

Calculates the variance of a given column as a standalone value.