O(1) Creates an empty dataframe

A record that contains information about how and what rows are grouped in the dataframe. This can only be used with aggregate.

Get a specific column as a vector.

You must specify the type via type applications.

For showing the dataframe as markdown in notebooks.

Returns a dataframe as a two dimentions vector of floats.

All entries in the dataframe must be doubles. This is useful for handing data over into ML systems.

O(n) Convert a list to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> fromList [(1 :: Int), 2, 3, 4]
[1,2,3,4]

O(n) Converts a column to a list. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toList Int column
[1,2,3,4]
> toList Double column
exception: ...

Our representation of a column is a GADT that can store data based on the underlying data.

This allows us to pattern match on data kinds and limit some operations to only some kinds of vectors. E.g. operations for missing data only happen in an OptionalColumn.

O(n) Convert an unboxed vector to a column. This avoids the extra conversion if you already have the data in an unboxed vector.

Examples:

> import qualified Data.Vector.Unboxed as V
> fromUnboxedVector (V.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Convert a vector to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> import qualified Data.Vector as V
> fromVector (V.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

Checks if a column is of a given type values.

Checks if a column contains missing values.

Checks if a column contains numeric values.

A safe version of toVector that returns an Either type.

Unwraps a value from an Any type.

Wraps a value into an Any type. This helps up represent rows as heterogenous lists.

A left fold for dataframes that takes the dataframe as the last object. This makes it easier to chain operations.

Example

ghci> D.fold (const id) [1..5] df

-----------------
index |  0  |  1
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13
3     | 4   | 14
4     | 5   | 15
5     | 6   | 16
6     | 7   | 17
7     | 8   | 18
8     | 9   | 19
9     | 10  | 20

O(n) Renames a single column.

Example

ghci> import qualified Data.Vector as V

ghci> df = insertVector "numbers" (V.fromList [1..10]) D.empty

ghci> D.rename "numbers" "others" df

--------------
index | others
------|-------
 Int  |  Int
------|-------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(1) Get DataFrame dimensions i.e. (rows, columns)

Example

ghci> D.dimensions df

(100, 3)

O(k) Get column names of the DataFrame in order of insertion.

Example

ghci> D.columnNames df

["col_a", "col_b", "col_c"]

Adds a vector to the dataframe. If the vector has less elements than the dataframe and the dataframe is not empty the vector is converted to type `Maybe a` filled with Nothing to match the size of the dataframe. Similarly, if the vector has more elements than what's currently in the dataframe, the other columns in the dataframe are change to `Maybe Type` and filled with Nothing.

Example

ghci> import qualified Data.Vector as V

ghci> D.insertVector "numbers" (V.fromList [1..10]) D.empty

---------------
index | numbers
------|--------
 Int  |   Int
------|--------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(n) Add a column to the dataframe.

Example

ghci> D.insertColumn "numbers" (D.fromList [1..10]) D.empty

---------------
index | numbers
------|--------
 Int  |   Int
------|--------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(k) Add a column to the dataframe providing a default. This constructs a new vector and also may convert it to an unboxed vector if necessary. Since columns are usually large the runtime is dominated by the length of the list, k.

O(n) Adds an unboxed vector to the dataframe.

Same as insertVector but takes an unboxed vector. If you insert a vector of numbers through insertVector it will either way be converted into an unboxed vector so this function saves that extra work/conversion.

O(n) Clones a column and places it under a new name in the dataframe.

Example

ghci> import qualified Data.Vector as V

ghci> df = insertVector "numbers" (V.fromList [1..10]) D.empty

ghci> D.cloneColumn "numbers" "others" df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 1
1     | 2       | 2
2     | 3       | 3
3     | 4       | 4
4     | 5       | 5
5     | 6       | 6
6     | 7       | 7
7     | 8       | 8
8     | 9       | 9
9     | 10      | 10

O(n) Renames many columns.

Example

ghci> import qualified Data.Vector as V

ghci> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)

ghci> df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 11
1     | 2       | 12
2     | 3       | 13
3     | 4       | 14
4     | 5       | 15
5     | 6       | 16
6     | 7       | 17
7     | 8       | 18
8     | 9       | 19
9     | 10      | 20

ghci> D.renameMany [("numbers", "first_10"), ("others", "next_10")] df

--------------------------
index | first_10 | next_10
------|----------|--------
 Int  |   Int    |   Int
------|----------|--------
0     | 1        | 11
1     | 2        | 12
2     | 3        | 13
3     | 4        | 14
4     | 5        | 15
5     | 6        | 16
6     | 7        | 17
7     | 8        | 18
8     | 9        | 19
9     | 10       | 20

O(n * k ^ 2) Returns the number of non-null columns in the dataframe and the type associated with each column.

Example

ghci> import qualified Data.Vector as V

ghci> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)

ghci> D.describeColumns df

-----------------------------------------------------------------------------------------------------
index | Column Name | # Non-null Values | # Null Values | # Partially parsed | # Unique Values | Type
------|-------------|-------------------|---------------|--------------------|-----------------|-----
 Int  |    Text     |        Int        |      Int      |        Int         |       Int       | Text
------|-------------|-------------------|---------------|--------------------|-----------------|-----
0     | others      | 10                | 0             | 0                  | 10              | Int
1     | numbers     | 10                | 0             | 0                  | 10              | Int

Creates a dataframe from a list of tuples with name and column.

Example

ghci> df = D.fromNamedColumns [("numbers", D.fromList [1..10]), ("others", D.fromList [11..20])]

ghci> df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 11
1     | 2       | 12
2     | 3       | 13
3     | 4       | 14
4     | 5       | 15
5     | 6       | 16
6     | 7       | 17
7     | 8       | 18
8     | 9       | 19
9     | 10      | 20

Create a dataframe from a list of columns. The column names are "0", "1"... etc. Useful for quick exploration but you should probably always rename the columns after or drop the ones you don't want.

Example

ghci> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

ghci> df

-----------------
index |  0  |  1
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13
3     | 4   | 14
4     | 5   | 15
5     | 6   | 16
6     | 7   | 17
7     | 8   | 18
8     | 9   | 19
9     | 10  | 20

Create a dataframe from a list of column names and rows.

Example

ghci> df = D.fromRows [A, B] [[D.toAny 1, D.toAny 11], [D.toAny 2, D.toAny 12], [D.toAny 3, D.toAny 13]]

ghci> df

-----------------
index |  A  |  B
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13

O (k * n) Counts the occurences of each value in a given column.

Example

ghci> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

ghci> D.valueCounts @Int "0" df

[(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1)]

CSV read parameters.

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsv "./data/taxi.csv"

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsvWithOpts "./data/taxi.csv" (D.defaultReadOptions { dateFormat = "%d%-m%-Y" })

Read text file with specified delimiter into a dataframe.

Example

ghci> D.readSeparated ';' D.defaultReadOptions "./data/taxi.txt"

Read TSV (tab separated) file from path and load it into a dataframe.

Example

ghci> D.readTsv "./data/taxi.tsv"

Read a parquet file from path and load it into a dataframe.

Example

ghci> D.readParquet "./data/mtcars.parquet"

O(k * n) Take a range of rows of a DataFrame.

O(k * n) Take the first n rows of a DataFrame.

O(n * k) Filter rows by a given condition.

filter "x" even df

O(k * n) Drop the first n rows of a DataFrame.

O(n) Selects a number of columns in a given dataframe.

select ["name", "age"] df

O(n) select columns by column predicate name.

Criteria for selecting columns whose indices are in the given (inclusive) range.

selectBy [byIndexRange (0, 5)] df

Criteria for selecting a column by name.

selectBy [byName "Age"] df

equivalent to:

select ["Age"] df

Criteria for selecting columns whose name satisfies given predicate.

selectBy [byNameProperty (T.isPrefixOf "weight")] df

Criteria for selecting columns whose names are in the given lexicographic range (inclusive).

selectBy [byNameRange ("a", "c")] df

Criteria for selecting columns whose property satisfies given predicate.

selectBy [byProperty isNumeric] df

O(k) cuts the dataframe in a cube of size (a, b) where a is the length and b is the width.

cube (10, 5) df

O(k * n) Drop the last n rows of a DataFrame.

O(n) inverse of select

exclude ["Name"] df

O(n * k) removes all rows with Nothing from the dataframe.

filterAllJust df

O(k) a version of filter where the predicate comes first.

filterBy even "x" df

O(k) removes all rows with Nothing in a given column from the dataframe.

filterJust "col" df

O(k) returns all rows with Nothing in a give column.

filterNothing "col" df

O(k) filters the dataframe with a boolean expression.

filterWhere (F.col @Int x + F.col y F.> 5) df

Creates n folds of a dataframe.

Example

ghci> import System.Random
ghci> D.kFolds (mkStdGen 137) 5 df

Split a dataset into two. The first in the tuple gets a sample of p (0 <= p <= 1) and the second gets (1 - p). This is useful for creating test and train splits.

Example

ghci> import System.Random
ghci> D.randomSplit (mkStdGen 137) 0.9 df

Sample a dataframe. The double parameter must be between 0 and 1 (inclusive).

Example

ghci> import System.Random
ghci> D.sample (mkStdGen 137) 0.1 df

O(k * n) Take the last n rows of a DataFrame.

O(k * n) groups the dataframe by the given rows aggregating the remaining rows into vector that should be reduced later.

Aggregate a grouped dataframe using the expressions given. All ungrouped columns will be dropped.

Filter out all non-unique values in a dataframe.

Calculates the sum of a given column as a standalone value.

Calculates the Pearson's correlation coefficient between two given columns as a standalone value.

Show a frequency table for a categorical feaure.

Examples:

ghci> df <- D.readCsv "./data/housing.csv"

ghci> D.frequencies "ocean_proximity" df

----------------------------------------------------------------------------
index |   Statistic    | <1H OCEAN | INLAND | ISLAND | NEAR BAY | NEAR OCEAN
------|----------------|-----------|--------|--------|----------|-----------
 Int  |      Text      |    Any    |  Any   |  Any   |   Any    |    Any
------|----------------|-----------|--------|--------|----------|-----------
0     | Count          | 9136      | 6551   | 5      | 2290     | 2658
1     | Percentage (%) | 44.26%    | 31.74% | 0.02%  | 11.09%   | 12.88%

Calculates the inter-quartile range of a given column as a standalone value.

Calculates the mean of a given column as a standalone value.

Calculates the median of a given column as a standalone value.

Calculates the skewness of a given column as a standalone value.

Calculates the standard deviation of a given column as a standalone value.

Descriptive statistics of the numeric columns.

Calculates the variance of a given column as a standalone value.