O(1) Creates an empty dataframe

A record that contains information about how and what rows are grouped in the dataframe. This can only be used with aggregate.

Get a specific column as a vector.

You must specify the type via type applications.

Returns a dataframe as a two dimentions vector of floats.

All entries in the dataframe must be doubles. This is useful for handing data over into ML systems.

O(n) Convert a list to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> fromList [(1 :: Int), 2, 3, 4]
[1,2,3,4]

O(n) Converts a column to a list. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toList Int column
[1,2,3,4]
> toList Double column
exception: ...

O(n) Convert a vector to a column. Automatically picks the best representation of a vector to store the underlying data in.

Examples:

> import qualified Data.Vector as V
> fromVector (V.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

O(n) Converts a column to a boxed vector. Throws an exception if the wrong type is specified.

Examples:

> column = fromList [(1 :: Int), 2, 3, 4]
> toVector Int column
[1,2,3,4]
> toVector Double column
exception: ...

Our representation of a column is a GADT that can store data based on the underlying data.

This allows us to pattern match on data kinds and limit some operations to only some kinds of vectors. E.g. operations for missing data only happen in an OptionalColumn.

O(n) Convert an unboxed vector to a column. This avoids the extra conversion if you already have the data in an unboxed vector.

Examples:

> import qualified Data.Vector.Unboxed as V
> fromUnboxedVector (V.fromList [(1 :: Int), 2, 3, 4])
[1,2,3,4]

A left fold for dataframes that takes the dataframe as the last object. this makes it easier to chain operations.

Example

ghci> D.fold (const id) [1..5] df

-----------------
index |  0  |  1
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13
3     | 4   | 14
4     | 5   | 15
5     | 6   | 16
6     | 7   | 17
7     | 8   | 18
8     | 9   | 19
9     | 10  | 20

O(n) Renames a single column.

Example

ghci> import qualified Data.Vector as V

ghci> df = insertVector "numbers" (V.fromList [1..10]) D.empty

ghci> D.rename "numbers" "others" df

--------------
index | others
------|-------
 Int  |  Int
------|-------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(1) Get DataFrame dimensions i.e. (rows, columns)

Example

ghci> D.dimensions df

(100, 3)

O(k) Get column names of the DataFrame in order of insertion.

Example

ghci> D.columnNames df

["col_a", "col_b", "col_c"]

Adds a vector to the dataframe. If the vector has less elements than the dataframe and the dataframe is not empty the vector is converted to type `Maybe a` filled with Nothing to match the size of the dataframe. Similarly, if the vector has more elements than what's currently in the dataframe, the other columns in the dataframe are change to `Maybe Type` and filled with Nothing.

Example

ghci> import qualified Data.Vector as V

ghci> D.insertVector "numbers" (V.fromList [1..10]) D.empty

---------------
index | numbers
------|--------
 Int  |   Int
------|--------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(n) Add a column to the dataframe.

Example

ghci> D.insertColumn "numbers" (D.fromList [1..10]) D.empty

---------------
index | numbers
------|--------
 Int  |   Int
------|--------
0     | 1
1     | 2
2     | 3
3     | 4
4     | 5
5     | 6
6     | 7
7     | 8
8     | 9
9     | 10

O(k) Add a column to the dataframe providing a default. This constructs a new vector and also may convert it to an unboxed vector if necessary. Since columns are usually large the runtime is dominated by the length of the list, k.

O(n) Adds an unboxed vector to the dataframe.

Same as insertVector but takes an unboxed vector. If you insert a vector of numbers through insertVector it will either way be converted into an unboxed vector so this function saves that extra work/conversion.

O(n) Clones a column and places it under a new name in the dataframe.

Example

ghci> import qualified Data.Vector as V

ghci> df = insertVector "numbers" (V.fromList [1..10]) D.empty

ghci> D.cloneColumn "numbers" "others" df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 1
1     | 2       | 2
2     | 3       | 3
3     | 4       | 4
4     | 5       | 5
5     | 6       | 6
6     | 7       | 7
7     | 8       | 8
8     | 9       | 9
9     | 10      | 10

O(n) Renames many columns.

Example

ghci> import qualified Data.Vector as V

ghci> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)

ghci> df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 11
1     | 2       | 12
2     | 3       | 13
3     | 4       | 14
4     | 5       | 15
5     | 6       | 16
6     | 7       | 17
7     | 8       | 18
8     | 9       | 19
9     | 10      | 20

ghci> D.renameMany [("numbers", "first_10"), ("others", "next_10")] df

--------------------------
index | first_10 | next_10
------|----------|--------
 Int  |   Int    |   Int
------|----------|--------
0     | 1        | 11
1     | 2        | 12
2     | 3        | 13
3     | 4        | 14
4     | 5        | 15
5     | 6        | 16
6     | 7        | 17
7     | 8        | 18
8     | 9        | 19
9     | 10       | 20

O(n * k ^ 2) Returns the number of non-null columns in the dataframe and the type associated with each column.

Example

ghci> import qualified Data.Vector as V

ghci> df = D.insertVector "others" (V.fromList [11..20]) (D.insertVector "numbers" (V.fromList [1..10]) D.empty)

ghci> D.describeColumns df

-----------------------------------------------------------------------------------------------------
index | Column Name | # Non-null Values | # Null Values | # Partially parsed | # Unique Values | Type
------|-------------|-------------------|---------------|--------------------|-----------------|-----
 Int  |    Text     |        Int        |      Int      |        Int         |       Int       | Text
------|-------------|-------------------|---------------|--------------------|-----------------|-----
0     | others      | 10                | 0             | 0                  | 10              | Int
1     | numbers     | 10                | 0             | 0                  | 10              | Int

Creates a dataframe from a list of tuples with name and column.

Example

ghci> df = D.fromNamedColumns [("numbers", D.fromList [1..10]), ("others", D.fromList [11..20])]

ghci> df

------------------------
index | numbers | others
------|---------|-------
 Int  |   Int   |  Int
------|---------|-------
0     | 1       | 11
1     | 2       | 12
2     | 3       | 13
3     | 4       | 14
4     | 5       | 15
5     | 6       | 16
6     | 7       | 17
7     | 8       | 18
8     | 9       | 19
9     | 10      | 20

Create a dataframe from a list of columns. The column names are "0", "1"... etc. Useful for quick exploration but you should probably alwyas rename the columns after or drop the ones you don't want.

Example

ghci> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

ghci> df

-----------------
index |  0  |  1
------|-----|----
 Int  | Int | Int
------|-----|----
0     | 1   | 11
1     | 2   | 12
2     | 3   | 13
3     | 4   | 14
4     | 5   | 15
5     | 6   | 16
6     | 7   | 17
7     | 8   | 18
8     | 9   | 19
9     | 10  | 20

O (k * n) Counts the occurences of each value in a given column.

Example

ghci> df = D.fromUnnamedColumns [D.fromList [1..10], D.fromList [11..20]]

ghci> D.valueCounts @Int "0" df

[(1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1)]

CSV read parameters.

Read CSV file from path and load it into a dataframe.

Example

ghci> D.readCsv ".datataxi.csv" df

Read text file with specified delimiter into a dataframe.

Example

ghci> D.readSeparated ';' D.defaultOptions ".datataxi.txt" df

Read TSV (tab separated) file from path and load it into a dataframe.

Example

ghci> D.readTsv ".datataxi.tsv" df

Read a parquet file from path and load it into a dataframe.

Example

ghci> D.readParquet ".datamtcars.parquet" df

O(n * k) Filter rows by a given condition.

filter "x" even df

O(k * n) Take a range of rows of a DataFrame.

O(k * n) Take the first n rows of a DataFrame.

O(k * n) Drop the first n rows of a DataFrame.

O(n) Selects a number of columns in a given dataframe.

select ["name", "age"] df

O(n) select columns by column predicate name.

O(k) cuts the dataframe in a cube of size (a, b) where a is the length and b is the width.

cube (10, 5) df

O(k * n) Drop the last n rows of a DataFrame.

O(n) inverse of select

exclude ["Name"] df

O(n * k) removes all rows with Nothing from the dataframe.

filterJust df

O(k) a version of filter where the predicate comes first.

filterBy even "x" df

O(k) removes all rows with Nothing in a given column from the dataframe.

filterJust df

O(k) filters the dataframe with a row predicate. The arguments in the function must appear in the same order as they do in the list.

filterWhere (["x", "y"], func (\x y -> x + y > 5)) df

O(n) select columns by index range of column names.

O(n) select columns by index range of column names.

O(k * n) Take the last n rows of a DataFrame.

O(k) Apply a function to a given column in a dataframe.

O(k) Safe version of the apply function. Returns (instead of throwing) the error.

O(k) Apply a function to a combination of columns in a dataframe and add the result into alias column.

O(k * n) Apply a function to given column names in a dataframe.

O(k) Convenience function that applies to an int column.

O(k) Convenience function that applies to an double column.

O(k * n) Apply a function to a column only if there is another column value that matches the given criterion.

applyWhere "Age" (<20) "Generation" (const "Gen-Z")

O(k) Apply a function to the column at a given index.

Replace all instances of Nothing in a column with the given value.

O(k * n) groups the dataframe by the given rows aggregating the remaining rows into vector that should be reduced later.

Aggregate a grouped dataframe using the expressions give. All ungrouped columns will be dropped.

Filter out all non-unique values in a dataframe.

O(k log n) Sorts the dataframe by a given row.

sortBy "Age" df

Sort order taken as a parameter by the sortby function.

Add two dataframes side by side/horizontally.

Join two dataframes using SQL join semantics.

Only inner join is implemented for now.

Equivalent to SQL join types.

Inner join of two dataframes. Note: for chaining, the left dataframe is actually on the right side.

Calculates the sum of a given column as a standalone value.

Calculates the Pearson's correlation coefficient between two given columns as a standalone value.

Calculates the median of a given column as a standalone value.

Calculates the variance of a given column as a standalone value.

Calculates the mean of a given column as a standalone value.

Calculates the skewness of a given column as a standalone value.

Show a frequency table for a categorical feaure.

Examples:

ghci> df <- D.readCsv ".datahousing.csv"

ghci> D.frequencies "ocean_proximity" df

----------------------------------------------------------------------------
index |   Statistic    | <1H OCEAN | INLAND | ISLAND | NEAR BAY | NEAR OCEAN
------|----------------|-----------|--------|--------|----------|-----------
 Int  |      Text      |    Any    |  Any   |  Any   |   Any    |    Any
------|----------------|-----------|--------|--------|----------|-----------
0     | Count          | 9136      | 6551   | 5      | 2290     | 2658
1     | Percentage (%) | 44.26%    | 31.74% | 0.02%  | 11.09%   | 12.88%

Calculates the inter-quartile range of a given column as a standalone value.

Calculates the standard deviation of a given column as a standalone value.

Descriprive statistics of the numeric columns.