Safe Haskell	None
Language	Haskell2010

DataFrame.Internal.ColumnMerge

Description

Concatenation of per-chunk Columns (e.g. from parallel CSV chunks). Text columns merge at the byte level via TextChunk / mergeTextChunks, so no per-chunk Text values are ever materialized.

Synopsis

data TextChunk = TextChunk {
- tcBytes :: !Array
- tcUsed :: !Int
- tcOffsets :: !(Vector Int)
- tcBitmap :: !(Maybe Bitmap)
}
mergeColumns :: [Column] -> Column
mergeTextChunks :: [TextChunk] -> Column
packedFromTextChunk :: TextChunk -> Column
packValidity :: Int -> MVector s Word8 -> ST s Bitmap
spliceBitmaps :: [(Maybe Bitmap, Int)] -> Maybe Bitmap
tcRows :: TextChunk -> Int

Documentation

data TextChunk Source #

A frozen text-builder chunk: raw UTF-8 bytes plus row offsets (row i spans bytes [offsets!i, offsets!(i+1))) and an optional validity bitmap. Text values are only created when chunks merge into a Column.

Constructors

TextChunk
Fields tcBytes :: !Array tcUsed :: !Int tcOffsets :: !(Vector Int) tcBitmap :: !(Maybe Bitmap)

mergeColumns :: [Column] -> Column Source #

Merge per-chunk columns into one column: one allocation + memcpy per payload, with bitmaps spliced across non-byte-aligned chunk boundaries. All chunks must have the same element type.

mergeTextChunks :: [TextChunk] -> Column Source #

Merge text chunks into one packed-text Column: one byte-array copy per chunk, one offset rebase, then wrap the shared buffer + offsets as PackedText (no per-row header, decode deferred).

packedFromTextChunk :: TextChunk -> Column Source #

Freeze a builder chunk directly into a packed-text column: no Text materialization, no UTF-8 validation pass (deferred to decode). Not yet called by any reader.

packValidity :: Int -> MVector s Word8 -> ST s Bitmap Source #

Pack a 0/1 byte-per-row validity prefix into a bit-packed Bitmap.

spliceBitmaps :: [(Maybe Bitmap, Int)] -> Maybe Bitmap Source #

Splice chunk bitmaps end to end at the bit level. Nothing if no chunk carries a bitmap; chunks without one count as all-valid otherwise.

tcRows :: TextChunk -> Int Source #