dataframe-core-2.2.0.0: Core data structures for the dataframe library.
Safe HaskellNone
LanguageHaskell2010

DataFrame.Internal.Utf8

Description

UTF-8 validation and decodeUtf8Lenient-parity slice decoding used by ColumnBuilder to turn shared byte buffers into Text.

Synopsis

Documentation

isValidUtf8Slice :: Array -> Int -> Int -> Bool Source #

Whether len bytes starting at off are well-formed UTF-8.

isUtf8Boundary :: Word8 -> Bool Source #

Whether a byte may start a code point (i.e. is not a continuation byte). Field slices of a valid buffer are themselves valid iff every field starts on a boundary.

lenientDecodeSlice :: Array -> Int -> Int -> Text Source #

Decode a byte slice exactly like decodeUtf8Lenient: greedy decode at each position; any byte that cannot begin a complete, valid sequence within the slice becomes one U+FFFD and decoding resumes at the next byte.

sliceTextVector :: Array -> Vector Int -> Vector Text Source #

Slice forced Text values off a shared array; row i spans bytes [offs!i, offs!(i+1)). Fast path validates the whole span once when every field starts on a code-point boundary; else per-field validation with lenient decode.