Safe Haskell	None
Language	Haskell2010

DataFrame.IO.Parquet.Seeking

Description

This module contains low-level utilities around file seeking

potentially also contains all Streamly related low-level utilities.

later this module can be renamed / moved to an internal module.

Synopsis

data SeekableHandle
data SeekMode
- = AbsoluteSeek
- | RelativeSeek
- | SeekFromEnd
data FileBufferedOrSeekable
- = FileBuffered !(IORef Int64) !ByteString
- | FileSeekable !SeekableHandle
type ForceNonSeekable = Maybe Bool
advanceBytes :: Int -> FileBufferedOrSeekable -> IO ByteString
mkFileBufferedOrSeekable :: ForceNonSeekable -> Handle -> IO FileBufferedOrSeekable
mkSeekableHandle :: Handle -> IO (Maybe SeekableHandle)
readLastBytes :: Integer -> FileBufferedOrSeekable -> IO ByteString
seekAndReadBytes :: Maybe (SeekMode, Integer) -> Int -> FileBufferedOrSeekable -> IO ByteString
seekAndStreamBytes :: MonadIO m => Maybe (SeekMode, Integer) -> Int -> FileBufferedOrSeekable -> m (Stream m Word8)
withFileBufferedOrSeekable :: ForceNonSeekable -> FilePath -> IOMode -> (FileBufferedOrSeekable -> IO a) -> IO a

Documentation

data SeekableHandle Source #

This handle carries a proof that it must be seekable. Note: Handle and SeekableHandle are not thread safe, should not be shared across threads, beaware when running parallel/concurrent code.

Not seekable: - stdin / stdout - pipes / FIFOs

But regular files are always seekable. Parquet fundamentally wants random access, a non-seekable source will not support effecient access without buffering the entire file.

data SeekMode #

A mode that determines the effect of hSeek hdl mode i.

Constructors

AbsoluteSeek	the position of `hdl` is set to `i`.
RelativeSeek	the position of `hdl` is set to offset `i` from the current position.
SeekFromEnd	the position of `hdl` is set to offset `i` from the end of the file.

Instances

Instances details

Enum SeekMode	Since: base-4.2.0.0
Instance details Defined in GHC.IO.Device Methods succ :: SeekMode -> SeekMode # pred :: SeekMode -> SeekMode # toEnum :: Int -> SeekMode # fromEnum :: SeekMode -> Int # enumFrom :: SeekMode -> [SeekMode] # enumFromThen :: SeekMode -> SeekMode -> [SeekMode] # enumFromTo :: SeekMode -> SeekMode -> [SeekMode] # enumFromThenTo :: SeekMode -> SeekMode -> SeekMode -> [SeekMode] #
Ix SeekMode	Since: base-4.2.0.0
Instance details Defined in GHC.IO.Device Methods range :: (SeekMode, SeekMode) -> [SeekMode] # index :: (SeekMode, SeekMode) -> SeekMode -> Int # unsafeIndex :: (SeekMode, SeekMode) -> SeekMode -> Int # inRange :: (SeekMode, SeekMode) -> SeekMode -> Bool # rangeSize :: (SeekMode, SeekMode) -> Int # unsafeRangeSize :: (SeekMode, SeekMode) -> Int #
Read SeekMode	Since: base-4.2.0.0
Instance details Defined in GHC.IO.Device Methods readsPrec :: Int -> ReadS SeekMode # readList :: ReadS [SeekMode] # readPrec :: ReadPrec SeekMode # readListPrec :: ReadPrec [SeekMode] #
Show SeekMode	Since: base-4.2.0.0
Instance details Defined in GHC.IO.Device Methods showsPrec :: Int -> SeekMode -> ShowS # show :: SeekMode -> String # showList :: [SeekMode] -> ShowS #
Eq SeekMode	Since: base-4.2.0.0
Instance details Defined in GHC.IO.Device Methods (==) :: SeekMode -> SeekMode -> Bool # (/=) :: SeekMode -> SeekMode -> Bool #
Ord SeekMode	Since: base-4.2.0.0
Instance details Defined in GHC.IO.Device Methods compare :: SeekMode -> SeekMode -> Ordering # (<) :: SeekMode -> SeekMode -> Bool # (<=) :: SeekMode -> SeekMode -> Bool # (>) :: SeekMode -> SeekMode -> Bool # (>=) :: SeekMode -> SeekMode -> Bool # max :: SeekMode -> SeekMode -> SeekMode # min :: SeekMode -> SeekMode -> SeekMode #

data FileBufferedOrSeekable Source #

If we truely want to support non-seekable files, we need to also consider the case to buffer the entire file in memory.

Not thread safe, contains mutable reference (as Handle already is).

If we need concurrent / parallel parsing or something, we need to read into ByteString first, not sharing the same handle.

Constructors

FileBuffered !(IORef Int64) !ByteString
FileSeekable !SeekableHandle

type ForceNonSeekable = Maybe Bool Source #

For testing only

advanceBytes :: Int -> FileBufferedOrSeekable -> IO ByteString Source #

Note: this does not guarantee n bytes (if it ends early)

mkFileBufferedOrSeekable :: ForceNonSeekable -> Handle -> IO FileBufferedOrSeekable Source #

Smart constructor for FileBufferedOrSeekable, tries to keep in the seekable case if possible.

mkSeekableHandle :: Handle -> IO (Maybe SeekableHandle) Source #

Smart constructor for SeekableHandle

readLastBytes :: Integer -> FileBufferedOrSeekable -> IO ByteString Source #

Read from the end, useful for reading metadata without loading entire file

seekAndReadBytes :: Maybe (SeekMode, Integer) -> Int -> FileBufferedOrSeekable -> IO ByteString Source #

Note: this does not guarantee n bytes (if it ends early)

seekAndStreamBytes :: MonadIO m => Maybe (SeekMode, Integer) -> Int -> FileBufferedOrSeekable -> m (Stream m Word8) Source #

Warning: the stream produced from this function accesses to the mutable handler. if multiple streams are pulled from the same handler at the same time, chaos happen. Make sure there is only one stream running at one time for each SeekableHandle, and streams are not read again when they are not used anymore.

withFileBufferedOrSeekable :: ForceNonSeekable -> FilePath -> IOMode -> (FileBufferedOrSeekable -> IO a) -> IO a Source #

With / bracket pattern for FileBufferedOrSeekable

Warning: do not return the FileBufferedOrSeekable outside the scope of the action as it will be closed.