http-conduit-downloader: HTTP downloader tailored for web-crawler needs.

HTTP/HTTPS downloader built on top of http-conduit and used in https://bazqux.com crawler.

• Handles all possible http-conduit exceptions and returns human readable error messages.

• Handles some web server bugs (returning deflate data instead of gzip, invalid gzip encoding).

• Uses OpenSSL instead of tls package since it doesn't handle all sites.

• Ignores invalid SSL sertificates.

• Receives data in 32k blocks internally to reduce memory fragmentation on many parallel downloads.

• Download timeout.

• Total download size limit.

• Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.

• Can be used with external DNS resolver (hsdns-cache for example).

Modules

[Index]

Flags

Automatic Flags

Name	Description	Default
network-uri	Get Network.URI from the network-uri package	Enabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

• http-conduit-downloader-1.0.22.tar.gz [browse] (Cabal source package)
• Package description (as included in the package)

Maintainer's Corner

Package maintainers

• VladimirShabanov

For package maintainers and hackage trustees

• edit package information

Candidates

• 1.0.31