conduit

Streaming data processing library. conduit is a solution to the streaming data problem, allowing for production, transformation, and consumption of streams of data in constant memory. It is an alternative to lazy I/O which guarantees deterministic resource handling. For more information about conduit in general, and how this package in particular fits into the ecosystem, see the conduit homepage. Hackage documentation generation is not reliable. For up to date documentation, please see: http://www.stackage.org/package/conduit.
Lift a conduit into a segment.
Your intended one-stop-shop for conduit functionality. This re-exports functions from many commonly used modules. When there is a conflict with standard functions, functions in this module are disambiguated by adding a trailing C (or for chunked functions, replacing a trailing E with CE). This means that the Conduit module can be imported unqualified without causing naming conflicts. For more information on the naming scheme and intended usages of the combinators, please see the Data.Conduit.Combinators documentation.
If this is your first time with conduit, you should probably start with the tutorial: https://github.com/snoyberg/conduit#readme.
Deprecated: Use ConduitT directly
A new, experimental API to replace Network.HTTP.Conduit. For most users, Network.HTTP.Simple is probably a better choice. For more information, see: https://haskell-lang.org/library/http-client For more information on using this module, please be sure to read the documentation in the Network.HTTP.Client module.

Simpler API

The API below is rather low-level. The Network.HTTP.Simple module provides a higher-level API with built-in support for things like JSON request and response bodies. For most users, this will be an easier place to start. You can read the tutorial at: https://haskell-lang.org/library/http-client

Lower-level API

This module contains everything you need to initiate HTTP connections. If you want a simple interface based on URLs, you can use simpleHttp. If you want raw power, http is the underlying workhorse of this package. Some examples:
-- Just download an HTML document and print it.
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L

main = simpleHttp "http://www.haskell.org/" >>= L.putStr
This example uses interleaved IO to write the response body to a file in constant memory space.
import Data.Conduit.Binary (sinkFile) -- Exported from the package conduit-extra
import Network.HTTP.Conduit
import Conduit (runConduit, (.|))
import Control.Monad.Trans.Resource (runResourceT)

main :: IO ()
main = do
request <- parseRequest "http://google.com/"
manager <- newManager tlsManagerSettings
runResourceT $ do
response <- http request manager
runConduit $ responseBody response .| sinkFile "google.html"
The following headers are automatically set by this module, and should not be added to requestHeaders:
  • Cookie
  • Content-Length
  • Transfer-Encoding
Note: In previous versions, the Host header would be set by this module in all cases. Starting from 1.6.1, if a Host header is present in requestHeaders, it will be used in place of the header this module would have generated. This can be useful for calling a server which utilizes virtual hosting. Use cookieJar If you want to supply cookies with your request:
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import Network
import Data.Time.Clock
import Data.Time.Calendar
import qualified Control.Exception as E
import Network.HTTP.Types.Status (statusCode)

past :: UTCTime
past = UTCTime (ModifiedJulianDay 56200) (secondsToDiffTime 0)

future :: UTCTime
future = UTCTime (ModifiedJulianDay 562000) (secondsToDiffTime 0)

cookie :: Cookie
cookie = Cookie { cookie_name = "password_hash"
, cookie_value = "abf472c35f8297fbcabf2911230001234fd2"
, cookie_expiry_time = future
, cookie_domain = "example.com"
, cookie_path = "/"
, cookie_creation_time = past
, cookie_last_access_time = past
, cookie_persistent = False
, cookie_host_only = False
, cookie_secure_only = False
, cookie_http_only = False
}

main = do
request' <- parseRequest "http://example.com/secret-page"
manager <- newManager tlsManagerSettings
let request = request' { cookieJar = Just $ createCookieJar [cookie] }
fmap Just (httpLbs request manager) `E.catch`
(\ex -> case ex of
HttpExceptionRequest _ (StatusCodeException res _) ->
if statusCode (responseStatus res) == 403
then (putStrLn "login failed" >> return Nothing)
else return Nothing
_ -> E.throw ex)
Cookies are implemented according to RFC 6265. Note that by default, the functions in this package will throw exceptions for non-2xx status codes. If you would like to avoid this, you should use checkStatus, e.g.:
import Data.Conduit.Binary (sinkFile)
import Network.HTTP.Conduit
import qualified Data.Conduit as C
import Network

main :: IO ()
main = do
request' <- parseRequest "http://www.yesodweb.com/does-not-exist"
let request = request' { checkStatus = \_ _ _ -> Nothing }
manager <- newManager tlsManagerSettings
res <- httpLbs request manager
print res
By default, when connecting to websites using HTTPS, functions in this package will throw an exception if the TLS certificate doesn't validate. To continue the HTTPS transaction even if the TLS cerficate validation fails, you should use mkManagerSetttings as follows:
import Network.Connection (TLSSettings (..))
import Network.HTTP.Conduit

main :: IO ()
main = do
request <- parseRequest "https://github.com/"
let settings = mkManagerSettings (TLSSettingsSimple True False False) Nothing
manager <- newManager settings
res <- httpLbs request manager
print res
For more information, please be sure to read the documentation in the Network.HTTP.Client module.
A component of a conduit pipeline, which takes a stream of input, produces a stream of output, performs actions in the underlying monad, and produces a value of result when no more output data is available.
A light-weight wrapper around Network.Wai to provide easy conduit support.
Consumes a stream of input values and produces a stream of output values, without producing a final result. Since 0.5.0
More efficient query execution functions for beam-postgres. These functions use the conduit package, to execute beam-postgres statements in an arbitrary MonadIO. These functions may be more efficient for streaming operations than MonadBeam.
A module containing Conduit facilities for hash based functions. this module is vaguely similar to the crypto-conduit part related to hash on purpose, as to provide an upgrade path. The api documentation is pulled directly from this package and adapted, and thus are originally copyright Felipe Lessa.
A module containing Conduit facilities for hmac based functions.
The module extends functionality available in Network.HTTP.Req with Conduit helpers for streaming big request bodies. The package re-uses some pieces of code from the http-conduit package, but not to the extent that depending on that package becomes reasonable.
This module exports ToSourceIO and FromSourceIO for ConduitT instances.
Mix NQE processes with conduits for easy concurrent IO.
Conduit interface for cassava
Help Wanted / TODOs Please feel free to send me a pull request for any of the following items:
  • TODO Block checksumming
  • TODO Dictionary support
  • TODO Performance: Write a version of compress that emits ByteStrings of known constant length. That will allow us to do compression in a zero-copy fashion, writing compressed bytes directly into a the ByteStrings (e.g using unsafePackMallocCString or equivalent). We currently don't do that (instead, use allocaBytes + copying packCStringLen) to ensure that the ByteStrings generated are as compact as possible (for the case that `written < size`), since the current compress conduit directly yields the outputs of LZ4F_compressUpdate() (unless they are of 0 length when they are buffered in the context tmp buffer).
  • TODO Try enabling checksums, then corrupt a bit and see if lz4c detects it.
  • TODO Add `with*` style bracketed functions for creating the LZ4F_createCompressionContext and Lz4FramePreferencesPtr for prompt resource release, in addition to the GC'd variants below. This would replace our use of finalizeForeignPtr in the conduit. finalizeForeignPtr seems almost as good, but note that it doesn't guarantee prompt resource release on exceptions; a `with*` style function that uses bracket does. However, it isn't clear yet which one would be faster (what the cost of mask is compared to foreign pointer finalizers). Also note that prompt freeing has side benefits, such as reduced malloc() fragmentation (the closer malloc() and free() are to each other, the smaller is the chance to have malloc()s on top of the our malloc() in the heap, thus the smaller the chance that we cannot decrease the heap pointer upon free() (because "mallocs on top" render heap memory unreturnable to the OS; memory fragmentation).
Streaming (de)serialization and encode-decode functions for the IDX format used in the MNIST handwritten digit recognition dataset [1]. Both sparse and dense decoders are provided. In either case, the range of the data is the same as the raw data (one unsigned byte per pixel).

Links

1) http://yann.lecun.com/exdb/mnist/
This module provides Conduit wrapper for Cisco Webex Teams list APIs.