conduit -package:http-conduit
Streaming data processing library.
conduit is a solution to the streaming data problem, allowing
for production, transformation, and consumption of streams of data in
constant memory. It is an alternative to lazy I/O which guarantees
deterministic resource handling.
For more information about conduit in general, and how this package in
particular fits into the ecosystem, see
the conduit homepage.
Hackage documentation generation is not reliable. For up to date
documentation, please see:
http://www.stackage.org/package/conduit.
Lift a conduit into a segment.
Your intended one-stop-shop for conduit functionality. This re-exports
functions from many commonly used modules. When there is a conflict
with standard functions, functions in this module are disambiguated by
adding a trailing C (or for chunked functions, replacing a trailing E
with CE). This means that the Conduit module can be imported
unqualified without causing naming conflicts.
For more information on the naming scheme and intended usages of the
combinators, please see the
Data.Conduit.Combinators
documentation.
Deprecated: Use ConduitT directly
A component of a conduit pipeline, which takes a stream of
input, produces a stream of output, performs actions
in the underlying monad, and produces a value of
result when no more output data is available.
A light-weight wrapper around Network.Wai to provide easy
conduit support.
Consumes a stream of input values and produces a stream of output
values, without producing a final result.
Since 0.5.0
More efficient query execution functions for
beam-postgres.
These functions use the
conduit package, to execute
beam-postgres statements in an arbitrary
MonadIO.
These functions may be more efficient for streaming operations than
MonadBeam.
A module containing Conduit facilities for hash based functions.
this module is vaguely similar to the crypto-conduit part related to
hash on purpose, as to provide an upgrade path. The api documentation
is pulled directly from this package and adapted, and thus are
originally copyright Felipe Lessa.
A module containing Conduit facilities for hmac based functions.
The module extends functionality available in
Network.HTTP.Req
with Conduit helpers for streaming big request bodies.
The package re-uses some pieces of code from the
http-conduit
package, but not to the extent that depending on that package becomes
reasonable.
Mix NQE processes with conduits for easy concurrent IO.
Conduit interface for cassava
Help Wanted / TODOs
Please feel free to send me a pull request for any of the following
items:
- TODO Block checksumming
- TODO Dictionary support
- TODO Performance: Write a version of compress that emits
ByteStrings of known constant length. That will allow us to do
compression in a zero-copy fashion, writing compressed bytes directly
into a the ByteStrings (e.g using unsafePackMallocCString or
equivalent). We currently don't do that (instead, use allocaBytes +
copying packCStringLen) to ensure that the ByteStrings generated are
as compact as possible (for the case that `written < size`), since
the current compress conduit directly yields the outputs of
LZ4F_compressUpdate() (unless they are of 0 length when they are
buffered in the context tmp buffer).
- TODO Try enabling checksums, then corrupt a bit and see if lz4c
detects it.
- TODO Add `with*` style bracketed functions for creating the
LZ4F_createCompressionContext and Lz4FramePreferencesPtr for prompt
resource release, in addition to the GC'd variants below. This would
replace our use of finalizeForeignPtr in the conduit.
finalizeForeignPtr seems almost as good, but note that it
doesn't guarantee prompt resource release on exceptions; a `with*`
style function that uses bracket does. However, it isn't clear
yet which one would be faster (what the cost of mask is
compared to foreign pointer finalizers). Also note that prompt freeing
has side benefits, such as reduced malloc() fragmentation (the closer
malloc() and free() are to each other, the smaller is the chance to
have malloc()s on top of the our malloc() in the heap, thus the
smaller the chance that we cannot decrease the heap pointer upon
free() (because "mallocs on top" render heap memory unreturnable to
the OS; memory fragmentation).
Streaming (de)serialization and encode-decode functions for the IDX
format used in the MNIST handwritten digit recognition dataset [1].
Both sparse and dense decoders are provided. In either case, the range
of the data is the same as the raw data (one unsigned byte per pixel).
Links
1)
http://yann.lecun.com/exdb/mnist/
This module provides Conduit wrapper for Cisco Webex Teams list APIs.
Break up a stream of values into vectors of size n. The final vector
may be smaller than n if the total number of values is not a strict
multiple of n. No empty vectors will be yielded.