conduit package:lz4-frame-conduit

Help Wanted / TODOs Please feel free to send me a pull request for any of the following items:

• TODO Block checksumming
• TODO Dictionary support
• TODO Performance: Write a version of compress that emits ByteStrings of known constant length. That will allow us to do compression in a zero-copy fashion, writing compressed bytes directly into a the ByteStrings (e.g using unsafePackMallocCString or equivalent). We currently don't do that (instead, use allocaBytes + copying packCStringLen) to ensure that the ByteStrings generated are as compact as possible (for the case that `written < size`), since the current compress conduit directly yields the outputs of LZ4F_compressUpdate() (unless they are of 0 length when they are buffered in the context tmp buffer).
• TODO Try enabling checksums, then corrupt a bit and see if lz4c detects it.
• TODO Add `with*` style bracketed functions for creating the LZ4F_createCompressionContext and Lz4FramePreferencesPtr for prompt resource release, in addition to the GC'd variants below. This would replace our use of finalizeForeignPtr in the conduit. finalizeForeignPtr seems almost as good, but note that it doesn't guarantee prompt resource release on exceptions; a `with*` style function that uses bracket does. However, it isn't clear yet which one would be faster (what the cost of mask is compared to foreign pointer finalizers). Also note that prompt freeing has side benefits, such as reduced malloc() fragmentation (the closer malloc() and free() are to each other, the smaller is the chance to have malloc()s on top of the our malloc() in the heap, thus the smaller the chance that we cannot decrease the heap pointer upon free() (because "mallocs on top" render heap memory unreturnable to the OS; memory fragmentation).

Conduit implementing the official LZ4 frame streaming format Conduit implementing the official LZ4 frame streaming format. Compatible with the lz4 command line utility.