Reader oriented encoder #210
Comments
|
Hello, it makes sense to have it. I currently dont have time to work on this but am open to PRs :). |
|
Kind of weird how http.Request requires reader for the body, yet Go supplied compressors, like gzip, have not support for handling it. Nobody cares about uploads? :-) |
oakad
added a commit
to oakad/lz4_issue_210
that referenced
this issue
Nov 13, 2023
In Go, `http.Request` requires an `io.ReadCloser` for an upload body. To facilitate compression of data being uploaded we want to wrap a source reader (such an `io.File`) with a proposed `lz4.CompressingReader`, whereupon the said reader can be set directly into `http.Request` Body field. This benefits not only the plain `http.Request`, but also the various network file storage APIs which build on top of Go's http client. Signed-off-by: Alex Dubov <[email protected]>
|
Added a pull request (#211) . Seems to work. |
pierrec
added a commit
that referenced
this issue
Dec 8, 2023
CompressingReader: compressed data reader stream (#210)
|
Closed as PR is merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now,
Writeris used for compression andReaderis used for decompression. However, some storage APIs, namely AWS S3 and kind, insist on having aReaderas a body of file to upload.S3 style APIs are incredibly popular and will stay with us for the indefinite future.
We, of course, can work around this limitation by means of a helper goroutine and an
io.Pipe, but this adds quite a bit of overhead when many objects are to be uploaded in compressed form.From the look of it, it should not be too difficult to make an
EncodingReaderobject, which takes a plain dataReaderas a parameter and exposes aReaderfor compressed data (that is, performs a logical opposite of the presently availablelz4.Readerobject).I think, such a feature will be most useful for people using cloud storage.
The text was updated successfully, but these errors were encountered: