Working on HTTP client
I’ve been working pretty hard on an HTTP client in cl-async that supports streaming. The current implementation of the HTTP client in Libevent2 as it is only supports simple requests:
- Init request, set in headers (must be predefined)
- Send in body content, must be known and fully loaded into memory
- Fire off request, trigger callback when response has been parsed
This works fine for simpler things, but I’m currently porting Drakma to use
cl-async, and the easiest way to do this is to support streaming. I built a
stream implementation on top of the cl-async
socket type, which seems to work
great (check out the
stream branch in cl-async). The problem is that Drakma
has a hard time knowing when the response is fully loaded. I have three options:
- Reprogram the streaming implementation in Drakma (ie Chunga) to be async. This would require a lot of work and state management which, honestly, seems like too much work.
- Reprogram Drakma to not use streams. This severely limits the number of features that Drakma implements, some of which may be crucial to users of the library. I don’t really find this method acceptable, although it seems to be the easiest to implement.
- Hack libevent to support streaming in the HTTP client such that the entire HTTP request is streamed in (like how Drakma currently does it). Once a response comes through, libevent will parse it and fire the callback when it’s ready to be parsed, at which point Drakma can read from the stream.
I’ve decided to go with option #3. While fairly dangerous in that I have to
use a lot of “private” functions in the libevent
evhttp implementation, it
provides the most features for users of the library.
There are a lot of hurdles to this implementation though. Not only does
libevent not export some of the functions needed to do this, it actually
defines them as static, making it literally impossible to call them from lisp.
This means that I’m re-implementing a lot of existing C code in lisp/CFFI that
could easily be avoided with a simple
foreign-funcall if I had the option.
So, things right now are in an unknown state. I’m working my best to get a
working prototype out of the project. I think if worse comes to worse, I might
circumvent libevent’s HTTP client altogether, and implement an extremely simple
HTTP parser that basically takes an array of HTTP data and returns
t if the
response has finished downloading, and
nil if it’s still getting data. Then
once it’s finished, I can just stream the data into Drakma and it won’t even
know that the data loaded async.
Check out the latest version of the http-stream implementation
on the future+stream branch
of cl-async. Please note that
http-stream is completely broken! It can stream
data out, but doesn’t trigger any complete callbacks yet.