Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

HTTP Parser

Incremental HTTP/1.x parser used by the HTTP gclasses. Feeds bytes from a gbuffer_t and invokes callbacks on each complete header, body chunk, and message.

Source code:

Limits

A parser holds the head of a message (the request target or the status text, and the names and values of its headers) and, with on_message_event, its body. Both have a limit, set in the struct after ghttp_parser_create():

FieldDefault (0)What it limits
max_header_sizeGHTTP_MAX_HEADER_SIZE, 64 KBThe bytes of the target or status text, the header names and the values of one message, trailers too, plus GHTTP_HEADER_COST (64) for each header. Never more than half the max block.
max_body_sizehalf the max block (with on_message_event), or no limit (with on_body_event alone, which holds nothing)The bytes of the body of one message. Never more than half the max block with on_message_event.

The cost of a header is what it holds in memory beyond its text, which an empty header holds too: without it, 240 KB of a:\r\n on the wire held 9 MB for as long as the head stayed unfinished. With the default limit a message carries at most 1008 headers of a one byte name and no value: (65536 - 1) / (1 + 64), the 1 being the target /. A real head is not near it: 30 headers cost 1920 bytes.

A message over a limit is refused: ONE warning (“HTTP headers larger than their limit, message refused” with header_received, max_header_size and the name of the header, or “HTTP body larger than its limit, message refused” with body_received, max_body_size), headers_refused or body_refused set, and ghttp_parser_received() answers -1. What then:

The head is put together in linear time: each byte is copied once, into a buffer that grows by doubling, and a header goes into jn_headers once, when its value ends, without the OWS (spaces and tabs) around it (RFC 9110 §5.5): Upgrade: websocket \t is "websocket", and X-A: a b stays "a b". Up to 7.30.15 the trailing OWS was kept. A header with an empty value, or of OWS only, is kept as "". A value that is not utf-8 is left out, with ONE warning per message however many there are (“cannot store header value (invalid utf-8?), the first one of the message”). A name that comes again is one field, its values joined in their order with ", " (RFC 9110 §5.3): two Retry-After: 120 and Retry-After: 60 are "120, 60", which a reader of one value refuses instead of taking the last. Those of COOKIE are joined with "; ", what a cookie parser splits on. SET-COOKIE is never joined (its values hold commas, RFC 6265): the last one is kept. The chunked trailers enter jn_headers with the same rules, so headers is a dict of strings in every kw, a refused response too. The fields a trailer must not carry (RFC 9110 §6.5.1) do not enter: Host, Origin, Connection, Upgrade, Trailer, the authentication and cookie fields, Content-Type, Content-Encoding, Content-Range, Cache-Control, Expect, Max-Forwards, Pragma, Range, TE, Location and Retry-After. They are left out with ONE warning per message (“HTTP field that a trailer must not carry left out, the first one of the message”), so a trailer Content-Type: application/json never turns a text/plain body into json. A trailer Content-Length or Transfer-Encoding is a protocol violation of llhttp, and the whole message is refused. Up to 7.29.3 a trailer replaced the header of its name. Example:

POST /x HTTP/1.1
Host: h
X-A: one
Transfer-Encoding: chunked

2
ok
0
Host: evil
X-A: two
X-A: three

leaves {"HOST": "h", "X-A": "one, two, three", ...}, with ONE warning for the Host of the trailers.

Each message starts clean. With on_message_event the state is released when it is published. Without it (raw_body_data, the handshake of a websocket) url, jn_headers and message_completed stay readable until the next message begins. The limits are per message: two heads of 40 KB on one keep-alive connection pass.

GHTTP_PARSER *parser = ghttp_parser_create(gobj, HTTP_REQUEST, 0, 0, EV_ON_MESSAGE, TRUE);
parser->max_header_size = 16*1024;     // a head of 16 KB at most
parser->max_body_size = 1024*1024;     // a body of 1 MB at most

if(ghttp_parser_received(parser, bf, len) < 0) {
    if(parser->headers_refused) {
        /* answer 431, then drop */
    } else if(parser->body_refused) {
        /* answer 413, then drop */
    } else {
        /* a protocol violation: drop */
    }
}

The kw of on_message_event:

KeyTypeDescription
http_parser_typeintHTTP_REQUEST or HTTP_RESPONSE.
urlstrThe target of a request.
response_status_codeintThe status of a response.
request_methodintThe method of a request (llhttp_method_t).
headersdictThe headers, the names in upper case (CONTENT-TYPE).
body_sizeintThe bytes of the body.
bodyjsonThe body, for application/json, or "" when there is no body.
gbufferintThe body in a gbuffer_t *, for any other type.
body_too_large, headers_too_largeintA refused response: the limit it passed (no body nor gbuffer).
protocol_errorstrA response that broke the protocol after its status line (no body nor gbuffer).

ghttp_parser_create()

Creates and initializes a new GHTTP_PARSER instance for parsing HTTP messages. The parser processes incoming HTTP data and triggers events when headers, body, or complete messages are received.

GHTTP_PARSER *ghttp_parser_create(
    hgobj gobj,
    llhttp_type_t type,
    gobj_event_t on_header_event,
    gobj_event_t on_body_event,
    gobj_event_t on_message_event,
    BOOL send_event
);

Parameters

KeyTypeDescription
gobjhgobjThe parent GObj that will handle the parsed HTTP events.
typeenum http_parser_typeThe type of HTTP parser (HTTP_REQUEST, HTTP_RESPONSE, or HTTP_BOTH).
on_header_eventgobj_event_tThe event triggered when HTTP headers are fully received.
on_body_eventgobj_event_tThe event triggered when a portion of the HTTP body is received.
on_message_eventgobj_event_tThe event triggered when the entire HTTP message is received.
send_eventBOOLIf TRUE, events are sent using gobj_send_event(). Otherwise, they are published using gobj_publish_event().

Returns

A pointer to the newly created GHTTP_PARSER instance, or NULL if memory allocation fails.

Notes

The returned GHTTP_PARSER instance must be destroyed using ghttp_parser_destroy() when no longer needed.


API break (7.3.0): ghttp_parser_reset() was removed from the public API. Calling it from inside an llhttp callback (for example, on_message_complete) corrupted llhttp’s internal state machine and silently swallowed pipelined HTTP/1.1 messages. Callers that need a fresh parser for a new connection must destroy the current parser and create a new one via ghttp_parser_create() — see c_prot_http_sr::ac_connected, c_prot_http_cl::ac_connected and c_websocket::ac_connected for the canonical pattern.


ghttp_parser_destroy()

Releases all resources associated with a GHTTP_PARSER instance, including allocated memory and internal buffers.

void ghttp_parser_destroy(GHTTP_PARSER *parser);

Parameters

KeyTypeDescription
parserGHTTP_PARSER *Pointer to the GHTTP_PARSER instance to be destroyed.

Returns

This function does not return a value.

Notes

After calling ghttp_parser_destroy(), the parser pointer must not be used as it becomes invalid.


ghttp_parser_received()

Parses an HTTP message from the given buffer using the ghttp_parser_create() instance.

int ghttp_parser_received(
    GHTTP_PARSER *parser,
    char *bf,
    size_t len
);

Parameters

KeyTypeDescription
parserGHTTP_PARSER *Pointer to the GHTTP_PARSER instance handling the HTTP parsing.
bfchar *Pointer to the buffer containing the HTTP message data.
lensize_tNumber of bytes available in bf for parsing.

Returns

The bytes consumed: all of len, or with an upgrade request (HPE_PAUSED_UPGRADE) the bytes up to its blank line (the rest belongs to the new protocol). A stopped parser (ghttp_parser_stop()) answers len and parses none. -1 on a protocol violation (one warning, “Protocol violation: non-HTTP data received”, with a dump of at most 256 bytes) or a message over a limit (headers_refused, body_refused, see Limits): the caller drops the connection.

Notes

It runs llhttp_execute() over the bytes, and the events of the parser are published from inside it.

int n = ghttp_parser_received(parser, gbuffer_cur_rd_pointer(gbuf), gbuffer_leftbytes(gbuf));
if(n < 0) {
    gobj_send_event(gobj_bottom_gobj(gobj), EV_DROP, 0, gobj);  // already logged
}

ghttp_parser_finish()

Signals end-of-stream to the underlying llhttp parser (the TCP peer closed the socket). Required to complete HTTP messages whose terminator is the connection close itself — HTTP/1.0 responses without Content-Length, and HTTP/1.1 responses with neither Content-Length nor Transfer-Encoding: chunked. Without this call, on_message_complete will never fire for such messages and the subscriber will miss EV_ON_MESSAGE.

int ghttp_parser_finish(GHTTP_PARSER *parser);

Parameters

KeyTypeDescription
parserGHTTP_PARSER *Pointer to the GHTTP_PARSER instance.

Returns

0 on success, -1 on protocol error (for example a partial message was in flight when the peer closed).

Notes

Call it from the gobj’s disconnect handler (ac_disconnected / mt_stop). Called inside an event published by the parser (the subscriber dropped the connection, and the disconnection came at once), it is ghttp_parser_stop() and answers 0.

PRIVATE int ac_disconnected(hgobj gobj, gobj_event_t event, json_t *kw, hgobj src)
{
    PRIVATE_DATA *priv = gobj_priv_data(gobj);
    (void)ghttp_parser_finish(priv->parsing_response);  // an EOF-terminated response ends here
    ...
}

ghttp_parser_stop()

The connection of the parser ends (it is dropped, or it closed): nothing more is delivered, not even the rest of the bytes being parsed now. Requests pipelined in one read after the one whose subscriber dropped the connection are not published.

void ghttp_parser_stop(GHTTP_PARSER *parser);

Parameters

KeyTypeDescription
parserGHTTP_PARSER *The parser. NULL is ignored.

Returns

This function does not return a value.

Notes

Safe inside an llhttp callback. After it, ghttp_parser_received() takes every byte and parses none, and ghttp_parser_finish() does nothing. A new connection needs a new parser.

PRIVATE int ac_drop(hgobj gobj, gobj_event_t event, json_t *kw, hgobj src)
{
    PRIVATE_DATA *priv = gobj_priv_data(gobj);
    ghttp_parser_stop(priv->parsing_request);   // what is left of the read is of nobody
    gobj_send_event(gobj_bottom_gobj(gobj), EV_DROP, 0, gobj);
    KW_DECREF(kw)
    return 0;
}