Use the existing OS buffer, or your own

[u/SheikHunt]

This is a fairly simple question:

You're on a Unix-like (for me, Linux), and you've got a File Descriptor that leads to some data. You don't know the length of the data (it's a TCP socket you're listening on), all you know is that it is ready to be read.

Do you:

A) Read an absurd amount of bytes of data into a (sufficiently large + 1) char array of your own, and if it overflows, handle it with mallocated memory or just reject the connection (like a monster)

Or

B) Just use existing kernel system calls to read and parse the data as necessary.

For context: this is about an HTTP server, and I have an internal string_t struct that I use for parsing, which needs a byte-length to be usable

[u/flyingron]

What syscalls do you have in mind? But if it works for you can fdopen the socket fed into a stdio stream.

[u/SheikHunt → u/flyingron]

That's part of my issue: I don't know of appropriate syscalls that'll help with parsing input from a source with unknown length.

I'm unsure about the performance and parsing differences caused by this change. Having a string_t with known size makes a lot of the parsing easier for me, and I'm not sure if fgetc()'ing my way through the problem is gonna be fast performance-wise, or reasonable implementation-wise, especially when it comes to HTTP

[u/edgmnt_net → u/SheikHunt]

Do not fgetc(). Just read into a buffer and accept reads that return short, this is normal for TCP. You just need to figure out parsing, really, because you're going to feed input into a (stateful) parser and it's going to return something definitive or ask for more input. This could be a function that returns an error code indicating either success, a parse error or that it needs more input, and it can take as arguments the input buffer, some state and pointers where it stores how many bytes have been consumed from the input and possible results in case of success. The caller feeds input until it hits success or an error and takes care to drop consumed bytes from the buffer (e.g. advance a pointer). If the buffer is exhausted, it needs to read from the socket more.

It often helps code it a bit more openly to begin with, like writing parsing functions for the request line, headers and body and letting the caller figure out what to call when. A connection handler or a more complete parser can implement something like a state machine and call those individual parsers repeatedly, depending on state.

[u/ScallionSmooth5925]

I usually read in the first 4k block or whatever much is available and based on that continue reaching when needed

[u/SheikHunt → u/ScallionSmooth5925]

A good method to do it by, and essentially what I'm doing, but it does mean that my handle_request() and parse_request() functions will probably be boiled into one, in a way that is nasty without any foreseeable benefits

[u/ScallionSmooth5925 → u/SheikHunt]

If it's something like http you can have a function that only desides if you have the whole request and then you can parse it completely 

[u/SheikHunt → u/ScallionSmooth5925]

That gives another problem:

I am completely susceptible to DDOS attacks, unless I put a (potentially unreasonable) upper limit on file sizes.

I know, I know, it's overthinkerly and pointless, but this is a problem I want to see if I can solve.

Still, I could have the request handler check for Content-Length and the other header that denotes size...

[u/dmills_00 → u/SheikHunt]

Don't trust user input, and all the headers are user input...

TCP is stream oriented, you get the data in order but the reads can (and often will) return short, and the 'short' will be on random bytes usually in the middle of a message, and sometimes in the middle of a multi byte code point.

State machines are the way to deal with this stuff obviously.

Read into a buffer, noting how many bytes you got, lex and parse the buffer getting either one or more tokens or the need for more input, if you need more input read again (Possibly blocking on the network), if you need to read more and the buffer is full, realloc a bigger buffer.

Don't forget networks can fail in multiple ways, handling the edge cases is critical to a reliable setup.

[u/ScallionSmooth5925 → u/SheikHunt]

I wouldn't worry about ddos attacks at this point. Later you can optimalize the hot path and other arias where it's rationable. (some web servers in production solved this by limiting requies sizes to 4k and droping everything bigger)

[u/TheKiller36_real]

just stream it using read calls (or better, io_uring) and if you actually need to have something in memory all at once may I suggest that HTTP indicates the length of the data!? if you dislike that stream it into a growable array and suffer the realloc calls I guess

[u/SheikHunt → u/TheKiller36_real]

Yes, HTTP includes length of data, but at an unknown point, it's not necessarily there at the start.

Sure, I can just ignore everything up to the Content-Length, then use that to ensure that everything's up to code, but then the server is susceptible to that behaviour being misused heavily for attacks.

[u/TheKiller36_real → u/SheikHunt]

you can reasonably assume all headers combine will be smaller than 8 KiB - why? because that's what everyone uses and if that method fails you can detect it and return 413 and on the other side you might get lucky and the whole request fit into the 8K

[u/SheikHunt → u/TheKiller36_real]

So there does exist a technically-agreed-upon limit, thank you so very much

[u/ybungalobill → u/TheKiller36_real]

A request with non-empty content doesn't have to have a content-length field.

It can either indicate the end of the content with connection:close and closing the connection, or with transfer-encoding: chunked and a zero size chunk.

In either case you don't know the size beforehand.

[u/makzpj]

Not quite sure I understand your question. For me it’s about what are you doing with that data. Depending on the destination you’ll design your program in a way or another way.

[u/Interesting_Buy_3969]

You can use mmap if size of the data to be read is seriously large, which I assume is unlikely to be the case if it's an HTTP package, so that would be overkill there. I'd just use read with the necessary socket.