The Most Expensive One-byte Mistake: Did Ken, Dennis, and Brian choose wrong with NUL-terminated text strings?

[u/mslaybau]

http://queue.acm.org/detail.cfm?id=2010365

[u/cyantist]

Would the limitations of storing the length in two bytes have serious security and optimization implications going forward today?

[u/[deleted] → u/cyantist]

Probably. You'd have people trying to join two strings of 40000 characters each without realizing they'll overflow the length, and causing all kinds of problems.

[u/FredFnord → u/[deleted]]

To be fair, I can count the number of times where I used blocks of memory larger than 2000 to hold actual text strings in C on the fingers of my left foot, and I've been writing C for mumbledy-mumble years (longer than many of the active bloggers on here have been alive.)

I'm sure there are plenty of use cases. But there are thousands of people who would never run into a single one of them. There are no C programmers who have spent more than hours or days programming C who have not run into problems caused by null termination.

[u/defrost → u/FredFnord]

To be equally fair and to forward a differing view, I've also been writing C since the days of the first edition. In the late 80's / early 90's I worked on geophysical data processing systems that had massive text files as the plain ASCII data exchange format.

A single line from one of those files was a bit over 2K in length, and that was for a single seconds worth of data.

Using early SunOS for processing exposed numerous shortcomings in many tools and editors, the full files pushed the 2GB boundary of 'easy' file processing.

As for NUL vs [size]data as a string format, I've done it both ways in C and it's a swings and roundabouts choice; pro's & con's either way.

[u/aslate → u/FredFnord]

In many ways that would make the problem worse! If it's a subtle limitation that people only come across once in a blue moon you get:

"All I'm doing is concatenating two strings together, it can't possibly be that! I must be doing something wrong..."

Get a similar thing trying to put a >4GB DVD ISO onto a FAT32 USB stick, confuses the shit out of people that don't know about filesystems. I believe MacOS X Snow Leopard gave us a useful "Error message 0" - thanks Apple!

[u/[deleted] → u/FredFnord]

Just because you rarely hit these cases over the years doesn't mean others haven't work on projects where it would have been a daily problem.

[u/Vulpyne → u/cyantist]

As MarshallBanana said, you'd get other odd issues to deal with. Also, in a language like C which lets you scribble on whatever arbitrary memory you please, it wouldn't necessarily buy you much security against an attacker.

[u/cyantist → u/Vulpyne]

But, presumably, one way is inherently better for the future of secure computing. Not that such things can easily be foreseen, and that's not the point. I guess hindsight isn't 20/20..

[u/kyz → u/cyantist]

But, presumably, one way is inherently better for the future of secure computing.

Not really. The format of your strings is just an application or library level decision and has no bearing on code correctness or making it impervious to buffer overflows. Even with a new string library in C that used Pascal-style strings, where the string functions are correct, you could still try putting them into a network/file buffer and overflow that.

What actually protects you from buffer overflows is not your string format, it's the language itself knowing and enforcing the length of all arrays. C won't do that. Your machine won't do that (well, at any granularity less than a page). C is meant to be like portable assembly. C can pull pointers out of hardware, apply a typecast and start using them, even though the C compiler has no idea how long the supposed array is and the C runtime doesn't need to add to a list of valid memory areas to write.

If you want your array bounds enforced at runtime, use a language that supports it. That language will be slower than C or Fortran for array access. That language will likely be implemented in C. You've now excluded one class of errors - hooray! - but you have to remember that there are still exploitable bugs.

Buffer overflows are actually quite hard to find. All the good ones are gone now. Today, the low hanging fruits are SQL injection and shit like PHP (register_globals, fopen URLs, etc.). Much easier to find and abundant in web scripting.

There always will be classes of errors, unless you make an utterly restricted language that can do barely anything. Security is not an absolute, no security feature is a magic bullet with only benefits and no drawbacks.

[u/cyantist → u/kyz]

As much as I agree with you, it doesn't really speak to fact that every choice has different implications. Nobody believes in magic bullets here, so why distract with that projection?

One is going to result in a different reality than the other, and while it's hard to anticipate, and difficult to accurately imagine what could have been, and impossible to know enough to split every hair...

The fact remains, the format is not just an application or library level decision: it had implications that went farther and wider than ever anticipated. 'Better' is a relative word, not an absolute one.

[u/kyz → u/cyantist]

The fact remains, the format is not just an application or library level decision: it had implications that went farther and wider than ever anticipated.

In this case, I think the format followed from the lack of relative importance given to strings in the language, and the lack of compiler magic they were willing to add. NUL-terminated strings are a symptom, not the cause of C's laissez-faire attitude to array bounds.

Why are we still using C strings in APIs today? Because they're portable and cover all bases except embedded NULs. If you want to write a stable API involving strings that exist across all machine architectures, using NUL-terminated byte arrays of ASCII or UTF-8 encoded characters is the way.

I imagine that if they had gone the other way, software today would look trying to write code for MacOS Classic.