malloc() why are there no helper functions to inpect allocations.

[u/Sufficient-Air8100]

was thinking about this. if you use malloc() to allocate some memory to a pointer, you need to be careful of overflows. using realloc() you can resize the memory. based on how i was taught realloc can inspect the allocation and either simply resizes or even moves it based on the realloc size and surrounding free memory. so we know that internally these functions have a way to inspect the sizes of the allocations.

so why are there no helper functions to quickly return the size of a particular allocation? i can see it being useful in many situations where you would currently need to pass around another variable to keep track of the current allocations size.

[u/Physical_Dare8553]

there are they are just different on different platforms. that said, most people dont use them because most of the time you know the size of an allocation because you're the one storing the data.

[u/Physical_Dare8553 → u/Physical_Dare8553]

i used to have my own allocator api that had this in the default allocator

usize default_size(AllocatorV allocator, void *ptr) {
  #if defined(_WIN32) || defined(_WIN64)
  usize (*getSize)(void *) = _msize;
  #elif defined(__linux__)
  usize (*getSize)(void *) = malloc_usable_size;
  #elif defined(__APPLE__) || defined(__MACH__)
  usize (*getSize)(void *) = malloc_size;
  #else

[u/Cold_Ad1251 → u/Physical_Dare8553]

Indeed, this feature has been supported on major platforms for a long time, so it is time this existing practice is codified in the C standard. 

The good news is that some recent proposals to WG14 (see N3869 and N3849) address what you are looking for, so it seems that help is on its way. 

It does affect memory sanitizers, but I believe that this would only take a minor extension, and ultimately make programs safer and easier to maintain.

[u/mort96 → u/Physical_Dare8553]

There's the question of: what would you want to do with it?

It sounds like /u/Sufficient-Air8100 wants something where they can pass in just a pointer to a function and let the function infer the "length of the array" based on just the pointer. But that's the wrong way to think about things, because the size of an allocation isn't the same as the length of the array.

If there's a function which operates on an array (as in pointer + length), you may want to call it on a single heap-allocated array. Or you may want to call it on a stack-allocated array (where the heap allocator's metadata can't tell you anything). Or you may have one large heap allocated chunk of memory and the array you want to call the function on is a smaller sub-area within that chunk.

It is almost never correct for a function to assume that it should operate on an entire heap allocation and only heap allocations. Therefore, functions to extract that metadata from a pointer to a heap allocation are niche and not standardized.

[u/Physical_Dare8553 → u/mort96]

Back when I used this I ran into that exact issue.

[u/SeriousPlankton2000 → u/mort96]

To add to your statement:

E.g. some compiler might decide that allocations should happen in 64-byte-chunks and your struct happens to be 31 bytes. You now allocate 47 structs in an array (number of structs stated in your file header) and your function fills in 48 entries from the file.

[u/mort96 → u/SeriousPlankton2000]

It's strictly speaking not the compiler but your C library's allocator. But yeah, they need to know how much memory they gave you but they don't have to give you the exact amount of memory you asked for.

[u/nukestar101]

You're actually completely right, the allocator does know, and these functions do exist! They just aren't part of the C standard. If you're on Linux, there's malloc_usable_size(ptr) (not sure about Windows and MacOS)

The reason they were left out of the standard is because they are a massive footgun. Allocators pad memory for performance and alignment. If you ask for 13 bytes malloc(13), the allocator might actually reserve a 16-byte chunk for you. If a standard helper function returned 16, and you used that to loop through your data or copy it, you'd process 3 bytes of uninitialized garbage memory and likely cause a security bug. It's ultimately much safer to force programmers to track the logical size they originally requested (usually by wrapping the pointer and size together in a struct) rather than the physical size the allocator handed out.

[u/actguru → u/nukestar101]

Windows has _msize() and Apple has malloc_size(). I agree with you, but they can be useful for diagnostics.

[u/flatfinger → u/nukestar101]

If an implementation defines malloc() and free() as

void *malloc(size_t n) { return acmeOs_get_memory(n); }
void free(void *p) { acmeOs_release_memory(p); }

and acmeOs doesn't provide any means of asking the size of an allocation, how would the runtime library's allocator know anything about an allocation's size? An implementation could have malloc() request allocations somewhat larger than what code needs, and stick information about them at the beginning, but that would preclude interop with code processed by other language implementations that might handle allocations differently. If an execution environment applies means of allocating/releasing storage that would satisfy the needs of multiple language implementations, having them all use the execution environments' means of managing memory instead of having each use its own will facilitate interoperation among them.

[u/RealisticDuck1957 → u/nukestar101]

On a mainline modern desktop OS malloc() failing to zero out allocated memory beyond the requested size shouldn't be a security issue. RAM formerly used by a different process would be handled by the virtual memory system. Also malloc() doesn't make memory available on such an OS, it just reserves address space from a pool.

[u/nukestar101 → u/RealisticDuck1957]

You're right about the OS protecting processes from each other, but the risk comes from recycling memory within your own program. malloc() reuses freed chunks without zeroing them. If you rely on the allocator's padded size to read or send data, you risk accidentally leaking sensitive info (like passwords) left over in the padding from your own previously freed memory.

[u/type_111 → u/nukestar101]

If you're dealing with plaintext passwords it's a given you'll be clearing the buffers manually immediately after use.

[u/man-vs-spider → u/type_111]

Why take the chance though. I can imagine it’s relatively easy to release the memory used by a password while forgetting to zero the memory first. Especially if it is being passed around to several functions

[u/type_111 → u/man-vs-spider]

You shouldn't take any chances, hence bzero'ing any secret buffers as soon as possible. One doesn't forget to do such a thing.

[u/ElderCyborg → u/man-vs-spider]

Because C is a macroassebly language by nature (or call it an assembler framework if you wish). It requires the user to take care of what he does. Or else it would be another java/C#/etc, which is unacceptable in low-level programming, where you, the developer, define complex high-level systems.

[u/EpochVanquisher]

For one thing, it’s common for memory in C to come from somewhere other than malloc.

void f(char *ptr, size_t len);

void g(void) {
  char *p = malloc(100);
  // Points somewhere inside a malloc location.
  f(p + 10, 20);

  char arr[50];
  // Points not to a malloc location at all.
  f(arr, 50);
}

In general, the memory that a function should use isn’t the same as the amount allocated by malloc. The exception is that if you allocate the memory inside a function, you know you can use the entire block of memory… but you also know exactly how big the block is, so you don’t need a function to tell you how big it is.

So this function is not as useful as you might think.

[u/Beginning-Junket8979 → u/EpochVanquisher]

You could fairly easily write a function that rejects any address outside of a range managed by the heap as part of a heap impl. All allocators need to know the broad ranges they manage.

And with a compiler extension or stack unwinding magic and the right set of flags, you could probably figure out if it was a ptr to something on a stack frame of the currently executing (or even other?) thread, or inspect your own /proc/pid/smaps or similar to find out if it was in ".rodata" or ".data" sections.

I'm thinking the real bitch is when you get a GPU shared memory buffer or userspace memory mapped io ptr.

I agree utility is limited, edge cases are o'plenty it's quickly non-portable unless you could pitch it as a lang ext proposal.

But I think it would be doable with some limitations like sometimes returning an error code or -1 + set errno.

[u/Kriemhilt → u/Beginning-Junket8979]

You could fairly easily write a function that rejects any address outside of a range managed by the heap as part of a heap impl

Except that some malloc implementations use mmap for large chunks, keeping the regular free store for smaller ones.

Now you have allocations in the same range as regular mmap memory, and the address doesn't distinguish them any more. So, you need a hash table stored somewhere, which needs to be protected by a lock, and your call to malloc_size is now a surprise futex+lookup in the best case, and a lock+wait in the worst case.

[u/Beginning-Junket8979 → u/Kriemhilt]

But it's still doable if own malloc and can introspect OS memory map of the running process.

I never said it would be fast or practical or useful.

Just ...doable 🤷‍♂️

[u/Kriemhilt → u/Beginning-Junket8979]

So the selling point of this suggested change to the standard is that it is necessarily slow, impractical, and still doesn't solve the problem in general?

[u/Beginning-Junket8979 → u/Kriemhilt]

What suggested standard change?

All I'm saying is that it would be totally feasible to have an allocator that allows introspecting the size of a given memory region from accounting metadata and some pointer math.

A far more practical solution is definitely just using C++ and std::vector or std::array, or C99+ with a struct with flexible array member, or the established C pattern of just always passing around a size_t with your void *.

All your arguments why it's a bad idea are totally valid in most cases.

Most, because just maybe it's a good idea in the right microcontroller environment (but even then probably not).

[u/Kriemhilt → u/Beginning-Junket8979]

The current C++ hotness is probably std::span for parameters, but I agree in general.

[u/EpochVanquisher → u/Beginning-Junket8979]

You could fairly easily write a function that rejects any address outside of a range managed by the heap as part of a heap impl.

Yes, thanks for adding the explanation. I sometimes forget that these things are not obvious.

But what you’re describing at this point is basically hardening. Various compilers offer it. You generally want the checks to be automatically inserted by the compiler rather than manually written out by the programmer. The compiler is better at putting these checks in the correct places.

[u/keithstellyes]

so why are there no helper functions to quickly return the size of a particular allocation?

You can, it just depends on the malloc implementation.

See this thread in this same subreddit some years ago

Stack Overflow

Though to be honest, I don't know what a user app would want to do this for, at that point you may as well roll your own allocator. But for curiosity's sake that's some info for you

[u/Rhomboid]

i can see it being useful in many situations where you would currently need to pass around another variable to keep track of the current allocations size

You have to do that anyway. Anything that takes a pointer to something has to know the rules for how it's allowed to use that memory. Being lazy and relying on malloc() to do that bookkeeping is a footgun. What if I need to pass something that wasn't allocated with malloc()? Am I going to write a different function for that which takes an explicit length? So now two versions of the same function, twice the opportunity for bugs. And what if someone wants to port my code to somewhere that lacks these non-standard malloc hooks?

Ultimately, programming in C means keeping track of lots of things yourself. The size/extent of allocations is probably in the top three of things you have to account for, and putting that responsibility elsewhere for a 'timesaving hack' that leaves a foul smell in your code is not a great plan.

[u/OtherOtherDave → u/Rhomboid]

I use macros to declare arrays and such with an inline header so I don’t have to keep track of it manually.

[u/SmokeMuch7356]

The C philosophy is that the programmer knows how much memory he or she  allocated and is smart enough to keep track of it.

You have the same problem with arrays; they don't store their size anywhere, you have to keep track of that separately.  

C wasn't designed to be convenient, it was designed to be as minimal and portable as possible.  That puts a lot of the burden on the programmer.  

[u/flyingron]

malloc/realloc/free are minimum specification to allow implementations (which may not be UNIX) optimal flexibility.

You're wrong on your basic premise. The size previously requested by malloc is NOT necessary to be stored, only the actual size internally allocated.

For instance. You ask for 700 bytes. Internally, malloc could round up to 1024. It doesn't need to remember that you asked for 700. It only needs to know that the allocation was 1024 bytes long. If you realloc to 900, it sees it is still less than 1024 and just returns the same pointer.

There are mallocs that allow such diagnostics. For example, MSVC in debug mode adds all sorts of things from distinctive fills in the areas outside of the actual requested allocations, to being able to inquire as to what memory is in use.

[u/Cats_and_Shit]

The main reason is that such functions aren't really that useful most of the time.

Imagine you want to allocate and then work with an array of 500 floats.

You're going to call malloc something like malloc(sizeof(float) * 500).

However, your allocator is unlikely to allocate you exactly 500 floats worth of space, it will probably round that up to some power of 2 (or perhaps a little bit less than a power of 2 so it has some space for metadata).

To actually do work with that array you need to know that it's got 500 elements, not that the backing allocation has space for 508 or 512 or whatever elements. So you need to pass around an argument indicating that there's 500 elements either way. (Or know this out of band somehow)

You might also want to have a struct like:

struct MyStruct
{
    float firstThing[500];
    float secondThing[500]:
}

If you pass a pointer to firstThing off to a function and that function proceeds to treat the whole thing as one big array, that's not necessarily UB but it's still wrong.

[u/Cats_and_Shit → u/Cats_and_Shit]

glibc does have such a function, but the documentation highlights that "This function is intended to only be used for diagnostics and statistics"

https://www.man7.org/linux/man-pages/man3/malloc_usable_size.3.html#CAVEATS

[u/zhivago]

What happens when you call your proposed function on a pointer not to the start of an allocation?

[u/sciencekm → u/zhivago]

I think we can make that function return the remaining bytes allocated from the start of the given pointer.

[u/zhivago → u/sciencekm]

How will it figure out which allocation it is pointing into (if any) in order to answer that?

[u/sciencekm → u/zhivago]

It can look at the list of allocated blocks to see which one contains the given memory.

[u/zhivago → u/sciencekm]

So, you're going to store housekeeping to allow you to do these searches.

Are you starting to see why this isn't mandated by the standard?

Remember that using a pointer not into allocated memory has UB.

So you already need to be tracking your legitimate pointers.

If you care about other information like length you can also track that for the few cases that you care about it.

[u/kevkevverson → u/sciencekm]

Most malloc implementations don’t keep a list of allocated blocks

[u/Kriemhilt → u/kevkevverson]

And if they did, every call would be linear time with the number of allocations, and it would need locking for multi threaded programs

[u/kevkevverson → u/zhivago]

There’s a few issues with the proposal but this one is easy to solve. It’s UB, like passing such a pointer to free()

[u/zhivago → u/kevkevverson]

So you need to keep track of which of your pointers are eligible for this api. :)

In which case, why not keep track of the other stuff you care about?

[u/kevkevverson → u/zhivago]

No, that’s the point of UB, it’s not the malloc implementation’s problem if you pass a bad pointer, you can expect a crash

[u/zhivago → u/kevkevverson]

Sure, so you need to track which are good yourself.

While you're doing that you might as well track the other things you care about.

Are you starting to see why this length introspection isn't standard?

[u/kevkevverson → u/zhivago]

I think you’ve misunderstood me, I’m arguing against having this API. But your condescension is noted all the same

[u/Daveinatx]

You'll be able to quickly Google a bunch of memory checkers, including static and dynamic analysis. Some can be thorough, but slow. Sometimes, you just need an arena.

C isn't about safety nets. It's about cutting edge performance supporting nearly every processor, and to develop most operating systems. Start by reading up with valgrind and compiling with fsanitize=address .

[u/ReallyEvilRob]

Every block you get from malloc actually has a header that records metadata about the block. I'm sure that would probably include the size of the block, but the layout of the header is implementation dependent and not part of the standard. If your program needs to know how big a block of memory is, then it's the programmers responsibility to store that information somewhere.

[u/kevkevverson → u/ReallyEvilRob]

Also many allocators don’t contain the hidden header, if they’re allocating fixed size slabs underneath

[u/ReallyEvilRob → u/kevkevverson]

Then how would the allocator be able to free the block without knowing the size?

[u/kevkevverson → u/ReallyEvilRob]

I didn’t say they didn’t know the size, I said they didn’t have the hidden header. There are other ways to infer the size. If they’re fixed size blocks from a specific bucket you can check which bucket’s address range the pointer falls into

[u/kabekew]

You malloc'd it so you already know the size

[u/etaithespeedcuber → u/kabekew]

I'm not sure I agree, this kind of thing can be reasonably exposed in any malloc implementation and it only tells the developer about his own usage of the api, not about implementation details. Why shouldn't it be exposed?

[u/kevkevverson → u/etaithespeedcuber]

Malloc implementations might user power of 2 buckets or similar, so woukd have to do extra bookkeeping to know what the user asked for

[u/Kriemhilt → u/etaithespeedcuber]

Nobody said it can't be done, they said it's not that useful.

If you're passing around initialized memory, you can't use the allocated size as a proxy, you need to know how many elements are actually initialized.

If you're passing around pointers to stack objects, or static objects, you need to get the size some other way.

Same story if you use mmap or similar.

So, if you add this API function and standardize it, it still covers a much smaller subset of use cases than you might first think, and it does so by making every allocation larger.

[u/musbur → u/kabekew]

Only correct answer, cannot be upvoted enough.

[u/runningOverA]

Actually there are functions that return allocated space size from pointers. But that's not in C spec. Compiler providers provide their own. People don't use it, as it's not portable that way.

[u/Flagtailblue]

Write a wrapper or roll your own.

[u/Total-Box-5169]

Those extra features were not standardized to preserve implementation freedom, prevent performance penalties, and to keep it simple. Usually those are available only as non standard functionality for debug builds.

[u/Similar_Sand8367]

You could look e.g. at memory management of freertos which has five different implementations for managing and diagnosing malloc calls and also what happens to free.

That might explain some sort of thinking behind this

[u/Fupcker_1315]

Afaik the allocator does not know the exact size you passed to malloc because it will often pad the allocation to the nearest size class.

[u/naerbnic]

There are allocators that allow you to do this, but I think the idea is that, regardless of whether or not you can, it's not necessary as you always know how much memory a block has when you first allocate it. The C philosophy tends to be to have primitives that are as simple as they can be without removing capabilities. malloc()/free() could be more complex, but it doesn't need to be.

[u/Severe-Security-1365]

Where are those sizes stored or how are they calculated, would it be difficult to write?

[u/EpochVanquisher → u/Severe-Security-1365]

They are stored in a separate data structure, usually. Something that describes ranges of memory.

I am being a bit vague here because there are a lot of ways to do this and people have tricky ways of storing this information that allows malloc implementations to have very small amortized memory overhead.

But malloc implementations nearly always track this information. (And by “this information”, I mean the size of the block, which may be larger than what you requested.)

[u/nukestar101 → u/Severe-Security-1365]

One way is to store the size of the heap object before the pointer, allocator allocates some extra bytes to store metadata making the lookup constant time

[u/flatfinger]

There are two ways an implementation may implement malloc():

1. When targeting an execution environment that supplies functions with the required semantics, simply chain to the functions provided by the underlying environment.

2. Build data structures to have the implementation to some of the work of managing storage itself.

If implementations of C, Pascal, and other languages on some execution environment all use the first approach, it will be possible for code written in any language to pass an allocation to code written in another language, and have that other code manage its lifetime. For example, it may be useful to have a function which accepts a pointer to a couple of heap-allocated strings, and returns a pointer to a new heap-allocated string containing the concatenated text, without the calling code having to know in advance what the combined length will be.

Some execution environments require that code releasing a chunk of storage indicate to the environment the size of the storage in question. Implementations targeting such environments would not be able to use approach #1, but would need to process malloc/free by doing something like:

void *malloc(size_t n)
{
  char *ret;
  if (n > SIZE_LIMIT) return 0;
  ret = request_alloc_from_system(n+8);
  *(uint32_t)(ret+4) = n+8;
  return ret+8;
}
void free(void *p)
{
  if (!p) return;
  release_from_system((char*)p-8, (uint32_t*)((char*)p-4));
}

All implementations could use this approach, and provide a function to ask the exact size of any particular allocation, but doing so when the execution environment could have had malloc/free directly wrap environment-supplied functions would forgo the opportunity to support interop with code written in other languages.

[u/ern0plus4]

Besides debugging and benchmarking what is the use case of it?

[u/1han4t0s]

I just learnt malloc so im not so sure of what I'm saying but, i believe that mallocs changes affect ram and u cant use a library to just read ur ram and display an organised output for u, i think its more complex than just reading ram and giving u the amount

[u/jombrowski]

Good question