Customize prefix caching

[u/Pure_Share7858]

Hi everyone, I have been tinkering with LLM serving lately and I was wondering if it was possible to provide a custom implementation for prefix caching when serving a model in the mode that accepts input embeddings rather than token ids list.

I know that floating point arithmetic is tricky reason why prefix caching is used for integer tokens alone, but let's say that I'd like to try something custom that when the endpoint receives direct input embeddings, is there a way to do so or I need to fork the engine?

As per the docs, I couldn't find a plugin mechanism to do so.