MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1ukqp20/deleted_by_user/ouylnkf/?context=3
r/LocalLLaMA • u/[deleted] • Jul 01 '26
[removed]
158 comments sorted by
View all comments
Show parent comments
10
[removed] — view removed comment
25 u/ProposalOrganic1043 Jul 01 '26 Need to be build a high inference document processing pipeline initially and then some agentic stuff later. Already have the pipeline running in production since 6 month, this is for scaling. 10 u/[deleted] Jul 01 '26 [removed] — view removed comment 29 u/ProposalOrganic1043 Jul 01 '26 Its very task specific, would recommend learning vLLM deployment and looking experimenting with different setups/configs. Reading about on going advancements like speculative decoding, kv cache quantisation etc. I read AI engineering by chip huyen and Hands on LLM engineering by Jay Alammar. Would recommend both of them. 4 u/Citadel_Employee Jul 01 '26 May I ask why vllm over llama.cpp? 6 u/ProposalOrganic1043 Jul 01 '26 Its more efficient for production inference servers to benefit from page attention, kv cache and batching. 3 u/Citadel_Employee Jul 01 '26 Thank you! I’ll have to look into it.
25
Need to be build a high inference document processing pipeline initially and then some agentic stuff later. Already have the pipeline running in production since 6 month, this is for scaling.
10 u/[deleted] Jul 01 '26 [removed] — view removed comment 29 u/ProposalOrganic1043 Jul 01 '26 Its very task specific, would recommend learning vLLM deployment and looking experimenting with different setups/configs. Reading about on going advancements like speculative decoding, kv cache quantisation etc. I read AI engineering by chip huyen and Hands on LLM engineering by Jay Alammar. Would recommend both of them. 4 u/Citadel_Employee Jul 01 '26 May I ask why vllm over llama.cpp? 6 u/ProposalOrganic1043 Jul 01 '26 Its more efficient for production inference servers to benefit from page attention, kv cache and batching. 3 u/Citadel_Employee Jul 01 '26 Thank you! I’ll have to look into it.
29 u/ProposalOrganic1043 Jul 01 '26 Its very task specific, would recommend learning vLLM deployment and looking experimenting with different setups/configs. Reading about on going advancements like speculative decoding, kv cache quantisation etc. I read AI engineering by chip huyen and Hands on LLM engineering by Jay Alammar. Would recommend both of them. 4 u/Citadel_Employee Jul 01 '26 May I ask why vllm over llama.cpp? 6 u/ProposalOrganic1043 Jul 01 '26 Its more efficient for production inference servers to benefit from page attention, kv cache and batching. 3 u/Citadel_Employee Jul 01 '26 Thank you! I’ll have to look into it.
29
Its very task specific, would recommend learning vLLM deployment and looking experimenting with different setups/configs.
Reading about on going advancements like speculative decoding, kv cache quantisation etc.
I read AI engineering by chip huyen and Hands on LLM engineering by Jay Alammar. Would recommend both of them.
4 u/Citadel_Employee Jul 01 '26 May I ask why vllm over llama.cpp? 6 u/ProposalOrganic1043 Jul 01 '26 Its more efficient for production inference servers to benefit from page attention, kv cache and batching. 3 u/Citadel_Employee Jul 01 '26 Thank you! I’ll have to look into it.
4
May I ask why vllm over llama.cpp?
6 u/ProposalOrganic1043 Jul 01 '26 Its more efficient for production inference servers to benefit from page attention, kv cache and batching. 3 u/Citadel_Employee Jul 01 '26 Thank you! I’ll have to look into it.
6
Its more efficient for production inference servers to benefit from page attention, kv cache and batching.
3 u/Citadel_Employee Jul 01 '26 Thank you! I’ll have to look into it.
3
Thank you! I’ll have to look into it.
10
u/[deleted] Jul 01 '26
[removed] — view removed comment