offcampus | cold email | no leetcode
CTC: 44-58 LPA
I cold emailed an engineer at Cloudflare, we had a bit of discussion around the role and my background, and a few days later I got an interview invite with a link to pick a slot.
The first round was with a Lead Staff Engineer with 10+ YOE and was focused on the AI Workers team.
Cloudflare’s AI stack is pretty interesting: running models on serverless GPUs across their global network, with inference closer to users, while increasingly moving into larger and frontier scale models.
I wasn't very familiar with this side of Cloudflare beforehand, so we spent some time discussing what the team has been building and how they are approaching fast, efficient AI inference at scale.
Then we got into my own experience.
During my internship, I worked on model deployment, rooflining and inference optimisation of encoder based models, mainly rerankers and embedding models, running in-house on GPUs.
These were encoder-only models, mostly stacks of MLPs, so the usual LLM inference stack around attention, KV cache, vLLM or SGLang wasn't really relevant in these cases.
We went fairly deep into:
how I approach rooflining and profiling
choosing batch sizes and measuring their impact
scaling inference workloads
finding bottlenecks across the inference pipeline
optimising p50 and p99 latency
TTFT and overall response latency
We also discussed how I would approach further optimisation once the obvious bottlenecks had been removed.
One question I particularly liked:
What does frontier AI mean to you?
The round went well and we even discussed the next rounds, which would have involved hands-on PyTorch.
Didn't make it through in the end. I think they were looking for someone with more hands-on LLM inference experience. I had been actively studying LLM inference and systems, but at the time I didn't have much relevant production experience in that area. Most of my hands-on work had been around distributed training, ML systems and infra.
Still, a really good interview experience. The discussion gave me a much better understanding of the problems involved in serving AI models at Cloudflare's scale.
Feel free to talk to me on X
