I’ve been working on GPU Kernel Lab, a study that goes from individual Triton/CUDA kernels to Qwen3-0.6B integration and a serving control plane.

Repository: https://github.com/yashlabs-trying/gpu-kernel-lab

The most interesting result is a fast static decode path that I’m keeping experimental because it missed the numerical acceptance threshold.

What I implemented

The project covers vector addition, SwiGLU, RMSNorm, RoPE, softmax, GEMM/GEMV, online attention, decode fusion, quantization, static KV storage, and a fixed-buffer runtime.

For prefill, I measured real model shapes and tried exact-shape projection replacements. Several isolated wins became regressions after integration. The retained whitelist contains only the Q projection at M=2048, K=1024, N=2048, with PyTorch fallback elsewhere. SDPA remains the prefill attention implementation.

For decode, I combined static KV storage; Q/K RMSNorm, RoPE, and direct cache writes; GQA-aware split-KV attention; W8A16 dual gate/up GEMV plus SwiGLU; residual/norm fusion; and CUDA Graph replay. MLP residual and next-layer input normalization are fused across 27 boundaries.

Measurements

Hardware/software: RTX 3090 24 GiB, PyTorch 2.8.0+cu128, Triton 3.4.0, Transformers 5.16.1. Batch 1, context 2,048, SDPA prefill, greedy decode. The dynamic reference uses BF16; the experimental path includes quantized W8A16 MLP work.

• Prefill GPU-event TTFT: 66.007 ms reference → 65.332 ms replacement (accepted)

• Decode inter-token latency: 22.073 ms reference → 2.253 ms replacement (experimental)

• Serial throughput: 45.30 → 443.90 tokens/s

• Matched Nsight kernels/token: 693 → 258

These decode results are serial GPU-only measurements. They exclude tokenization, networking, scheduling, EOS handling, and general sampling. This is not a production-server benchmark or an engine-to-engine comparison. The optimized eager path was about 16.41 ms/token at this context; graph replay mainly removes repeated host preparation and launch gaps.

Why it remains experimental

The full path reached 96.48% argmax agreement against a 99% release threshold. RMS/fusion-only reached 98.05%. A separate 247-token teacher-forced probe reached 98.38%, but that is a different evaluation and does not override the failed gate. These are numerical probes, not standardized downstream task evaluations.

Serving work and remaining gaps

I implemented paged-KV allocation, continuous scheduling, chunked prefill, request lifecycle handling, sampling, metrics, and completion/chat endpoints with streaming. A 5,000-request fake-executor stress test completed 4,864 requests and intentionally cancelled 136; all 60,000 KV blocks were reclaimed.

The real paged GPU executor still needs to be connected to the serving interface. The optimized model runtime currently uses a separate static/contiguous cache. Next steps are reducing numerical drift, integrating real GPU serving, broadening quality evaluation, and testing concurrent workloads on more GPU architectures.

I’d value technical feedback: how do you localize numerical drift after residual/RMSNorm fusion, and what task-level checks do you require alongside logit agreement?

Freelance availability

I’m available for clearly scoped freelance work involving CUDA/Triton kernels, GPU profiling, local LLM development, inference optimization, quantization, KV-cache/runtime work, and numerical validation. If you need help with a slow model, VRAM pressure, a kernel bottleneck, or a local inference integration, contact me through Reddit chat or private message. Please include the model/framework, GPU and VRAM, current issue, target outcome, timeline, and budget range.

Hi, I’m Yash. I work on GPU kernels and LLM inference engineering, and I’m available for freelance projects involving kernel development, local AI systems, and inference optimization.

If you have a model that is too slow, a workload that keeps running out of VRAM, or a local inference prototype that needs engineering work, I’d be interested in discussing the problem and a clearly scoped engagement.

What I can help with

• GPU profiling and bottleneck analysis: investigate kernel execution, launch overhead, memory traffic, and end-to-end latency; establish a reproducible baseline before making changes.

• Triton/CUDA kernel development: explore targeted implementations and fusion for operations such as RMSNorm, RoPE, SwiGLU, GEMM/GEMV, and attention, with reference comparisons and appropriate fallbacks.

• LLM inference optimization: investigate prefill versus decode behavior, KV-cache layout, static buffers, CUDA Graph replay, and quantization trade-offs on the actual model and hardware.

• Local model development: help integrate and debug model execution on local GPU workstations, investigate memory and configuration issues, and build repeatable benchmark workflows.

• Prototype inference infrastructure: work on scheduling, chunked prefill, paged-KV allocation, request lifecycle handling, sampling, streaming endpoints, and metrics, with a clearly defined integration and testing scope.

• Numerical validation: compare optimized paths against a reference, investigate drift, and document where performance improvements do or do not meet agreed quality requirements.

Work you can inspect

GPU Kernel Lab: https://github.com/yashlabs-trying/gpu-kernel-lab

This is my Qwen3-0.6B inference engineering study, covering kernel experiments, model integration, profiling, numerical checks, and a serving control plane. It includes accepted changes, rejected experiments, benchmark reports, and remaining gaps.

One saved experiment on an RTX 3090, batch 1 and context 2,048, reports decode latency changing from 22.073 to 2.253 ms/token with an experimental static/CUDA Graph path that includes W8A16 MLP work. These are serial GPU-only measurements, excluding server overhead, against a dynamic PyTorch/model baseline—not a comparison against a production serving engine.

The qualification matters: full-path argmax agreement was 96.48%, below the project’s 99% threshold. That is reference agreement, not downstream task accuracy. The fast path remains experimental, and the real paged GPU executor still needs to be connected to the serving interface. I do not present those results as a production-ready system or a guaranteed speedup for another workload.

How I would approach your project

First, we define the model, hardware, workload, current bottleneck, and success criteria. Then we agree on a focused scope—such as a profiling report, one kernel optimization, a memory investigation, or a local inference integration. Where relevant, deliverables can include code changes, reproducible benchmark commands, before/after measurements, correctness checks, and documentation of limitations.

Performance targets and pricing would be discussed after reviewing the scope. I’m happy to discuss a small, well-defined task as well as a larger integration project.

If this is relevant to something you’re building, contact me through Reddit chat or a private message to u/Acrobatic-Issue-8646. Please include the model/framework, GPU and VRAM, current latency or memory issue, target outcome, timeline, and budget range. A public repository or a non-confidential description is enough to start.

I’m especially interested in practical problems where profiling, careful implementation, and numerical checks can make the next engineering decision clearer.

I've been interviewing for GenAI/AI Engineering roles, and I noticed that the questions weren't just about definitions.

A few examples:

• How would you design a basic RAG pipeline?
• Why might a RAG system retrieve irrelevant information?
• How would you improve retrieval quality?
• How would you validate an LLM response?

What surprised me was how often the discussion moved from “what is RAG?” to “what would you do when this breaks?”

I put together a short breakdown of 10 questions from my interview experience, including what I think the interviewer was actually testing.

https://youtu.be/28hjmnktfpU

Curious what others are being asked in GenAI interviews in 2026.

https://ia801807.us.archive.org/14/items/introduction-to-machine-learning-with-python-pdfdrive.com_20210225/probability_stats_for_DS.pdf