Today's Coding drill shifts away from the inference scheduling, tokenizer, and sampling problems covered in recent weeks toward another component nearly every LLM API gateway has to build itself: a token-bucket rate limiter. Unlike a generic API rate limiter, LLM rate limiting is measured in tokens, not requests — OpenAI and Anthropic both enforce dual RPM/TPM limits, and a single request can cost thousands of tokens instead of a flat 1. The core ideas covered here are lazy refill (computing how many tokens accrued using elapsed time × refill rate at check time, with no background timer), the check-then-deduct race condition (concurrent reads of the balance can over-admit requests), what to do when a single request's cost exceeds the bucket's total capacity, and extending the design to multiple API gateway instances sharing one global quota via an atomic Redis Lua script — using Redis's own clock rather than each server's local time. Today's practice question comes from a real question collected in an AI engineering interview question bank.
Today's Coding rotation goes a different direction from the batching schedulers and tokenizers covered in past weeks: implement a sample_token function that takes logits, temperature, top_k, and top_p. The core concepts are softmax's numerical stability trick (subtracting the max before exponentiating, or a large logit overflows to inf), temperature's edge case (T=0 is division by zero in the formula, and the industry convention is to fall back to greedy decoding instead of raising an exception), the standard application order (temperature reshapes the distribution first, then top-k trims to a fixed count, then top-p trims dynamically by cumulative probability), and the most common top-p bug — the cumulative-probability cutoff has to compare against the running total *before* adding the current token, not after. The practice question comes from a real AI engineering interview question bank, extended with edge cases like top_k exceeding vocabulary size and a zero-denominator renormalization.
Today's Coding rotation tackles a question Anthropic asks across nearly every engineering-adjacent loop: design and implement a single-GPU inference batching scheduler that accepts up to 100 inputs per batch while callers block synchronously for results. The core tension is throughput versus latency. Concepts covered: dynamic batching's two trigger conditions (batch-size cap vs. wait-time cap), length bucketing to cut padding waste, continuous batching (iteration-level scheduling) versus static batching, how PagedAttention manages KV cache memory with paging, and backpressure for bounding tail latency when the queue fills up. The practice question turns Anthropic's system-design prompt into an actual scheduler class you can implement, with a full breakdown and sample answer.
Today's Coding rotation is a real question from Anthropic's Technical Screen — implement a case-sensitive longest-match tokenizer: greedily match the longest possible vocabulary entry at each position, emit the exact token id and the consumed original text, and support two modes for characters that don't match anything — emit one unknown token per character, or coalesce a whole unmatched run into a single unknown span. Core concepts covered: the greedy longest-match-first strategy behind subword tokenizers like WordPiece, why a growing vocabulary forces you from a linear scan onto a trie (paired with Aho-Corasick-style failure links, the same idea behind Google's LinMaxMatch, to get from O(n²) down to linear), how the unknown-run merge choice actually affects downstream training, and the classic off-by-one traps in Python string slicing.
Today's Coding rotation works through a real xAI interview question — implement dynamic batching for token decoding, refilling finished slots in a fixed-capacity batch B while correctly handling three stop conditions (stop_token, stop_sequence, max_tokens) and keeping the slot_id-to-request_id mapping correct. Core concepts cover how continuous batching trades off GPU utilization against latency, using NumPy broadcasting and boolean masks instead of explicit loops, padding and attention masks for variable-length inputs, and why a production batch inference API should scale on queue depth rather than raw GPU utilization. Bonus: a meta-point on interview design — 2026 ML coding interviews increasingly skip rote algorithm recall in favor of realistic production scenarios like today's, testing engineering judgment instead.
ML coding interviews don't test whether you've memorized LeetCode templates — they test whether you can hand-write building blocks in NumPy (linear layers, residual connections, layer norm, causal self-attention) and read someone else's training/inference loop well enough to find the bug. Today covers four concepts: the real test in ML coding interviews is hand-implementation, not memorized answers; BPE tokenizer's core logic is 'count frequencies, merge iteratively, encode using the exact same merge order learned during training' — get the order wrong and encode no longer matches decode; batch inference trade-offs throughput against latency and has to handle padding waste across variable-length sequences; and the test for whether to vectorize a loop in NumPy is whether it has cross-element dependencies. The practice problem is a real technical-screen question from Glean — implementing a BPE tokenizer end to end, from training to encode/decode.
2026 ML coding rounds no longer just test 'can you build it from scratch' — they also test whether you can read code someone else broke. Today covers five topics: a state-machine design for LLM inference scheduling, strategies for debugging existing ML code, NumPy shape traps, leakage prevention in pandas time-series features, and computing AUC-ROC by hand. The practice problem is adapted from a recently leaked Anthropic OA: a simplified GPU request scheduler.
BytePlus ModelArk Coding Plan offers Lite ($10/month) and Pro ($50/month) subscriptions covering models such as DeepSeek-V4, GLM-5.2, and Seed-2.0 in tools including Claude Code and Cursor. Lite includes about 24,000 requests per month; Pro includes five times as many.
ML coding rounds don't test leetcode recall — they test whether you can implement attention, k-means, and other ML primitives from scratch using only NumPy, while articulating the shape and complexity at every step. Today covers five high-frequency topics: vectorized thinking, softmax numerical stability, shape tracking and complexity analysis, padding/masking for batch inference, and how to verify correctness when hand-coding algorithms.
AI Engineer coding interviews aren't identical to SWE — beyond LeetCode medium, you'll face ML-flavored problems (implementing a tokenizer, writing a batch inference pipeline, handling sparse matrices). Strategy: practice LeetCode medium to 70% pass rate, then spend remaining time on numpy/pandas operations, data processing pipelines, and ML-related programming problems.