0. Table of Contents

• Introduction
• Code
• Usage
• Case

1. Introduction

Training and serving large language models (LLMs) has become a core business for AI providers. To ensure high-quality user experience while optimizing infrastructure costs, providers need to closely monitor the performance of LLM executions in production. However, existing performance profiling tools fall short in this context, which are either too coarse-grained to capture performance bottlenecks, or too intrusive that incur significant performance degradations.

In this paper, we present LMTracer, a fine-grained and real-time performance profiling framework designed for LLM services in production. We find that the main reason for existing techniques to be inefficient comes from the disruption in highly asynchronous GPU execution flows. To address this, our key idea is to embed profiling logic into the execution through graph-embedded fencing, which compiles lightweight fence kernels that realize the profiling logic as special trace nodes in the GPU execution graph, and streams buffered profiling data to CPUs on demand for keeping the execution of user kernels uninterrupted. LMTracer has been deployed in production for over four months. It incurs only 0.58% overhead on average across various LLM workloads (including training, serving, and fine-tuning), and has proactively uncovered 15 latent performance issues that would otherwise translate into severe system failures. After fixing these issues, we have observed 21.2% latency reduction and 24.5% throughput improvement in LLM services.

2. Code

The code for LMTracer is available here.

The directory is organized as follows.

Directory	Description	Source Code
csrc	Fence kernel implementations that read GPU clock registers and write the data to the shadow buffers. When one of the shadow buffers is full, fence kernels issue doorbell signals to CPU asynchronously through CPU-GPU mapped memory.	globaltimer.cu
		layout.h
		timer_kernel.cu
		timer_kernel.cuh
		timer_mem.cu
		timer_op.cu
examples	End-to-end benchmark and profiling presets.	deepseek_sglang.yaml
		qwen_vllm.yaml
		multi_exporter_config.yaml
lmtracer/core	Core runtime of LMTracer: probe configuration/context management, runtime hooks, shadow buffer readout, parsing, metrics, and exporter abstractions.	probe_context.py
		probe_config.py
		decorators.py
		hooks.py
		process_binder.py
		trace_buffer.py
		dump_parser.py
		metrics.py
		exporter.py
		time_sync.py
lmtracer/cuda	Python bindings/type stubs for CUDA extensions used by probe recording and clock reads.	globaltimer.pyi
		timer_kernel.pyi
		timer_mem.pyi
lmtracer/plugins	Framework integration layer. Patches or wraps upstream runtime entrypoints so LMTracer can initialize and flush traces during serving/training.	vllm/platform.py
		vllm/v1/worker/lmtracer_gpu_worker.py
		vllm/v1/worker/lmtracer_gpu_model_runner.py
		sglang/srt/model_executor/model_runner.py
		megatron/training.py
		torch/_inductor/scheduler.py
		torch/_inductor/wrapper.py
		torch/_inductor/cuda_combined_scheduling.py
tools	Utilities for distributed command execution and converting binary dumps into Perfetto traces.	dump_parser.py
		dump_parser_multitrack.py
tests	Benchmark scripts and sanity checks for OpenAI-compatible endpoints and Megatron training workloads.	run_benchmark.py
		openai_perf.py

3. Usage

After cloning the repository, you can install LMTracer with the following command:

pip install ./lmtracer --no-build-isolation

Note: LMTracer does not support editable mode (i.e., the -e option for pip install) of Python installation, or the hooks to LLM frameworks will be invalidated.

Then, LMTracer can be automatically enabled for supported frameworks (e.g., vLLM, SGLang, and Megatron) and models (e.g., DeepSeek V3, Qwen3, and GPT OSS) without any code changes.

The supported versions of the tested LLM frameworks are:

• vLLM v0.9.2, v0.11.2
• SGLang v0.5.6, v0.5.7
• MegatronLM v0.15.2

You can also specify custom profiling configurations to capture different levels of details for different frameworks and models.

An example of LMTracer configuration is as follows.

llm_engine: sglang
probes:
  # LLM modules
  - phase: model
    module: sglang.srt.models.gpt_oss.GptOssModel
  - phase: decoder_layer
    module: sglang.srt.models.gpt_oss.GptOssDecoderLayer
  - phase: mlp
    module: sglang.srt.models.gpt_oss.GptOssSparseMoeBlock
  - phase: attention
    module: sglang.srt.models.gpt_oss.GptOssAttention
  # Functions or even code lines
  - phase: norm
    function: sglang.srt.layers.layernorm.RMSNorm.forward_cuda
    # Optional: Flexibility for code-line profiling
    line_start: 109
    line_end: 126
exporters:
  # Exporter 1: Save binary trace data to files
  - module: lmtracer.core.exporter.FileByteDataExporter
    params:
      output_dir: /tmp/lmtracer_exports
  # Exporter 2: Export metrics to Prometheus
  - module: lmtracer.core.exporter.SGLangMetricPrometheusExporter
    params: {}

An example profiling output for LLM inference is as follows.

4. Case

After the real-world deployment, LMTracer has proactively uncovered 15 latent performance issues. Below we list representative issues that LMTracer has helped identify, and the corresponding subsequent failures that would occur if the issue is not handled immediately.

Component	Root Cause	Symptom	Subsequent Failure
Configuration	Improper layer placement	PP stages are much longer.	Users' SLA is not satisfied.
Configuration	Improper parallelism	TP groups use slow paths.	Users' SLA is not satisfied.
Configuration	Not using CUDA graph	Separate kernel launching.	Some GPUs straggle.
Framework	Unbalanced scheduling	Some GPUs are busier.	Inefficiency propagates.
Framework	KV cache eviction	Repeated computation.	Some GPUs straggle.
Framework	Prefill blocking	High TTFT for users.	Decoding GPUs are underutilized.
Hardware	NVLink down	Training process is stalled.	Training is silently halted.
Hardware	RNIC flapping	Intermittent unconnectivity.	Training tasks are failed.