[Feature] DeepSeek V3 optimization

[zhyncs]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Usage

User Guide for Existing System

https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3

Features

• Support CUDA Graph @HandH1998 @ispobock
• Use BF16 for bmm @zhyncs
• Improve the accuracy for FP8 @HandH1998 @zhyncs @ispobock
• Tuning FP8 GEMM @HandH1998 @zhyncs
• FusedMoE tuning for H200 E=256,N=256,device_name=NVIDIA_H200,dtype=fp8_w8a8.json @BBuf
• Fuse Sigmoid Gate moe_kernels.cu @ispobock
• FP8 GEMM CUTLASS implementation f8_blockwise_scaling_pr_branch @fsygd
• TP+DP Attention @Ying1123
• Replace moe_align_block_size @HandH1998 @zhyncs
• Support nextn speculative decoding @merrymercy
• Support overlap scheduler @merrymercy
• FP8 GEMM Composable Kernel implementation @HaiShaw
• Support Pipeline Parallelism @yizhang2077

Related resources

No response

[libratiger]

Very quick response !
I understand that the overlap scheduler is model-independent and is a general optimization that should be supported by default.
At least some special optimizations are needed?

[merrymercy]

The overlap scheduler is model-independent but has not been supported when using dp attention. We have a private branch for this and will upstream it soon.

[fengyang95]

Is the memory sufficient for an 8 gpus instance? This model size is too large.

[zhyncs]

Is the memory sufficient for an 8 gpus instance? This model size is too large.

671B works on H200 * 8 with FP8 (671 < 141 * 8)

[zhyncs]

Hi @fengyang95 You can also consider multi node.

If you do not have GPUs with large enough memory, please try multi-node tensor parallelism (help 1 help 2).

[zhyncs]

FYI Due to the tight schedule, SGLang v0.4.1 currently only provides preliminary support for DeepSeek V3. To make it run more cost-efficiently, we need to complete most of the optimizations mentioned above. If you are interested in any of the above optimizations, feel free to join the SGLang Slack for discussions or contribute a PR. We hope to complete these optimizations quickly and appreciate any discussion and contributions.

[zhyncs]

Update: SGLang v0.4.1.post1 supports CUDA Graph for DeepSeek V3, please use the latest version.

pip install "sglang[all]==0.4.1.post1" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer