DeepSeek-V3 Technical Report

The paper claims DeepSeek-V3 is a 671B-parameter MoE model that activates 37B parameters per token.

“We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.”

quantity · unclear

consistent · high confidence — Matches the architecture spec restated later (61 layers, 256 routed experts, 8 activated, hidden dim 7168) and is consistent with widely reported specs for DeepSeek-V3 across the community.

To check: Cross-check against the released model config.json / HuggingFace card for DeepSeek-V3.

The paper states the pre-training corpus was 14.8T tokens, with SFT and RL post-training on top.

“We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities.”

quantity · unclear

consistent · high confidence — This token count is the figure cited consistently throughout the rest of the paper (hyper-parameter section, comparison to Qwen2.5's 18T) and widely repeated in secondary coverage.

To check: No independent corpus audit is possible since the data is not released; consistency is internal only.

The authors claim their model beats all open-source peers and is comparable to leading closed-source models.

“Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models.”

assertion · unclear

consistent · high confidence — This was corroborated by independent community benchmarking after release in late 2024/early 2025 — DeepSeek-V3 was broadly regarded as the strongest open model at the time and competitive with GPT-4o/Claude 3.5 Sonnet on many tasks, though it trailed on some (e.g. SimpleQA, SWE-Bench).

To check: Third-party leaderboards (LMSYS Chatbot Arena, independent evals) from Jan 2025 onward.

Full training of DeepSeek-V3 is claimed to have consumed 2.788M H800 GPU hours.

“Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training.”

quantity · unclear

consistent · medium confidence — This is the headline compute figure that became famous industry-wide; it is self-reported and unaudited, but I know of no credible counter-measurement, only debate about whether GPU-hours is the right lens for 'full training cost' (it excludes R&D, prior experiments, and capex).

To check: No independent audit exists; would require access to DeepSeek's cluster logs.

The authors claim the entire training run was stable, with no irrecoverable loss spikes and no rollbacks.

“Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks.”

assertion · unclear

unverifiable · medium confidence — A private operational claim about a specific training run with no loss curve shown for the main run; plausible given the maturity of the DeepSeek-V2 stack it's built on, but not independently checkable.

To check: Would require the actual training loss logs, which are not published.

The paper claims priority: the first validation of FP8 mixed-precision training at extremely large scale.

“In this work, we introduce an FP8 mixed precision training framework and, for the first time, validate its effectiveness on an extremely large-scale model.”

contrarian · unclear

contested · medium confidence — This priority claim is in tension with prior work the paper itself cites, notably FP8-LM (Peng et al., 2023b), which demonstrated FP8 training on a GPT-175B-scale model. DeepSeek's contribution is arguably the fine-grained tile/block quantization and CUDA-core accumulation fix at frontier scale with strong downstream benchmark results, not literally the first FP8 pretraining at large scale.

To check: Compare against FP8-LM (Peng et al. 2023) and NVIDIA/Microsoft FP8 training publications predating Dec 2024.

Total training cost is put at $5.576M, conditional on an assumed $2 per H800 GPU-hour rental price.

“Assuming the rental price of the H800 GPU is $2 per GPU hour, our total training costs amount to only $5.576M.”

quantity · unclear

contested · medium confidence — The arithmetic (2.788M hours × $2 = $5.576M) checks out internally, but this became one of the most contested figures in AI in early 2025: analysts (e.g., SemiAnalysis) argued the figure excludes GPU capital expenditure, salaries, data curation, and — most pointedly — the cost of the many prior research/ablation runs the paper itself says are excluded, so it understates what it actually cost to build a frontier-class model.

To check: Compare to independent capex estimates for a comparable H800/H100 cluster and to DeepSeek's disclosed cluster size (reportedly larger, per various trade-press reports).

The authors bound their own cost figure: it excludes prior research and ablation experiments.

“Note that the aforementioned costs include only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data.”

assertion · unclear

consistent · high confidence — This is an accurate, self-aware scope disclosure that directly explains why the $5.576M figure is a floor, not a full R&D cost — it is the caveat that fuels the controversy noted in claim 6.

To check: No external check needed; it's a stated methodological boundary, consistent with the report's framing.

Pre-training throughput is reported as 180K H800 GPU hours per trillion tokens, i.e. 3.7 days on a 2048-GPU cluster.

“During the pre-training stage, training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours, i.e., 3.7 days on our cluster with 2048 H800 GPUs.”

quantity · unclear

consistent · medium confidence — Internally consistent: 180K hours / 2048 GPUs ≈ 87.9 hours ≈ 3.66 days, matching the '3.7 days' claim almost exactly, and is derived from the same accounting as the other GPU-hour figures in Table 1.

To check: Recompute 180,000/2048 and compare to 3.7 days; matches.

Reported knowledge-benchmark scores: 88.5 MMLU, 75.9 MMLU-Pro, 59.1 GPQA, claimed best among open-source models.

“On educational benchmarks such as MMLU, MMLU-Pro, and GPQA, DeepSeek-V3 outperforms all other open-source models, achieving 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA.”

quantity · unclear

consistent · high confidence — Exactly matches Table 6 in the same document (MMLU 88.5, MMLU-Pro 75.9, GPQA-Diamond 59.1), and DeepSeek-V3 leading open-source models on these benchmarks at release was widely echoed in independent coverage.

To check: Table 6 in this same report; third-party MMLU/GPQA leaderboards from early 2025.

The paper claims DeepSeek-V3 beats o1-preview on specific math benchmarks such as MATH-500.

“Notably, it even outperforms o1-preview on specific benchmarks, such as MATH-500, demonstrating its robust mathematical reasoning capabilities.”

assertion · unclear

plausible · medium confidence — DeepSeek-V3's reported MATH-500 score (90.2, from Table 6) is high; I recall o1-preview's publicly reported MATH-500 accuracy being in a comparable but somewhat lower range, so the claim is directionally plausible, though I don't have OpenAI's exact o1-preview MATH-500 number memorized with certainty.

To check: OpenAI's o1-preview system card / independent MATH-500 evaluations of o1-preview.

Unbalanced expert load causes routing collapse and hurts efficiency under expert parallelism.

“For MoE models, an unbalanced expert load will lead to routing collapse (Shazeer et al., 2017) and diminish computational efficiency in scenarios with expert parallelism.”

assertion · unclear

consistent · high confidence — This is textbook MoE knowledge going back to Shazeer et al.'s sparsely-gated MoE work, and the efficiency argument under expert parallelism is standard in the distributed-training literature.

To check: Shazeer et al. 2017 'Outrageously Large Neural Networks'; GShard/Switch Transformer discussions of load imbalance.

Auxiliary load-balancing losses, if strong enough to work, degrade model quality.

“However, too large an auxiliary loss will impair the model performance (Wang et al., 2024a) .”

assertion · unclear

consistent · medium confidence — This tension between load-balance strength and model quality is a recognized issue in MoE training discussed in the cited Wang et al. 2024a (the loss-free balancing paper) and echoed by other MoE practitioners, though the magnitude is architecture-dependent.

To check: Wang et al. 2024a ('Auxiliary-loss-free load balancing strategy for mixture-of-experts').

The bias-term, auxiliary-loss-free routing keeps load balanced while beating pure auxiliary-loss baselines.

“Through the dynamic adjustment, DeepSeek-V3 keeps balanced expert load during training, and achieves better performance than models that encourage load balance through pure auxiliary losses.”

assertion · unclear

plausible · medium confidence — Backed by the paper's own Table 5 ablation across two scales; largely true but not uniform — MMLU and MBPP actually regress slightly at the 228.7B scale, so 'better performance' holds on most but not all reported benchmarks, a nuance the paper elsewhere phrases more carefully as 'most.'

To check: Table 5 in this report; independent replications of loss-free MoE balancing (e.g., by other labs adopting the technique).

The authors claim to have proven that a multi-token prediction objective improves model performance.

“We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance.”

assertion · unclear

plausible · medium confidence — 'Prove' overstates what a two-scale ablation (Table 4) shows — consistent gains on most benchmarks, with at least one metric (MMLU at large scale) regressing slightly. The underlying idea builds on Gloeckle et al. 2024's MTP work, which the paper cites and modifies.

To check: Table 4 in this report; Gloeckle et al. 2024 'Better & Faster Large Language Models via Multi-token Prediction'.

FP8 training is claimed to stay within 0.25% relative loss error of a BF16 baseline.

“Notably, compared with the BF16 baseline, the relative loss error of our FP8-training model remains consistently below 0.25%, a level well within the acceptable range of training randomness.”

quantity · unclear

plausible · low confidence — Self-reported result from smaller validation runs (~1T tokens at DeepSeek-V2-Lite/V2 scale, per Appendix B.1), not from the actual 671B/14.8T run; plausible given the engineering detail described (fine-grained quantization, CUDA-core accumulation), but I cannot independently confirm the exact figure.

To check: Appendix B.1 of the paper; independent reproduction of FP8 vs BF16 training on comparable MoE scales.

Running all three Linear GEMMs in FP8 is claimed to double compute speed versus BF16, in theory.

“This design theoretically doubles the computational speed compared with the original BF16 method.”

assertion · unclear

consistent · high confidence — NVIDIA Hopper Tensor Cores do offer roughly 2x FP8 throughput over BF16/FP16, a well-documented hardware spec, and the paper honestly flags this as theoretical rather than measured end-to-end speedup.

To check: NVIDIA H100/H800 architecture whitepaper Tensor Core throughput tables for FP8 vs BF16.

In the authors' cluster NVLink delivers 160 GB/s against InfiniBand's 50 GB/s, a 3.2x ratio.

“NVLink offers a bandwidth of 160 GB/s, roughly 3.2 times that of IB (50 GB/s).”

quantity · unclear

plausible · medium confidence — Consistent with the known fact that H800 (the China-export SKU) has NVLink bandwidth deliberately reduced relative to full H100, and 50 GB/s IB matches a common single 400 Gb/s NIC-per-GPU configuration; I can't independently verify DeepSeek's exact per-GPU NVLink allocation but the ratio and absolute numbers are plausible for this hardware generation.

To check: NVIDIA H800 spec sheet / export-compliant SKU documentation; DeepSeek's cluster network topology if disclosed elsewhere.

The communication design leaves headroom to select up to 13 routed experts at unchanged communication cost, versus the 8 actually used.

“This implies that, although DeepSeek-V3 selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts (4 nodes $\times$ 3.2 experts/node) while preserving the same communication cost.”

quantity · unclear

plausible · medium confidence · novel — This is a straightforward derived-arithmetic consequence of the stated 4-node cap and 3.2x bandwidth ratio (4 × 3.2 = 12.8 ≈ 13), internally consistent, but it's a theoretical headroom claim not a tested configuration.

To check: Would require DeepSeek to actually run a 13-expert configuration and report communication cost, which is not done here.

Twenty streaming multiprocessors suffice to saturate both interconnects under their custom all-to-all kernels.

“Overall, under such a communication strategy, only 20 SMs are sufficient to fully utilize the bandwidths of IB and NVLink.”

quantity · unclear

plausible · low confidence — A specific engineering result from their custom warp-specialized kernels; plausible given the detailed kernel description (10 channels, dynamic warp allocation) but not independently verifiable without their code/benchmarks.

To check: DeepSeek's open-sourced communication kernels (if released) profiled for SM utilization.

FP8 GEMM accumulation on H800 Tensor Cores retains only about 14 bits, far below FP32.

“However, we observe that the accumulation precision of FP8 GEMM on NVIDIA H800 GPUs is limited to retaining around 14 bits, which is significantly lower than FP32 accumulation precision.”

quantity · unclear

consistent · high confidence — This became a well-documented and widely discussed limitation of NVIDIA Hopper's FP8 Tensor Core accumulation path after this paper's publication, cited by subsequent low-precision training work; the paper's own detailed explanation (32 mantissa products, 14-bit addition width) matches known Hopper microarchitecture behavior.

To check: NVIDIA Hopper architecture documentation on Tensor Core MMA accumulation; subsequent papers citing this exact 14-bit limitation.

A preliminary test found nearly 2% maximum relative error from limited Tensor Core accumulation at inner dimension K=4096.

“Taking GEMM operations of two random matrices with K = 4096 for example, in our preliminary test, the limited accumulation precision in Tensor Cores results in a maximum relative error of nearly 2%.”

quantity · unclear

plausible · low confidence — Explicitly framed as a 'preliminary test,' consistent in order of magnitude with the known worsening of FP8 accumulation error at large inner dimensions, but not something I can independently confirm at the precise 2% figure.

To check: Reproduction of the described random-matrix GEMM test on H800/H100 hardware.

An accumulation-promotion interval of 128 elements is identified as the minimal setting that improves precision without substantial overhead.

“Based on our experiments, setting $N_{C}=128$ elements, equivalent to 4 WGMMAs, represents the minimal accumulation interval that can significantly improve precision without introducing substantial overhead.”

quantity · unclear

plausible · low confidence · novel — A specific tuning result from their own kernel experiments; the concurrent-WGMMA overlap explanation is architecturally sound for Hopper but the exact optimal interval is an empirical choice I cannot verify independently.

To check: Would require replicating the CUDA-core promotion kernel and sweeping N_C on H800/H100.

The authors recommend that future accelerators implement higher-precision FP8 GEMM accumulation in hardware.

“Future chips need to adopt higher precision.”

assertion · unclear

plausible · medium confidence — A reasonable engineering recommendation grounded in their own measured 14-bit accumulation limit; this is an opinion/design suggestion to vendors rather than a testable empirical fact, though NVIDIA's subsequent Blackwell generation did move toward finer-grained (microscaling) formats, which the paper itself notes.

To check: NVIDIA Blackwell Tensor Core specifications for FP8/microscaling accumulation precision, released after this paper.

A near-memory casting approach would cut off-chip memory access by roughly half.

“In this case, BF16 elements can be cast to FP8 directly as they are read from HBM into the GPU, reducing off-chip memory access by roughly 50%.”

prediction · unclear

unverifiable · low confidence · novel — This is a speculative architectural proposal for hardware that doesn't exist (near-memory computing for FP8 casting); the '50%' is a back-of-envelope estimate (halving bytes moved by casting before the full read), plausible in principle but untested.

To check: Would require an actual near-memory-compute GPU implementation to measure real memory traffic reduction.

Storing AdamW optimizer moments in BF16 rather than FP32 caused no observable performance loss.

“We adopt the BF16 data format instead of FP32 to track the first and second moments in the AdamW (Loshchilov and Hutter, 2017) optimizer, without incurring observable performance degradation.”

assertion · unclear

plausible · medium confidence — BF16 optimizer state storage is a known memory-saving technique in the low-precision training literature, and the paper hedges by keeping master weights/gradients in FP32; the 'no observable degradation' claim is self-reported but plausible given this partial hedge.

To check: Compare against other large-scale training reports using BF16 AdamW moments (e.g., Meta, other MoE labs).

The base model is claimed to be the strongest open-source base model, beating LLaMA-3.1 405B on most benchmarks.

“Overall, DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base, and surpasses LLaMA-3.1 405B Base in the majority of benchmarks, essentially becoming the strongest open-source model.”

assertion · unclear

consistent · high confidence — This claim was broadly corroborated by independent third-party evaluations after release — DeepSeek-V3-Base was widely regarded as the strongest open base model of its era, though the comparison here is run entirely in DeepSeek's own internal evaluation harness, a point worth flagging.

To check: Independent base-model benchmarking from Hugging Face's Open LLM Leaderboard or similar third-party harnesses, early 2025.

Per-token training cost for the 671B MoE is claimed to be far below that of 72B or 405B dense models.

“Under our training framework and infrastructures, training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours, which is much cheaper than training 72B or 405B dense models.”

assertion · unclear

plausible · medium confidence — Directionally sound — a 37B-active MoE genuinely requires much less compute per token than a 72B or 405B dense model doing full forward/backward on all parameters — but the claim gives no actual GPU-hour figures for Qwen2.5-72B or LLaMA-3.1-405B training, so the comparison is asserted rather than shown.

To check: Published or leaked training compute figures for Qwen2.5-72B and LLaMA-3.1-405B, compared against the stated 180K GPU-hours/trillion tokens.

MTP ablations show consistent gains on most benchmarks at both tested scales.

“From the table, we can observe that the MTP strategy consistently enhances the model performance on most of the evaluation benchmarks.”

assertion · unclear

consistent · high confidence — Matches Table 4 in this same document as reproduced in the transcript; the hedge 'most' is honest given that MMLU actually decreases slightly at the large scale (67.5 → 66.6).

To check: Table 4 in this report.

Auxiliary-loss-free balancing beats purely auxiliary-loss baselines on most benchmarks at both tested scales.

“From the table, we can observe that the auxiliary-loss-free strategy consistently achieves better model performance on most of the evaluation benchmarks.”

assertion · unclear

consistent · high confidence — Matches Table 5 as reproduced in the transcript; again honestly hedged as 'most,' since MMLU and MBPP both regress slightly at the 228.7B scale.

To check: Table 5 in this report.

The benefit is attributed to batch-wise rather than sequence-wise balancing scope: a batch-wise auxiliary loss matches the loss-free method.

“The experimental results show that, when achieving a similar level of batch-wise load balance, the batch-wise auxiliary loss can also achieve similar model performance to the auxiliary-loss-free method.”

assertion · unclear

plausible · low confidence · novel — An interesting and specific mechanistic finding (attributing the loss-free method's benefit to batch-wise rather than sequence-wise scope) that I have not seen widely reproduced elsewhere; it's a controlled small-scale (1B/3B) ablation, self-reported, and plausible but not something I can independently confirm.

To check: Independent replication of batch-wise vs sequence-wise vs loss-free MoE balancing at matched scale.

On 1B MoE models the loss-free and batch-wise auxiliary methods tie at 2.253 validation loss, versus 2.258 for sequence-wise.

“To be specific, in our experiments with 1B MoE models, the validation losses are: 2.258 (using a sequence-wise auxiliary loss), 2.253 (using the auxiliary-loss-free method), and 2.253 (using a batch-wise auxiliary loss).”

quantity · unclear

consistent · medium confidence — Precise self-reported numbers from a small-scale controlled experiment; internally consistent with the paper's claim in the surrounding text, though naturally I cannot independently verify these exact figures.

To check: Would require access to the underlying 1B MoE ablation logs.

The auxiliary-loss-free model shows more domain specialization among experts than the auxiliary-loss baseline.

“As illustrated in Figure 9, we observe that the auxiliary-loss-free model demonstrates greater expert specialization patterns as expected.”

assertion · unclear

plausible · low confidence — A qualitative visual observation from their own domain-load figure on two example layers (with more in an appendix); plausible mechanistically (batch-wise balance permits sequence-level specialization) but based on a small, cherry-pickable sample of layers.

To check: Full-layer results in Appendix C of the paper, if examined systematically rather than via the two example layers shown.

The SFT dataset comprised 1.5M instances across multiple domains with per-domain creation methods.

“We curate our instruction-tuning datasets to include 1.5M instances spanning multiple domains, with each domain employing distinct data creation methods tailored to its specific requirements.”

quantity · unclear

unverifiable · medium confidence — The SFT dataset is not released, so this is an unverifiable self-report of dataset size and composition, though the general description (mixing R1-distilled reasoning data with human-verified non-reasoning data) is consistent with the surrounding methodological narrative.

To check: Would require the released SFT dataset, which DeepSeek has not published.

Rule-based reward verification is claimed to be more reliable than model-based rewards because it resists reward hacking.

“By leveraging rule-based validation wherever possible, we ensure a higher level of reliability, as this approach is resistant to manipulation or exploitation.”

assertion · unclear

consistent · high confidence — This is a well-established principle in RL-from-verifiable-rewards (RLVR) style training — deterministic checkers (boxed math answers, compiler-run test cases) are inherently harder for a policy to game than a learned reward model, a point echoed widely in subsequent reasoning-model literature (e.g., DeepSeek-R1, OpenAI's o-series discussions).

To check: Comparative reward-hacking studies of rule-based vs model-based RMs in RLHF/RLVR literature.

DeepSeek-V3 is claimed to be the first open-source model above 85% on Arena-Hard.

“Furthermore, DeepSeek-V3 achieves a groundbreaking milestone as the first open-source model to surpass 85% on the Arena-Hard benchmark.”

quantity · unclear

plausible · medium confidence — Consistent with the reported 85.5 score in Table 7, and no open model I'm aware of at that time (Llama-3.1-405B at 69.3 in the same table) had surpassed 85%; but I cannot fully audit the entire Arena-Hard leaderboard history to confirm the 'first' claim with certainty.

To check: Arena-Hard's public leaderboard history around December 2024.

DeepSeek-V3 beats Qwen2.5-72B by 16.4 points on Chinese SimpleQA despite being pre-trained on 20% fewer tokens.

“On the factual benchmark Chinese SimpleQA, DeepSeek-V3 surpasses Qwen2.5-72B by 16.4 points, despite Qwen2.5 being trained on a larger corpus compromising 18T tokens, which are 20% more than the 14.8T tokens that DeepSeek-V3 is pre-trained on.”

quantity · unclear

consistent · high confidence — The score gap matches Table 6 exactly (64.8 − 48.4 = 16.4), and Qwen2.5's technical report does state an ~18T-token pretraining corpus, so the token-count comparison is also consistent with publicly stated figures.

To check: Qwen2.5 technical report's stated pretraining token count; Table 6 of this report.

R1 distillation improves accuracy at the cost of substantially longer average responses.

“Our experiments reveal an interesting trade-off: the distillation leads to better performance but also substantially increases the average response length.”

assertion · unclear

consistent · high confidence — Matches Table 9 exactly (MATH-500 Pass@1 74.6→83.2 with length 769→1510), and this accuracy/length trade-off from long-CoT distillation is a pattern that became widely observed across the field in 2025 (e.g., in R1-distilled model families generally producing longer outputs).

To check: Table 9 of this report; subsequent community reports on R1-distilled model verbosity.

The authors generalize that distilling from long-CoT reasoning models is a promising post-training direction beyond math and code.

“Our research suggests that knowledge distillation from reasoning models presents a promising direction for post-training optimization.”

assertion · unclear

plausible · medium confidence — This generalizes beyond the paper's own evidence base (tested only in math and coding domains, as the surrounding text itself acknowledges), but the broader claim was substantially vindicated afterward — 2025 saw widespread adoption of long-CoT/R1-style distillation well beyond math and code across many labs.

To check: Post-2024 adoption of R1-distillation techniques by other labs/models outside math and coding.

The MTP-predicted second token is accepted 85-90% of the time across generation topics.

“Based on our evaluation, the acceptance rate of the second token prediction ranges between 85% and 90% across various generation topics, demonstrating consistent reliability.”

quantity · unclear

plausible · low confidence — A self-reported figure with no disclosed evaluation methodology (which topics, sample size, decoding settings); the range is plausible for a well-trained single-token draft model in speculative decoding but not independently checkable from the paper alone.

To check: Reproducing acceptance-rate measurement on the open-sourced DeepSeek-V3 checkpoint with its MTP module.

Speculative decoding via the MTP module yields a 1.8x tokens-per-second improvement.

“This high acceptance rate enables DeepSeek-V3 to achieve a significantly improved decoding speed, delivering 1.8 times TPS (Tokens Per Second).”

quantity · unclear

plausible · medium confidence — An 85–90% acceptance rate for a single extra draft token is consistent with speedups in the 1.5–1.9x range typically reported in the speculative decoding literature, so the 1.8x figure is in a believable range, though serving configuration details (batch size, hardware) that would affect it are not given.

To check: Standard speculative-decoding throughput formulas given acceptance rate; independent benchmarking of DeepSeek-V3's MTP-based speculative decoding.

The authors concede the model's recommended deployment footprint may be too large for small teams.

“Firstly, to ensure efficient inference, the recommended deployment unit for DeepSeek-V3 is relatively large, which might pose a burden for small-sized teams.”

assertion · unclear

consistent · high confidence — Matches the deployment description elsewhere in the paper (32 GPUs minimum for prefilling, 320 GPUs for decoding) and was widely echoed in the self-hosting community's subsequent reports about the difficulty of serving the full 671B model without heavy multi-node infrastructure.

To check: Community reports/benchmarks on self-hosting DeepSeek-V3 (e.g., vLLM, SGLang deployment guides from early 2025).

The deployment-size and speed limitations are predicted to resolve themselves as hardware advances.

“Fortunately, these limitations are expected to be naturally addressed with the development of more advanced hardware.”

prediction · unclear

unverifiable · low confidence — A prediction about future hardware trends with no timeline or specific mechanism given, essentially an optimistic assumption rather than a testable claim.

To check: Track deployment-cost/latency of subsequent DeepSeek releases (e.g., V3.1, R2 if any) against new hardware generations.

LLM-as-judge feedback is framed as a general mechanism converting unstructured information into rewards, enabling self-improvement.

“The LLM serves as a versatile processor capable of transforming unstructured information from diverse scenarios into rewards, ultimately facilitating the self-improvement of LLMs.”

assertion · unclear

consistent · medium confidence — This restates a now-familiar idea (LLM-as-judge / RLAIF, building on the cited Constitutional AI work of Bai et al. 2022) rather than proposing something new; it's a reasonable framing of a real technique rather than a strong novel empirical claim.

To check: Constitutional AI (Bai et al. 2022) and subsequent RLAIF/LLM-as-judge literature.

The lab states AGI as its ultimate goal, pursued via open-source releases — the company's own mission statement.

“DeepSeek consistently adheres to the route of open-source models with longtermism, aiming to steadily approach the ultimate goal of AGI (Artificial General Intelligence).”

assertion · unclear

unverifiable · high confidence — A statement of institutional mission/intent rather than an empirical or checkable claim; it is consistent with DeepSeek's actual behavior (releasing open weights for its major models) but 'aiming to approach AGI' is not something that can be verified or falsified.

To check: Not checkable as stated; only DeepSeek's continued open-release pattern is observable.

GRPO removes the same-size critic model from RL by estimating the advantage baseline from group sample scores.

“Similar to DeepSeek-V2 (DeepSeek-AI, 2024c) , we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024) , which foregoes the critic model that is typically with the same size as the policy model, and estimates the baseline from group scores instead.”

assertion · unclear

consistent · high confidence — This accurately describes GRPO as introduced in Shao et al. 2024 (DeepSeekMath) and reused in DeepSeek-V2/V3/R1; GRPO subsequently became one of the most widely adopted RL algorithms across the field in 2025 reasoning-model training, corroborating both the description and its practical value.

To check: Shao et al. 2024 'DeepSeekMath'; widespread subsequent adoption of GRPO by other labs (documented in numerous 2025 RL-for-LLM papers).

On engineering coding tasks the model is behind Claude 3.5 Sonnet while ahead of open-source peers.

“In engineering tasks, DeepSeek-V3 trails behind Claude-Sonnet-3.5-1022 but significantly outperforms open-source models.”

assertion · unclear

consistent · high confidence — Matches Table 6 exactly (SWE-Bench Verified: 42.0 vs Claude's 50.8; Aider-Edit: 79.7 vs 84.2), and Claude 3.5 Sonnet's strength on agentic coding/SWE-Bench relative to other models at the time is well corroborated by independent evaluations.

To check: Table 6 of this report; independent SWE-Bench Verified leaderboard results from late 2024/early 2025.

The authors claim their FP8 mixed-precision training framework was validated against BF16 baselines at two model scales.

“We validate our FP8 mixed precision framework with a comparison to BF16 training on top of two baseline models across different scales.”

assertion · unclear

consistent · high confidence — This matches the well-known DeepSeek-V3 technical report's stated methodology for validating its FP8 training pipeline; FP8 vs BF16 comparison as a validation strategy is standard practice (cf. Peng et al. 2023 FP8-LM, Fishman et al. 2024).

To check: Compare against the published DeepSeek-V3 paper (arXiv:2412.19437) Appendix B, and any third-party reproduction attempts of DeepSeek's FP8 recipe.

The small-scale FP8-vs-BF16 baseline was a ~16B-parameter MoE trained on 1.33T tokens.

“At the small scale, we train a baseline MoE model comprising approximately 16B total parameters on 1.33T tokens.”

quantity · unclear

unverifiable · medium confidence — This is an internal training-run specification reported only by DeepSeek; I recognize the ~16B MoE as consistent with the scale DeepSeek typically uses for ablations (similar to DeepSeek-V2-Lite), but I have no independent way to confirm the exact token count or parameter count claimed.

To check: DeepSeek's released model checkpoints/configs or any independent audit of their training logs for this ablation.

The large-scale FP8-vs-BF16 baseline was a ~230B-parameter MoE trained on ~0.9T tokens.

“At the large scale, we train a baseline MoE model comprising approximately 230B total parameters on around 0.9T tokens.”

quantity · unclear

unverifiable · medium confidence — ~230B total parameters is close to DeepSeek-V2's ~236B total parameter architecture, so the scale is plausible, but the specific token count and the run itself are internal, unaudited details I cannot independently verify.

To check: Cross-reference with DeepSeek-V2 technical report architecture specs and any independent replication of this ablation.

FP8 training tracks BF16 training loss to within 0.25% relative error when high-precision accumulation and fine-grained quantization are used.

“We show the training curves in Figure 10 and demonstrate that the relative error remains below 0.25% with our high-precision accumulation and fine-grained quantization strategies.”

quantity · unclear

plausible · medium confidence — A sub-1% relative loss gap between FP8 and BF16 is broadly consistent with other FP8 training work (Peng et al. FP8-LM, Fishman et al. 2024 scaling FP8 to trillion tokens) reporting near-parity results, but the specific 0.25% figure is self-reported from a single EMA-smoothed curve comparison with no stated methodology for how 'relative error' is computed, so I cannot independently confirm the number.

To check: Independent reproduction of DeepSeek's FP8 recipe (e.g., via open-source frameworks implementing their tile-wise/block-wise quantization) compared against BF16 training curves.

Tile-wise quantization controls feature-outlier error but imposes a cost: different activation groupings for forward and backward passes.

“Although our tile-wise fine-grained quantization effectively mitigates the error introduced by feature outliers, it requires different groupings for activation quantization, i.e., 1x128 in forward pass and 128x1 for backward pass.”

assertion · unclear

consistent · high confidence — This matches the fine-grained quantization scheme (1x128 activation tiles, 128x128 weight blocks) described in the main body of the DeepSeek-V3 technical report, which I recall as a distinctive, well-documented design choice of that paper.

To check: Cross-reference against Section 3.3 of the DeepSeek-V3 paper describing the FP8 quantization scheme.

Activation-gradient computation (Dgrad) is unusually precision-sensitive compared with other quantized tensors.

“The results reveal that the Dgrad operation which computes the activation gradients and back-propagates to shallow layers in a chain-like manner, is highly sensitive to precision.”

assertion · unclear

plausible · medium confidence — Gradient quantization being more fragile than weight/activation quantization is a recognized theme in low-precision training literature (e.g., Xi et al. 2023 on 4-bit training instability, Wortsman et al. 2023 on low-precision training instabilities), so the qualitative claim fits known patterns, though the specific mechanistic framing ('chain-like' error propagation) is DeepSeek's own interpretation from a single ablation.

To check: Independent ablations quantizing Dgrad at varying granularities across different model scales and architectures.

Applying 128x128 block-wise quantization to activation gradients caused training divergence in a ~16B MoE run of ~300B tokens.

“Specifically, block-wise quantization of activation gradients leads to model divergence on an MoE model comprising approximately 16B total parameters, trained for around 300B tokens.”

quantity · unclear

unverifiable · low confidence — This is a single, non-replicated internal ablation result reported by the authors with no repeated trials, seeds, or divergence curves shown beyond the referenced figure; I have no independent basis to confirm whether divergence would reliably occur at this specific token count or is an artifact of this particular run.

To check: Independent reproduction of 128x128 block-wise Dgrad quantization on a comparable MoE model to see if divergence recurs.

The authors hypothesise, explicitly as a hypothesis, that token-imbalanced activation gradients produce token-correlated outliers that explain the precision sensitivity.

“We hypothesize that this sensitivity arises because activation gradients are highly imbalanced among tokens, resulting in token-correlated outliers (Xi et al., 2023) .”

assertion · unclear

plausible · medium confidence — Xi et al. (2023) indeed documents outlier and imbalance issues in low-bit training of activations/gradients, so grounding the hypothesis in that work is reasonable; the claim is explicitly framed as a hypothesis rather than a proven mechanism, which is appropriately hedged.

To check: Direct measurement of gradient outlier distributions across tokens in the ablated model, compared to the outlier patterns described in Xi et al. 2023.

Block-wise (128x128) quantization is structurally unable to handle token-correlated gradient outliers.

“These outliers cannot be effectively managed by a block-wise quantization approach.”

assertion · unclear

plausible · low confidence — This is a strong generalized claim ('cannot be effectively managed') inferred from a single divergence event plus an unverified outlier hypothesis; it is coherent with the general principle that coarser-grained quantization is worse at handling localized outliers (cf. SmoothQuant, GPTQ literature on outlier-aware quantization), but the categorical wording outruns the single-run evidence offered.

To check: Testing alternative block-wise granularities or outlier-aware block quantization schemes on the same Dgrad tensors to see whether any block-wise variant succeeds.

The auxiliary-loss-free load-balancing strategy produces more specialised experts than the auxiliary-loss-based baseline at every layer.

“The auxiliary-loss-free model tends to have greater expert specialization across all layers, as demonstrated in Figure 11.”

assertion · unclear

plausible · medium confidence — This is consistent with the intuition behind the companion paper (Wang et al. 2024a, 'Auxiliary-loss-free load balancing strategy for mixture-of-experts') from the same lab: removing an explicit balancing loss should let routing follow natural token-expert affinity more closely, increasing specialization, at some risk of load imbalance. I cannot independently verify the 'across all layers' universality claim from the figure alone.

To check: Independent expert-load analysis comparing aux-loss-free vs aux-loss-based MoE training on a held-out corpus like the Pile, replicated by outside researchers.