DeepSeek V4 Flash gets smaller files and long-context tests

DeepSeek V4 Flash is being tested in 2-bit, 3-bit, and 4-bit GGUF files. The practical goal is to shrink a large AI model so it can run on personal hardware or smaller servers while lowering memory use and inference cost. Related experiments combine vLLM 0.24.0, llama.cpp branches, DSpark, NVFP4, and quantized KV cache work to test long context and concurrent use.

One setup targets two DGX Spark machines with 1.5 million context tokens, a 3 million KV token pool, and 12 concurrent requests, while another checks whether a REAP-pruned NVFP4 DeepSeek V4 Flash setup stays steady at long context on a single Spark machine. A llama.cpp branch has merged fixes for quantized KV cache behavior and compares perplexity against f16 to watch for quality loss. An RTX 5090 MoE setup reports token generation around 22.7 to 21.3 tokens per second and prompt processing around 1105 to 927 tokens per second across prompts from 8,192 to 65,536 tokens.

There is still some confusion around model formats, including a case where a GGUF release is described as MXFP4 even though the original model page lists other tensor types, so users need to verify what they are actually running.