Quant optimized for quality / speed on a Strix Halo 128GiB system. Possibly also beneficial on DGX Spark and similar systems.

The TL;DR is this quant achieves both superior quality and speed compared to homogenous Q6_K.

See the GLM version for more details on theory and comparisons.

GGUF

Model size

119B params

Architecture

mistral4

Hardware compatibility

We're not able to determine the quantization variants.

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Beinsezii/Mistral-Small-4-119B-2603-GGUF-HALO

Base model

Quantized

(36)

this model

Collection including Beinsezii/Mistral-Small-4-119B-2603-GGUF-HALO