Qwen

Qwen3-Embedding-0.6B

0.6B parameters · reasoning · Base · Qwen3 family

Long context — how it uses memory →
apache-2.0 hash verified source matched revision pinned
Good for: RAG & retrieval

Which version should I download?

Default pick: GGUF · Q8_0 — Near-lossless — best quality for this small retrieval model.

Set your rig to confirm it fits and to see if a higher-quality quant runs fully on your hardware.

Set your rig

Quantizations

Why these quants? Which quant should I pick? What do these mean?

GGUF · Q8_0

609.54 MB

Set your rig
Est. speed
—
Swarm
2S / 0L webseed OK

Near-lossless — best quality for this small retrieval model.

Magnet .torrent

Want a different quant? Request it on the board →

Runs fully on

RTX 5090 · Q8_0 RTX 5060 Ti 16GB · Q8_0 RTX 5070 Ti · Q8_0 RTX 5080 · Q8_0 RTX 5070 · Q8_0

Green = the model's best quant fits fully in GPU/unified memory at 8K context. Tap a card for its full "what runs on it" page.

Context

Advertised 32k · no independent evidence yet

No independent long-context evidence has been graded for this model yet — the advertised window above is the maintainer's number, not a usable-context claim.

Run it

Runtime completeness (Q8_0 torrent): llama.cpp ✅ Ollama – vision sidecar –

  • – Ollama — runs the GGUF directly; no Modelfile bundled
  • – vision sidecar — not a vision model

Context / KV 8,192 tokens · FP16 set above the quant table

Generic commands (no rig set). GPU-offload values assume the model fits on your GPU — set your rig for values tuned to your hardware.

Start the server 
llama-server -m Qwen3-Embedding-0.6B-Q8_0.gguf -c 8192 -ngl 999 --embeddings

Embedding model: POST to the /embedding endpoint (no chat completion).

Save as Modelfile next to the GGUF (Modelfile) 
FROM ./Qwen3-Embedding-0.6B-Q8_0.gguf
PARAMETER num_ctx 8192
PARAMETER num_gpu 999
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
# Chat template: Ollama uses the template embedded in the GGUF (no TEMPLATE directive needed).
Create 
ollama create qwen-qwen3-embedding-0-6b -f Modelfile
Run 
ollama run qwen-qwen3-embedding-0-6b

GGUF is not a first-class vLLM format.

vLLM GGUF support is experimental and single-file only; prefer safetensors/GPTQ/AWQ for production. If you must, pass the .gguf path to `vllm serve` with --load-format gguf on a recent vLLM.

transformers does not load GGUF weights.

GGUF weights run under llama.cpp, Ollama or LM Studio.

Load Qwen3-Embedding-0.6B-Q8_0.gguf, set the context length to 8192 tokens. Set GPU offload to Max (all layers).

MLX runs MLX-format weights only (Apple Silicon). This quant is a GGUF build.

GGUF weights run under llama.cpp, Ollama or LM Studio.

Technical details

Chat template

{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0].role == 'system' %}
        {{- messages[0].content + '\n\n' }}
    {%- endif %}
    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0].role == 'system' %}
        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
    {%- set index = (messages|length - 1) - loop.index0 %}
    {%- if ns.multi_step_tool and message.role == "user" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
        {%- set ns.multi_step_tool = false %}
        {%- set ns.last_query_index = index %}
    {%- endif %}
{%- endfor %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {%- set content = message.content %}
        {%- set reasoning_content = '' %}
        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
            {%- set reasoning_content = message.reasoning_content %}
        {%- else %}
            {%- if '</think>' in message.content %}
                {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
            {%- endif %}
        {%- endif %}
        {%- if loop.index0 > ns.last_query_index %}
            {%- if loop.last or (not loop.last and reasoning_content) %}
                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
            {%- else %}
                {{- '<|im_start|>' + message.role + '\n' + content }}
            {%- endif %}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}
        {%- if message.tool_calls %}
            {%- for tool_call in message.tool_calls %}
                {%- if (loop.first and content) or (not loop.first) %}
                    {{- '\n' }}
                {%- endif %}
                {%- if tool_call.function %}
                    {%- set tool_call = tool_call.function %}
                {%- endif %}
                {{- '<tool_call>\n{"name": "' }}
                {{- tool_call.name }}
                {{- '", "arguments": ' }}
                {%- if tool_call.arguments is string %}
                    {{- tool_call.arguments }}
                {%- else %}
                    {{- tool_call.arguments | tojson }}
                {%- endif %}
                {{- '}\n</tool_call>' }}
            {%- endfor %}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is false %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- endif %}
{%- endif %}

Stop strings

<|im_end|> <|endoftext|>

Evidence & provenance

Source

Provenance
https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
Revision pin
97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3
Manifest
Present

License

Name
apache-2.0
Commercial use
yes
Access
Open

How verification works →

File hashes (SHA-256)

  • qwen3-embedding-0.6b-q8-0/LICENSE cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30
  • qwen3-embedding-0.6b-q8-0/Qwen3-Embedding-0.6B-Q8_0.gguf 06507c7b42688469c4e7298b0a1e16deff06caf291cf0a5b278c308249c3e439
  • qwen3-embedding-0.6b-q8-0/README.md 5dc596e802ce79342678281d42ec4b98fbee2fee241758df330cdc35cc15d29f

Explore further

Appears in

What runs on 6 GB of VRAM What runs on a 8 GB Mac Coding models that fit 6 GB Writing models that fit 6 GB RAG models that fit 6 GB RAG models that fit 8 GB

Also consider

Get an email when a better quant fits your rig.

Set your rig once; we'll alert you when a new or better-fitting build lands.

Set your rig

Performance reports

Real-world throughput reported by the community (and scraped sources).

View performance leaderboard
Community

Reviews, sampler presets and community runtime reports. Be the first to contribute.

Reviews (0)

Log in to write a structured review of this model.

No reviews yet.

Sampler presets

Log in to share a sampler preset or vote on presets.

No presets yet.

Community runtime reports

Unofficial, community-submitted "it loads for me" reports. These are not the official verified-working badge — an admin reviews reports before anything is marked officially verified.

Log in to report whether a quant loads in your runtime.

No community reports yet.

selected to compare · pick at least 2

Compare ()