Running Qwen 3.8 Flash Next Locally: Strix Halo, Strata, and Gufo

The Model

I spent a lot of time over the last couple of weeks testing Qwen 3.8 Flash Next on a Strix Halo mini-PC with 128 gig of RAM, and also on my NVIDIA-powered workstation. Qwen 3.8 Flash Next is an open-weight model from Alibaba which posts better scores than GPT-6 Luna (High) and Claude Haiku 5.5. It’s a kind of model known as mixture of experts, and it has 125 billion parameters and 6 billion activated per token.

Mixture of experts models are large models that save resources during generation by using a dynamically selected subset of the parameters during generation. The idea is to get the smarts of a 125-billion-parameter model by using the most appropriate 6 billion parameters, given the input context, to generate the next token.

I’ve been happily using Qwen 3.8 27B on my workstation, but it’s much slower on the Strix Halo because it is what we call a dense model. It has 27 billion parameters active during generation. Qwen 3.8 Flash Next also posts better scores than the 27B, but until very recently I could not run it well on my workstation because of its size.

The version of Qwen 3.8 Flash Next I settled on takes 111 GB to load, versus 23 GB for Qwen 3.8 27B.

But buzz on Reddit about this open-source project called Strata caught my attention. Strata works on both of these computers, and people were posting impressive performance numbers. It seemed impossible, and I honestly went into it on a rainy Saturday expecting to kill some time and be disappointed.

I was not disappointed, and I was pretty surprised by the whole experience.

It makes a great way to explain a few things about AI system architecture and design, and it served as a great reminder of why model evaluations and visibility into the entire process are necessary, as you’ll soon see!

Inference Engines

The service that takes the prompt, turns it into tokens, uses a model to generate the output tokens, and sends it out as text is called an inference engine.

When you use a cloud provider, the inference engine is hidden from you behind an application programming interface layer, also known as an API. We have a few de facto APIs, and most current inference engines support them.

AI applications like my own GuideAnts can use these standard APIs to use all of the popular cloud providers, like OpenAI and Microsoft Foundry, and use them the exact same way they use local inference engines - through a standard API. Which, in this case, is the OpenAI chat API.

Out of the box, GuideAnts uses a very popular inference engine named llama.cpp. I, like many others, chose it because it supports a lot of different models and hardware. It is a great project! But, running Qwen 3.8 Flash Next well, given its large memory footprint, requires a bunch of very specific optimizations. This is where projects like Strata and the one I wound up with in the end, Gufo, come in.

And, because they offer OpenAI-compatible APIs, configuring GuideAnts to use them is easy and directly supported.

So, in the end, I didn’t just test different versions of the model on different servers. I also used a couple different inference engines aside from llama.cpp, and I ran the Qwen 3.8 Flash model on OpenRouter during some tests.

Quantization

Now, like I said, I was skeptical.

One thing I was especially skeptical about was the amount of quantization involved, and how much it would affect quality.

Quantization involves taking the numbers in the model and using less precision. So instead of a 16-bit number with a lot of decimal places, we use less bits, which means fewer decimals and less precision.

So if you see Q4_K_XL, it means that most of the model’s numbers are 4 bits, but the numbers for the “keys” to values are larger. That’s made to save space at more expense to some numbers and less expense to others.

The recommended model quant for Strata, and the most popular, is from a team at IST Austria, and it comes in 2-bit and 3-bit flavors.

Such was my skepticism that I decided to start with a larger and slower quant from Unsloth, a 4-bit quant with XL key values.

First Tests

Following the docs, I had no trouble getting Strata started and serving inference on my two devices, and started testing using some manual test prompts.

My test prompt was: “make a detailed SVG image of a trout playing tennis.”

This is deliberately vague. The model is set to the recommended reasoning level, which is high, and it has Python, Bash, and some other tools and skills. These give it the ability to do things like visually inspect the file using the OpenAI SDK and Qwen 3.8 Flash Next’s own vision modality. It can use these if it discovers them in the process of thinking and examining its runtime environment.

We got images, and performance on the Strix was surprisingly good. Not blazing fast, but usable, with prefill from cold in the few hundreds of tokens per second and output consistently in the mid-40s.

The CUDA workstation was just blazing fast, hitting thousands of tokens per second for prefill and between 100 and 200 TPS sustained output.

So, just going from the initial output, things looked promising. But a closer look showed some real issues.

The Tool-Call Bug

Look at this conversation.

It took exactly one turn. It used its tools to do vision QA, and we got an image. It even tried to make a fun outfit for the trout. Looks like a success, right?

A closer look shows there were in fact a lot of problems.

Before we get into that, there is a major lesson here. This model, set on high thinking, completed the task - it used 5.5 million tokens in the process and made 90 total tool calls, which took time and money.

In this run, we saw 40 tool-call failures! Nearly half of all tool calls failed with the same error. The model wrote a bad tool call and, specifically, in every single case, it took the tool definition which specifies an argument named script:

And it added an extra argument, causing the tool call to fail.

It is remarkable that the task was completed successfully. It corrected itself and repeated the error several times!

After a few tests, I switched to the Qwen3.8-Flash-Next-UD-IQ4_XS quant, but it didn’t help, and I got similar results. With the two Unsloth q4 quants and Strata, 42% of all tool-call results failed because of the extra arguments argument, even as tests were passing overall due to the model’s tenacity!

So, then, I decided to test the IST Austria IQ3_S quant. This one surprised me! I got much better results. The extra arguments argument still showed up, but was now reduced to 7% of all tool calls.

Test Loops and the Fix

At that point, I started setting up test loops to collect more results from the two machines.

The test prompt was:

Select one activity from activities.json at random. Then choose an animal not already represented in the file list. Then create a detailed SVG of the animal engaged in the activity with a unique file name. Once finished, save it as a png file in ./final-images. Then clean up any work or test files created (other than the PNG and SVG) in the process and report the final results.

Here are a couple of my favorites. A quetzal baking bread and a beaver walking a dog.

It seems this desire to use arguments is deeply embedded in the model training. I’ve used these tools with a couple dozen models and haven’t ever seen anything quite like this bias. I ran some tests with Qwen 3.8 Flash on OpenRouter and saw it there too!

Because I own the tool definitions - remember, they come from the harness - I can, and did, address this by changing the tool definition to use arguments instead of script, and that entire class of errors disappeared.

The big lesson here is that it required a combination of evaluation tests, detailed telemetry, and analysis, but we were able to move the quality needle and the cost needles by tuning the tool-call description (remember, that is part of the prompt) and paradoxically by using a smaller quant.

Missing Spaces, and Gufo

Unfortunately, we had another issue which was causing around 13% of all tool calls to fail, and overall, getting rid of the arguments failure did not have a big impact on the total error rate. Instead, we saw a different error which we could not fix become very common!

Now the inference engine output was missing space tokens somewhat randomly in code. This is not something the harness can fix!

I tried a few things, but without any success with Strata, and so I decided to try another inference engine made specifically for the Strix hardware, named Gufo.

Gufo bills itself as “The Open Source Strix Halo AI engine”, so the downside is that I don’t have the option of running it on the CUDA workstation.

I used the Unsloth UD_Q4_K_XL quant and the total failure rate for all tool calls was reduced to 10%, and they were just normal “oops the code didn’t work” kinds of failures of all different kinds.

The shocking number is that I am seeing perfectly usable prompt processing times of around 1400 TPS and sustained output around 40 TPS which is quite usable.

On my CUDA workstation, I will be sticking with Qwen 3.8 27B for most of my tasks because they are tested and the workflows ‘work’, but these Strix Halo results have me reconsidering how I lay out my processing budget and where I run certain kinds of tasks.

This “Quokka walking the dog” image was produced using Gufo. The run made two successful QA vision calls and made adjustments. The turn used only 29 tool calls!

The moral of the story is - this is why we test and why evals are important. I came into this skeptical and with some bias based on lots of experience. I came out with a powerful addition to my toolkit!

As always, I want to say how much I appreciate the open-source community, the creators and maintainers of these projects, and the good people who share their work and experiences.

Until next time, this is Doug Ware, thanks for reading!