It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
dwa3592 2 minutes ago [-]
>>It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
reidrac 4 minutes ago [-]
> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
dwa3592 22 seconds ago [-]
i am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
1 minutes ago [-]
hparadiz 1 hours ago [-]
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
trollbridge 28 minutes ago [-]
I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
neomindryan 50 minutes ago [-]
Thank you for sharing!
broabprobe 4 minutes ago [-]
[delayed]
deltamidway 19 minutes ago [-]
He's shown me his set up in his basement. It's sick! Talk about your 3d printer next!
throwaway2027 2 hours ago [-]
That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows)
Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around...
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
neomindryan 23 minutes ago [-]
hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.
okokwhatever 18 minutes ago [-]
To me context means everything.
Tokens per second is a great metric but in the real world context window is the deal breaker when a real use case is on the table.
Truly amazing. This gives a peek into the future for what's possible.
neomindryan 2 hours ago [-]
Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish.
The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.
otherjason 1 hours ago [-]
This reads as pretty clearly AI-generated text, which is against HN guidelines.
FL410 29 minutes ago [-]
The PR? He said it was AI in the comment you replied to...
I don't think the post itself reads like AI at all, but that's just me.
logicallee 8 minutes ago [-]
I think "this" refers to its parent comment. Part of it sounds like Claude wrote it. AI-generated comments aren't allowed on HN.
pkghost 55 minutes ago [-]
Here's the thing: life also imitates art. If you invert your load-bearing assumption, it could be that he just reads too much slop. But my honest take? You might be right.
OsamaJaber 11 minutes ago [-]
[dead]
rvba 54 minutes ago [-]
Sorry for asking here but literally nobody knows:
Android studio connected to a local model disconnects automatiacally after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens. I do 130 sec.
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
https://news.ycombinator.com/item?id=48354801
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
A 10 year old Xeon is all you need
https://news.ycombinator.com/item?id=48353348
The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.
I don't think the post itself reads like AI at all, but that's just me.
Android studio connected to a local model disconnects automatiacally after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens. I do 130 sec.