Sandeep Das

@sdas86.bsky.social

Running local models on a Mac and writing down what actually works.

trained a 9.4m model to do school maths with no llm at inference, then had a verifier pick one answer out of 96 samples. sampling more made it worse: 8.5% at 192, 8.3% at 288. the extras were plausible wrong answers it had to choose between. weights: hf.co/codelion/sprog-9m

wired two macs together over thunderbolt to see if local inference would go faster. it doesn't. what you get is room for a bigger model. any layer that spans the two machines pays the link, and that's far slower than memory bandwidth.

been benchmarking local models on a mac and tok/s turned out to be the wrong thing to watch. what you actually feel is the wait before the first token. it grows with prompt length and more ram doesn't help.

the 2x-not-10x coding thing rings true. the model writes stuff fast, but i still have to figure out if it did the right thing. half the time it already had a working fix in its first couple tries, i just couldn't tell which without running them. that checking is where the time goes now.

most brands can't say whether chatgpt or perplexity ever name them when someone asks for the best tool in their category. quick check: run ~10 buyer questions through the models, see who gets recommended. free scan that does it for you: merageo.com

meraGPT: an always-on hosted box running Nous Research's open Hermes agent. memory lives on the machine as plain-text files. you bring your own OpenAI/Anthropic key so it stays on the box, no token markup. $9.99/mo, no GPU so it's for agent + text work. meragpt.com