Because all Chinese users must use a VPN to access Codex.

I only had the patience to read long books in high school; now I really want to but they're too difficult.

Have you tried audiobooks?

I found this Librivox audiobook to have a good narrator.

https://librivox.org/crime-and-punishment-version-3-by-fyodo...

I listened to the first third of the book while at work (the second Librivox version) [1]. I'm not sure if he pronounced the names right, because I've heard others pronounce them differently, and I'm not sure who's correct. In any case, having listened to the names, I found it quite easy to read the rest of the physical book, having the pronunciation and prosody of many the names already embedded in me. Having multiple names per person was quite confusing, still.

https://librivox.org/crime-and-punishment-version-2-by-fyodo...

You just have to build a habit. Nothing happens unless it becomes habitual.

Of course, that "just" is doing a lot. I'm saying it's doable, not that it's easy.

The Mac Mini's price is already quite low while its performance keeps getting stronger.

What are the use cases for these small models? Is there anyone using models of this scale in their daily life who could share their experience?

I have vLLM running on a Linux machine in my basement, connected with Tailscale, and I use small models as part of tasks like this:

- Transcribing scanned documents into formatted text

- Captioning/describing images and classifying them for audience suitability (includes anti-spam)

- Matching documents with relevant Wikipedia pages for tagging

I don't use them like frontier models. I break the work down into micro-tasks with one clear goal for each prompt. I write a lot of glue software to make the complete flow work. I was working on all of these tasks before LLMs appeared on the scene. The LLMs have allowed me to replace a lot of complicated code with less code plus a model, while achieving better results.

I use local models for reasons of cost and control. I already had the workstation and GPU. The only running cost is electricity. I have used proprietary models from OpenAI and Google for some of these tasks, but I also encountered churn when the models I built my tools around were retired. I don't worry about that when I have the weights saved locally.

I've got a home-built dictation app that uses a local model to clear up the text and fix grammar. It was super easy to build. I’m extending it to capture meeting notes and summarise too. All on-device.

I saw a little app the other day, I think someone posted on here, that looks at your screenshot and renames the file based off the contents of the file.

There's tons of little examples like that. For a lot of use cases, you really don't need the frontier models.

That's a great user case. Am sorry using parakeet but sometimes it garbles up things. Can you open source it?

Mac only I'm afraid, but I already did [1]. Packaged it too as I figured it might be useful to others [2] (and I'd want to install it on machine's that I might not have Xcode on)

[1]: https://github.com/robgough/dictator [2]: https://dictator.robgough.net

I spent the best part of a couple weeks making improvements and tidying up the UI, but to actually get something working was essentially only a couple of prompts.

Handy is open source and works flawlessly for me with Parakeet v3.

I think small models have a very good niche for specific tasks. I utilise a fine tuned Phi-4 model (smaller than this one) that fits in about 3.5gb of RAM (not vram) for the document processing side of things for the desktop app I develop (a bit of a shameless plug - whistle-enterprise.com).

If you have a very specific idea for local model use you can find a way to make it work very well, you don't even need to have a graphics card or NPU chip. You just have to be extremely constrained in how it's used. I think as a generic chatbot they're not great, I'd use a hosted SOTA model and I'm a big fan of local LLMs myself.

Thank you for sharing your usecase! I like your product very much!

Could you talk a bit how you did the finetuning? Did you use unsloth or any other tool and how went the verification to proof the outcome?

Thank you!

Yea absolutely, but man, where to even start, it is very specific.

Fundementally I didn't use any wrappers like unsloth or axolotl, although I have used the latter before a year or two back and it was good, but I needed something very very custom. I also wanted the whole fine tuning pipeline to exported OpenVino model to be seamless.

I heavily leaned on codex, claude and some manual sleuthing around the internet to understand what I needed. I'd played about with QLoRA finetuning with axolotl before and felt most comfortable with that. So I needed to keep everything as stripped down as possible and figured I can just utilise the 3 main huggingface libraries (transformers, peft and datasets) and also bitsandbytes (as suggested by claude to quantize the model to keep this working on my GPU) along with some custom scripts generated by claude/codex (each cross referencing each other) that will do the different stages of the training run.

The next part was the data. Obviously didn't have access to thousands of meetings and associated output documents but I did have a 3090ti sitting there and a codex subscription. So I set about working out what format I needed the data in (many thanks again, to claude/codex) and started generating hundreds of different transcripts, different amounts of speakers, content, tones, subjects, spelling mistakes - like all the different things you could think a meeting would have. Then it's a case of actually generating a good meeting document off the back of the transcripts and creating the "gold standard" that we'd use.

I'm going to gloss over a lot here as I'd rather not detail it as it relates to some propriatary stuff that I had to work through, but you basically pair the transcripts together and run the training.

At the verification stage, there was pretty much 3 things:

1. "just" do some regex string matching to see if there's any of the source transcript key facts in the output to ensure fact preservation. Same with owner fabrication (who said what), I don't want something attributed to someone when it wasn't them that said it and then finally markdown validation.

2. Using codex/claude to validate the transcript and output from the model - I used the latest frontier models, probably overkill for my task, but they were good at the job

3. Finally me going through some actual recordings of myself, groups, meetings and manually verifiying the output

So a fair bit of work, and for context I'm on version 10 now, so it's been a journey!

Thank you so much for the insight, I really appreciate it!

I use small models like Gemma to improve transcriptions from ASR models amongst other micro-tasks. I actually built out a fine-tuning whisper pipeline with all local (smaller) models meaning no cloud/big-tech co is able to train/sell my (private) data.

Repo is https://github.com/Rebreda/listenr - mainly geared toward Whisper fine-tuning, AMD hardware and local inference

I use them for research on new features. If my feature is going to interact with a frontier language model in prod, I start with these free local ones which are all competent enough to produce structured output, make tool calls, interact with mcp etc. I don’t care much for the content at the early phase of engineering, I care about the schema & failure modes.

Then when I’m getting close to feature-complete, I’ll move to a hosted frontier model for the final integration.

Cost savings are enormous if you’re making dozens of calls to language models a minute.

I don't know about this model, but the next one up, the 31B I've been using as an agentic coding assistant in OpenCode, and basically anything that's easy enough that I'd trust Sonnet to handle, I trust Gemma 4 to handle and it's been doing a great job, it surprises me positively much more often than negatively. I not infrequently run into situations where Gemma 4 fails to do the task and I switch to Opus 4.7 and it fails also.

In theory, locally you'd use these where lossiness is acceptable for audio transcription and image labeling (as simple examples).

In practice I haven't got around to building something around multimodality since I'm primarily using their text generation capabilities.

"Small" models are the ones I can run myself on my own terms. LLMs aren't useful enough for me to justify spending hundreds of euros on a GPU with 16GB VRAM or something, and that's assuming I have the rest of the desktop just laying around. Back when I checked (before the RAM price hike), these models weren't meaningfully better than 4-8GB ones anyway, you'd have to go for the top tier cards at 24 or 32 GB iirc to get something vaguely in the direction of the SaaS versions, and that was absolutely out of my budget. Even if that changed, so have hardware prices so it'd probably still work out the same

I've used Gemma for reviewing and categorizing my writing online over several years (~5 million words across a forum for an OSS project I work on, HN, reddit, etc.), experimenting with training LoRAs (again, on my own writing, since I don't have to worry about ethically sourcing the data if it's all mine), and I'm currently using it to perform web searches and extract data about a specific type of business. It's plenty smart to use a web search MCP to find all the businesses of the right type in a given city, read their website, extract business address, phone number, etc. among other things, and de-dupe and cross-check other sources.

I found Gemma 4 to be better, or at least more nuanced, than Gemini 2.5 Flash. And, the new Gemini 3.5 Flash is very good but is unrealistically expensive (ten times more expensive than DeepSeek or MiMo). So, since I don't need extremely fast performance, a self-hosted Gemma 4 wins for a bunch of stuff.

I've also found Qwen 3.6 27B to be shockingly good at finding security bugs for its size. It beats several larger models, and is close to Gemini Pro 3.1 (but Gemini 3.5 Flash surprisingly beats it soundly). Since it only costs electricity, and my electricity is cheap and 100% renewable, I can use it more broadly than I might otherwise use a hosted model.

All that said, the smart money is still on buying the subsidized tokens from the providers that offer them, rather than buying the hardware needed to run models that are 30+GB in size, as all of the ones I've been using regularly are (8-bit quantization, as they get a little dumber for every bit you drop below that). A $100 subscription to Claude or Codex currently provides access to the best models at a heavily discounted rate. And, DeepSeek/MiMo are extremely cheap, one or more orders of magnitude cheaper than the top models from Anthropic or OpenAI, if you need an API for automated usage. I spent about $4000 on my two inference machines (a Strix Halo with 128GB unified RAM, and a new desktop build based around two cheap old 32GB AMD data center GPUs), which buys a lot of tokens for tiny models like this...probably a couple/few years worth. But, I like tinkering, so having an excuse to play with hardware is its own reward. If it happens to pay me back some of that money, that's a bonus.

Of course, as the major providers decide they need to ring the cash register and stop burning money on subsidized tokens, that math may change, and I may find I'm grateful to have already bought this stuff before the RAM prices made everything 2-3x more expensive.

But, I think if you're not interested in learning about the technology and doing your own training experiments and such, you should probably not try to run stuff locally most of the time.

So one of thr things you're using it for is to generate leads to spam businesses with unwanted LLM produced marketing materials it sounds like.

Wow LLMs are changing the world, what a utopia.

> So one of thr things you're using it for is to generate leads to spam businesses with unwanted LLM produced marketing materials it sounds like.

You don't know me. And, no.

Yes, all my emails gyer sorted out by a finetuned gemma. There are turned into images passes to the model, as multimodal is so practical.

I've yet to see someone answer a question like this with a decent, useful answer.

I moreso run other small special purpose models like Whisper, SAM, Matcha, CLIP etc. and then do contextual correction passes with models like this.

Think almost like unix pipelines, have used it for many workflows.

This is one https://post.bot/

What model is it using?

I do not know which model specifically, but I saw the founder answering a question about how it's a small model that's focused on just this one specific requirement.

I expect it to be something like https://huggingface.co/OuteAI/Llama-OuteTTS-1.0-1B-GGUF

No, this is a phone service. They charge $0.25 per minute on the phone on a call that would otherwise not connect.

Can you call a receptionist at 10pm and book an appointment? Or ask for directions? What if it's 10am and she's already on the line with someone else and you just want to ask if there's parking?

Please tell me what 0.25c x 60 is.

Yes, they're called after hours answering services and they're exponentially better because I get to talk to a human.

If my doctors office replaced a receptionist with this I would switch and leave bad reviews across every platform possible.

Ive already switched doctors once because they used an LLM transcription service during my appoitment that influenced the doctors recommendations for care. Sorry technology does not belong everywhere.

AI produces low quality work and will turn your business to shit.

Are you, perhaps, missing that $0.25/minute is only minutes on call? An agent not answering the phone for an hour is $0 (not $15).. for after-hours calls (rare) this is a meagre rate, compared to pay-per-hour (no matter the call volume) answering services.

I've always wanted to make a GUI client for pandoc

AI-generated music should be hard to detect

Harder to detect manually compared to image or video, but not necessarily harder to detect with another model. If it’s AI-generated MIDI (I have no idea if that’s the, or even a, way it’s done) there are probably patterns in the output similar to the way there are in generated text, but if it’s actually generating the audio itself then that should be pretty distinct at the finer-grained level that a model could analyze it at.

A Hacker News mobile client that I'm very proud of, specifically designed for non-native English speakers with automatic translation features.

https://haiker.app

I've transitioned between cloud services and self-hosting a few times:

1. Vercel Phase My first project used Vercel. Since my project was Next.js, the experience was decent. But as my project gained some users, I found that even for projects under 100 users, I needed to pay $20 per month. Since my service didn't require high performance, this cost felt steep.

2. Self-host Phase (Hetzner + Coolify) Later, I started setting up my own server with Hetzner and deploying with Coolify. Since Coolify is open-source and free, I only had to cover the cost of a VPS (even $5 a month was sufficient). I could deploy PostgreSQL instances and run a web server on it. But later I discovered that even this way, I still had to spend a lot of effort maintaining PostgreSQL and Redis. Even though they were containerized with Docker, managing them was still troublesome. I needed to pass various system and environment variables between services, which was very tedious.

3. Cloudflare Phase So later I switched to Cloudflare. With Cloudflare Workers, I can deploy fullstack applications and use D1 Database and Cloudflare KV to replace Redis. These features can be called directly within the Worker without needing to pass environment variables.

Plus, the local development experience is excellent and the pricing is very reasonable, so I've been using Cloudflare's entire suite ever since.

Yeah Cloudflare offering has become everything I wanted from AWS. So much simpler to deploy a basic full stack app + files. AWS has become considerably more difficult than self hosting.

I agree, I'm starting to like cloudflare increasingly aswell

Here are a couple reasons of mine (PS I'm still a little new)

1) V8 isolates for serverless functions to address cold start problems, sure the entire node env ain't there but libraries like Hono are designed to work in that env... Combine that with their near immediate start-up - simple lovely

2) UI, AWS to me feels soulless, like if there's an entire industry to make AWS UI not suck it's obvious their UI is just bad, upto the point where people pay a premium for a good UI. Cloudflare UI is so much nicer, atleast to me

I recently developed a library and for that I made a landing page and documentation with Astro (no server just static stuff), and I was checking out how to deploy this and Vercel and Cloudflare, Vercel had a 100Gb/ month of bandwidth free which is nice, what's even nicer is cloudflare has infinite (practical infinite not the theoretical infinite ofcourse)

And once again, that's just lovely to work with!

I've really enjoyed using CloudFlare and I've been impressed, but I'm afraid it will descend into a broken mess as they enthusiastically use more and more vibe-coding.

I've been using CF Workers since 2020 or so. The biggest con is that your app will be coupled to their infra. It's less coupled than eg Firebase but still.

For the past 10 years or so I've mostly used Heroku end then Fly. Last year I invested time into switching to self hosting with dokku for new projects. After the initial learning curve it's been great. Honestly don't see the point of using anything else except if I need to run something at the edge.

Went through a similar phase,

I think a mix of 2. and 3. is good for a small team or solo dev. Im throwing in a bit of homelab as well by adding some action runners and models on my desktop as well.

But cloudflare is great value for small teams. Not sure how it as at higher scale.

On the topic of env and config. It took me a while to get this write, and maybe overengineered.

But I invested a lot of time in trying to standardize env definitions, secrets manager, and per env config definition defined in my nx projects, and consumed by the commands or deployers. As well as pulumi for IaC.

I tried a couple of different approaches, but finally I just decided to use typescript as my config language. I use nx project.json but defined using typescript. And just define the env config as typescript functions to be injected to each command or deployment as a pure function of target env.

I really wanted to like Cloudflare Workers and I'm sure there's good technical reasons but the way you need to use a Wrangler project to do things like enabling email felt too much like I was about to get locked into the platform.

It seemed like the bindings you needed to set to allow email can't actually be set (or even seen once Wrangler sets them) from the console at all.

> Even though they were containerized with Docker, managing them was still troublesome

Did docker make it easier?

The only issue I have with PostgreSQL is a bit of migration effort moving to new major versions.

> I needed to pass various system and environment variables between services, which was very tedious.

Was docker making this harder?

I switched to Cloudflare and it's been a breath of fresh air - everything I need and the pricing is reasonable.

I had a similar experience recently. I had a huge Swagger JSON file that would waste too many tokens if added directly to the context, so I told the agent in memory to use jq to retrieve what it needs when it wants to check this document. This saved a lot of cost.