Deepseek v4 Pro feels like Claude Opus 4.6 in it's personality but here's what I did find out about costs:
I did cut loose Deepseek v4 on a decent sized Typescript codebase and asked it to only focus on a single endpoint and go in depth on it layer by layer (API, DTOs, service, database models) and form a complete picture of types involved and introduced and ensure no adhoc types are being introduced.
It developed a very brief but very to the point summary of types being introduced and which of them were refunded etc.
Then I asked it to simplify it all.
It obviously went through lots of files in both prompts but total cost? Just $0.09 for the Pro version.
On Claude Opus I think (from past experience before price hikes) these two prompts alone would have burned somewhere between $9 to $13 easily with not much benefit.
Note - I didn't use Open router rather used the Deepseek API directly because Open router itself was being rate limited by Deep seek.
I find a lot of the inefficiency also comes from the model just randomly poking around and grepping all the time which is the fault of the harness. I ended up building a Prolog based MCP where I use tree-sitter to parse the code into a graph, and then the model can just ask questions like 'what are all the functions connected to this function'. So, in case you're trying to focus on what a particular endpoint is doing, you can trivially and predictably trace the whole subgraphs of calls.
I don’t know if it exists already, but bazel would be very useful for the same type of MCP server. Since all dependencies are explicit you can pretty easily do a bazel (r)deps query to find related targets.
Similar idea, I find tree sitter is nice because it already supports a bunch of languages and it's easily extensible. Once you the AST, you can really have the LLM go to town with it.
I've been having the same experience. Tasks like "go through this entire module and pedantically make it match my preferred styleguide exactly" were not worth a couple dollars with frontier models. It's nice to be able to put deepseek flash on stupid, unnecessary or highly speculative tasks without thinking about the cost.
DeepSeek V4 Pro's pricing is blowing me away, particularly with how effective the cache is. I just burned 2M tokens and the total cost was 30¢. On Claude Code, I'd have used up multiple 5 hour windows by now, or else horrific amounts of API consumption, around $20-$30 I'm guessing.
> It obviously went through lots of files in both prompts but total cost? Just $0.09 for the Pro version.
When people say that LLMs aren't worth it, it kills me.
A lot of us, on average, make $100+ an hour. $0.09 is < 4 seconds of our time.
You can't even read the vast majority of prompt responses that fast.
LLMs will continue to get better (I'm doubtful at previous rates, all indications are showing that progress is slowing and costs are increasing disproportionately).
It seems like >50% of devs think LLMs provide less than 0 value. I just do not get it.
Did they use an LLM one time 3 years ago and decide it's never going to be worth it? Have they even tried? Or have you only ever tried it on 1 giant, monolythic proprietary codebase where they're a total expert and decided that an LLM isn't as good as them, so it's "completely worthless"?
They are shockingly unhelpful on my company's codebase.
But that doesn't mean they are flat-out worthless.
I know I'm guilty of making this sort of argument sometimes, but it's just not valid.
I don't get paid for every waking hour of every day. Often I'm using an LLM for something that's uncompensated, so my hourly wage equivalent is irrelevant.
And for times when we might use an LLM for something related to paid work, it's still money out of your paycheck (unless the employer is paying for it; go nuts in that case). And it's not like using the LLM lets you go home early if it saves you time. You just end up doing more work.
I still use them because they're a useful tool sometimes. But I don't pretend it has negligible or no cost. (Not to mention the externalities around electricity use, crazy data center buildout, skyrocketing GPU and RAM prices, etc.)
I don't understand, your employer doesn't pay for your AI use? If my employer didn't pay for it I just wouldn't use it at all out of principle. Just as I don't buy my own work laptop
I'm guessing downvoted because OpenRouter was mentioned in the note (which may not have been there originally), but aside from that this is a perfectly legitimate question. In order to reproduce we need to know how. Was it a coding agent like opencode, an IDE, or something else?
Microsoft just announced the availability of OpenAI GPT-5.5, which they are charging 30x for it. In contrast, they charge 7.5x for Claude Opus 4.6 and 1x for OpenAI GPT-5.4
Check out the token-based pricing, and compare GPT-5.5 with all other models.
When I check GH Copilot right now, it looks like Opus 4.7 multiplier was increased to 15x (I think it was 6x just a few days ago) but 4.6 is still at 3x. But these relatively cheap multipliers exist only until the end of the month.
That’s the classic phenomenon of cheaper pricing due to offshoring! If your expenses are in dollars then for sure recovery is going to be in dollars as well. Why is that a surprise to anyone?
Only similarity it has to Opus 4.6 is the 4 in the name. I do not understand these dishonest comparisons. OOS models are vool, cheap and promising for a future -- but why are we pretending they are better than they are?
Speak for yourself. I found switching from Opus 4.7 to be completely painless and in fact, due to the reliability of Anthropic’s API, less of a friction despite slower response times. Zero issues on a large mono repro
Hi, I am happy it works well for you. For me personally I struggle finding good use-cases in general for these OOS models. I am lightly technical but I do not manually code. So my flow is /grill-me (can take hours), make plan, review plan with 2. model, implement, review after implementation.
Maybe it is because my tasks are usually chunkier, or because I cant code myself that I struggle using cheaper models. Feels like at every stage of this process SOTA model improves it by 5%, which adds up.
But I am maybe ignorant of Opus level. My main driver is 5.5 and Opus is there for frontend and 2. opinion. In a past I also used Claude models for the chatting phase, but 5.5 took over recently. Maybe Deepseek is closer to Opus and I just overestimated the model compared to 5.5? I tried to give it benefit of being similar.
Recently I started experimenting with Deepseek Flash, maybe hoping if plan is solid enough it can implement quickly and cheaply, but for now it feels not worth it.
How do you use the model to see the benefits? Have you tried 5.5 and can you compare to that one as well?
In my experience, deep seek models are massively overrated in terms of how good they actually are at agantic usage, coding and writing, just because they are kind of the first open source entrant and the name a lot of people know. Try GLM 5.1, coding and writing just because they are kind of the first open source entrant and the name a lot of people know. Try GLM 5.1.
What provider are you using? I have it a shot through open router and saw some weird half formed words coming through occasionally, would love to switch over and give it a proper go
I have a gut feeling that these models can do just as well, has someone run a reasonable size task — >=1-2 days of designing and planning — and see it work well with these models?
* For me what worked well was the grill me skill(or its variation) at the design stage, the hygiene I followed here was have it ask one question at a time, resolving dependencies at the design stage and reading the hashed out plan closely. The use of a couple of other MCP tools like a documentation server like deepwiki and arxiv for grounding. Other tricks I use are having high signal tests and having claude either be able to read logs and code at the same time or embedding it in the execution(e.g. as a debugger, repl or devtools)
No duplicate the whole task e.g. I use grill-me skill for planning and it takes me ~3 hours and CC asks me 20-40 questions. Do the same grill-me with this and compare the outcomes. I admit Its quite a lot of work to duplicate, but i am really itching to do this over a few tasks and compare the final plan. Just need the time.
While the cost are lower than frontier models there are two factors that make DS4 Pro and K2.6 not as cheap as they might look.
For DS4 Pro there's a discount going on for the official API, which sometimes gets overlooked and mixed up in discussions. Simon uses the full price in the comparison, so that's not an issue here.
The other issue is that DS4 Pro and K2.6 often use way more reasoning tokens than the frontier models. In my testing there are certain pathological cases where a request can cost the same as with a frontier model because they use so much more tokens.
To be fair I'm using DS and kimi via 3rd party providers, so they might have issues with their setups.
But if you look at the Artificial Analysis pages of the models you'll see that DSv4 Pro uses 190M tokens and K2.6 170M tokens for their intelligence benchmark, while GPT 5.5 (high) only used 45M.[0][1][2]
I recommend looking at the "Intelligence vs. Cost to Run Artificial Analysis Intelligence Index" ("Intelligence vs Cost" in the UI). The open source models are still cheaper to run, but not by as much as you'd think just looking at the token prices.
They introduce very novel methods to improve long context efficiency and attention. HCA & mCH. It requires only 27% of flops for inference and 10% for KV cache than v3.2. This makes it super efficient. Think of this. For flops, we can now serve more than 3x the amount with the same number of compute, and you would need 30% of prior KV cache.
Furthermore, this release is a PREVIEW, DeepSeek is the real open labs and they not only cook up quite a bit with every single release, but they publish and share it. I'm running this locally.
Let me tell you how "CHEAP" this is. With v3.2 I would run out of GPU ram, spill into system ram with 256k context. It ran quite alright and I was happy with my 7tk/sec. With this, I'm 100% in GPU ram with full 1million token, run more than 2x fast while getting better results.
This is super cheap. moonshot has made it clear that they are starved for GPUs and that's why. If they had GPU capacity like we do in US and subsidized the models like we do here, they would be giving it away for free!
Sure that can happen but it hasn’t been my experience. I just spent a whole day using it for some pretty hefty refactors, many rounds of back-and-forths, thousands of lines of code changes, reviews, investigations, many subagents running parallel tasks, the works. Total cost $0.95, altogether.
I had attempted this with Opus 4.6 in the past and it burned through the $10 budget I’d given it before it returned from my initial prompt.
Even if it’s heavily discounted, it would still have cost me single digits for a complete solution vs double-digits for exactly nothing.
I didn't want to say that they're not cheaper to run, artificial analysis also shows that they're cheaper. My main point was about it being important to also look at token efficiency, not only cost per token, to get the full picture.
I agree! I don't find Claude models to be particularly efficient anyway though. Maybe when running through Claude Code? I don't know, I tried it a while back but it didn't suit me and I kept hitting bugs so I dropped it in favour of something that does something closer to what I want rather than what the provider wants!
Mostly OpenCode but I've been experimenting with Pi a bit lately.
I use Agent Hive [0] for more complex tasks. It sends off subagents with models and parameters I can configure for each different agent (i.e. a low-temp coder, a higher temp with some top_k / top_p for research and architecture, etc).
I'm surprised that people here don't care at all about these models openly training on your data, especially if you use them straight from the model developer. Whereas things like "GitHub now automatically opts everyone into using their code for model training" get hundreds of justifiably angry comments, I never see this brought up anymore on posts like these talking about using Chinese models through OpenRouter. This might be explained by "well they're different people", but the difference is very stark for that to be the whole explanation.
At least that’s what they’re telling you. It’s a ”trust me bro” scenario.
I’d rather use the phone home version (deepseeks own endpoint). The benefit is that I’m fairly certain that they actually host the model I’m paying for.
If you're not Chinese, and you start a company outside of China, and your whole pitch is "We run open weights and we have nothing to do with China", 1) why would send data to China?? 2) why would you risk your business to do a thing that makes no sense?
A fly by night operation created primarily for the purpose of collecting training data and corporate espionage will make whatever claims they think will get them the right traffic.
Well, the context was running the models via open router, not hosting 800B> models yourself. Of course, if given the option I believe most people would pick ”don’t share sensitive data”.
What I’m trying to say is that EVERYONE uses your data, even the sensitive type. So you might aswell use an endpoint that does what it says and treat EVERY endpoint whether that’s OpenAI or anthropic as if it’s collecting all of your data.
Some providers are based in the US or EU and would face legal repercussions for lying about what they do with your data. It's a bit more than "trust me bro". Off the top of my head, you can use Fireworks, for example, which is based in California and would face the same consequences for lying about their data policy as OpenAI or Anthropic would.
What, because they broke the law in one way, they'd break the law in every way? That's not how business works. The way business works is, I steal from other people to make a product, but then I don't steal from my customers, because if they find out, then I no longer have any customers. (Plus all their customers would sue them, which would both legally and financially tank them)
You definitely have a bone to pick. Chinese researchers usually have given the world the most cheap and consistent high quality research around LLMs. They don't pretend, they do the work and release the goodies. Mostly so cheap, every one in the world has a chance to use close to frontier models. Why would you respond with "Anger"?
You let us know what your real complaint is about and let's not feign indignation at open models and research.
I made no such claims. Maybe you have something to share about why we need to have a negative view of free and open models based on publicly available frontier research.
Anthropic and OpenAI took your data, trained their model, and tell you "we are not going to tell you anything how we trained our models, we are not giving your the weights our models, you will have to pay us to access the model trained from your data".
they took your rights and your data.
Chinese labs took your data, trained their model, and tell you "this paper details how our models are trained using your data, here is the final weights of our model trained from your data, feel free to use it for what you want, it is your model trained on your data".
they converted your data, everything is still in your hand under your control.
you couldn't see the difference?
Your specific question can actually be translated as -
1. why people don't stop Chinese labs so US monopoly can be maintained?
2. why people don't stop Chinese labs providing free models to those who would otherwise never be able to afford the same $200 USD/month Anthropic and OpenAI subscriptions.
3. why people don't complain Chinese labs publishing those trillion dollar secret ideas on model training.
well, because most people are not dickhead I guess?
Hold up. Look, this is all shades of grey but saying Chinese labs all release open weights stuff is kinda crazy thing to say.
Right now they are doing that because they are still trying to catch up to Anthropic, Google, and OpenAI.
The moment they have the special sauce, they will shut it down and you won't be able to run their stuff anymore outside of them. Why do I say that? We already have the evidence in the diffusion model arena. All the chinese labs were pumping out open weights models for image and video, the moment they got to SOTA, they stopped doing it. Less and less is being released.
Chinese companies aren't doing open weights models out of the goodness of their hearts, they are doing it because it help their entire industry catch up. Don't get it twisted, this is very much a US vs China battle here. China wants to win and I am not sure how they won't. Deepseek is the first major large model trained on Huawei chips. It won't be the last and I am betting that China will make up for lesser performance of those chips with more manufacturing and power generation.
I am very bullish on China winning the AI war here. But I also am not naive enough to think that the Chinese companies is doing open weights out of wanting to make the world a better place or the goodness of their hearts. It undercuts the american AI companies.
I am personally okay helping them as long as they publish the models and dont keep them closed. And I dont trust the settings where providers say they wont train on it.
Because they give it away for free and offer APIs at very acceptable rates. Not that hard to figure out, Robin Hood stealing our data tax back comes to mind.
User publishes to github => Copilot trains with GitHub data => MS Sells copilot => User workes for Microsoft (in the sense of giving it's labour for MS to make money)
User publishes to github => Deepseek trains with GitHub data => Deepseek gives model away for free => User did not work for Deepseek (in the sense of giving it's labour for Deepseek to make money)
It's totally fair to use GPL code, it just means all the models built by Anthropic, OpenAI, etc. using GPL-licensed source are themselves bound by the GPL. Plus, any works created downstream using those AI tools.
We're on the verge of a golden age of software as soon as someone finds a court with courage.
I think AI will create an open source dark age. Gradually, we'll see a lot less new good open source code. A gradual shift back to the proprietary world. Simmilar to the 1950-1990 period.
Things being public should not be enough. just because someone leaked your medical information to the public via a data breach should not make it fair game. There should be some rules.
My policy is that I don't allow agents to access all code. Some of it is shielded behind bind mounts. Maybe this is a pathetic, artisanal (or ego-driven), reaction of mine to the inevitable. I allow them to work on about 90% of the code (most codebases fully), with some code being considered too valuable to expose to the vendor. When data is involved, LLMs only get to see anonymized data.
This cute policy of mine won't affect anything though. The more we use the models, the more the models will replace this kind of work. Centralisation of power is inevitable; in Medival Europe, we used to have state & church ruling. In modern times but before the internet, it was probably state and banks. Maybe with ongoing digitization (bank offices disappearing) making banks less costly to operate; combined with with bank bailouts, maybe govenments will fully nationalize or at least banks will consolidate.
Then the AI companies will consolidate with the internet information and communication companies (Google/Meta for the US, and Alibaba/Tencent for China). Maybe we'll end up with a few de-facto governmental megacorps that rule in tandem and close cooperation with the formal government, who might handle mostly infra, utilities and the army. The megacorp would control narrative more and take more of a paternal role (educating and protecting the citizens, normally handled by formal governments).
AWS Bedrock has DeepSeek models running on their infrastructure. That should be enough to prevent training on user data (there's a markup compared to DeepSeek's pricing though).
And unfortunately AWS doesn't have prepaid billing, so you can't just give the internet access to your API key without getting FinDDoS'd.
I am fine with them training on my open source code (which is pretty bad but not the point, because they're providing the service for free). I will be super pissed if I pay for enterprise and they train on it though. I believe this is the opinion of majority programmers.
What do you mean specifically? Data passed through OpenRouter? Or that they too indiscriminately ingest data all over the web? If the former, I assume it's just that anyone still using them just doesn't care where the data comes from. If the latter, well, it seems like every day there's some news on some new model from somewhere, and it takes dedication to complain every time. There's also the factor that I believe DeepSeek is more open with the model, while others keep it entirely proprietary, which feels fairer and (personally) is also less offensive.
Do you really think OpenAI, Anthropic or any other entity in the same business respects your data?
The Chinese AI companies who release open weights actually deserve whatever input you give them. They are the reason why there is competition and not duopolies in the domain.
I think Google, and likely Anthropic, indeed do honor the settings chosen by the user. For Google in particular it'd be very surprising if they didn't. That's also why both do everything they can to trick users into allowing it.
OpenAI, I wouldn't be surprised if you were right.
You mean the same Anthropic, that wouldn't blink an eye at intentionally overcharging users hundreds of dollars just for having a HERMES.md file in a repo, would be above taking your data for... ethical reasons?
unfortunately the history of these big tech companies has shown that they do not care about data privacy and are even willing to lie about it. but I guess its irrelevant, in practice you have to assume the worst anyway since there is no way to verify it
Two factors. First is anti-americanism (or at least anti-american-capitalism).
But the more important one is the social contract. Github came far before LLM era. The branding around it is being the storage of open source projects and many users want to it stay away from AI hype. You won't expect LLM providers to stay away from AI hype (duh) so it's less an issue for them.
From the EU side. I think we'll make a cost comparison between the US ( where it's leaders are doing weird shit against the EU and pro Russia) vs China ( who at least gives cheap models and doesn't actually tries to take over an entire European country).
US has too much influence atm. I'm ok with switching between "bullies".
The biggest differentiator for me: DeepSeek just does what I ask. I've tried using both GPT and Claude for reverse engineering recently, both refused. I even got a warning on my OpenAI account.
Well, I'm using all the top models extensively on the very same codebase, my new compiler. I use deepseek for it's cheap API costs, when kimi, claude and codex are in their overbudget phase. I asked deepseek V4 Pro for an estimate of a new arm64 port. It said 4 weeks, I said, ok, do it. (I knew ncc was there, and tinycc was also known to the AI's). So it took it half an hour to produce a working arm64 port. First for arm64-elf, because this was easiest to test, and then also after more hours of back and forth the arm64-darwin port. (with crossbuild and github actions). It did cost me with all the subsequent fixes around $8 API costs.
So the experience: at the beginning deepseek was amazing. When it started to get expensive (china day time), I switched from Pro to Flash. No problem, same results. Some bitfield implementation was too complicated so I had to wait for Sonnet 4.6 tokens, kimi-2.6 did the rest. For the very hard problems I asked gpt-5.5, but this was only for one problem. minmax was horrible. didnt follow rules, and made lot of silly stuff.
But when the deepseek context window got filled, deepseek also started to become stupid. So either /clear, or /export and strip the file. And start a new session with the cleared sessions. kimi was overall better, but running into limits with my cheap moderate subscription. Paying private for it, as my companies' token budget is usually out after a week of work.
All in all it is worth it. My next compilers (perl 5+6=11) will be done with deepseek and kimi also.
regarding decompilation: recently we had to decompile a firmware for a USV we bought, but doesnt work on a new system. It only worked on a raspi. So I decompiled it with ghidra, and told my colleague, easy, that's how you do it. But my colleage didnt know about token budgets yet, and already threw opus at it. CoPilot Business account. He had working C files immediately, compilable for our new system. It ended up the USV was not beefy enough. But Opus was fantastic. The code was very short and simple C though.
it also sounds like a lot to manage, do you have some sort of agentic framework that's treating all of these llm's you have access to as sort of inputs that it optimizes?
Unfortunately not. I'm using plain kimi, opencode (with deepseek, gpt, minmax, whatever) and claude. claude is the best, but only for some hours. The trick is to get a good AGENTS.md file, good test cases and test runner to repro, like seemless docker and qemu calls. GNU autotools would be easiest, but here I'm using plain makefiles.
Also for LSP clangd being up-to-date a compile_commands.json is important.
git worktrees helped developing the arm port and fixing c-testsuite cases in parallel. I wanted to keep the costs down. About $15-$30 I think.
And for low-level problems, like ARM calling-convention in asm, those models are much better than simple algorithmic python problems. Just for the hardest problem I needed the big expensive gun, but never opus. This helps in deciding what to do with my next jit project.
Not op but I wrote llm-consortium to prompt multiple models and create a synthesis. And it can run on an openai endpoint using llm-model-gateway. It's expensive, naturally, but for situations where you absolutely must get max intelligence its hard to beat.
e.g.
Pelican Riding a Bicycle — Engineering Study by DeepSeek v4 Pro, Kimi K2.6, and GLM-5.1 (1 iteration in synthesis mode with DeepSeek v4 flash as judge)
I was using GPT 5.5 through Cursor recently, and it found what it thought to be a security-related issue. I read the code, didn't see what it was seeing, and said "Run the chain of operations against my local server and provide proof of the exploit."
It thought for a few seconds, then I got a message in the chat window UI saying OpenAI flagged the request as unsafe, and suggested I use a "safer prompt."
Definitely soured me on the model. Whatever guardrails they are putting are too hamfisted and stupid.
Personally, I'm not bothered very much by LLM confabulation, as long as it's the result of missing context. In most practical tasks, we either give context to the model, or tell it to find it itself using the internet. What I am concerned with is confabulation that contradicts available in-context information, but that doesn't seem to be what is measured here.
This must be easily benchmaxed because I have never gotten an "idk like" answer for the western frontier models. All my personal "real world" use cases will always resort to hallucinations.
The output of any LLM is always 100% hallucination by principle. On top of that, most benchmarks are at best an approximation of LLM quality. Your use case decides which one to use. That said, I haven't tested v4 yet but the old 3.2 is still a decent model. And concerning use cases, I had coding problems that Opus couldn't solve but a local 35B model did.
All the talk about frontier and SOTA is do dig deeper and deeper into the pockets of VCs and finally do an IPO.
We have an enterprise cursor account so I can try all the mainstream models. Using composer 2 on our own code which I obviously have the source code for I couldn't get it to turn on a debug flag to bypass license checks while I was troubleshooting something. Infuriating. It was like that old Patrick from SpongeBob meme.
I don't understand why we would turn the models into law enforcement officers. Things that are illegal are still illegal and we have professionals to deal with crimes. I don't need Google to be the arbiter of truth and justice. It's already bad enough trying to get accountability from law enforcement and they work for us.
They're probably worried about liability. Let's say that Oracle finds out you reverse engineered their DB using Gemini. You can be sure they will sue Google. Not just for providing the tools, but you could make the argument that it's actually Gemini doing the reverse engineering, and on Google's hardware no less.
The difference is IDA Pro doesn’t do something unless you instruct it to, an LLM is unpredictable and may end up performing an action you did not intend. I see it often, it presents me options and does wait for my response, just starts doing what it thinks I want.
This. It's going to be tricky for the frontier model labs to argue they didn't intentionally design their models to do so, when the models take illegal actions.
I'm not even sure how one would construct a viable legal argument around that for SOTA models + harnesses, given the amount of creative choices that go into building them.
It'd be something like "Yes, we spent billions of dollars and thousands of person-hours creating these things, but none of that creative effort was responsible for or influenced this particular illegal choice the model made."
And they're caught between a rock and a hard place, because if they cripple initiative, they kill their agentic utility.
Ultimately, this will take a DMCA Section 512-like safe harbor law to definitively clear up: making it clear that outcomes from LLMs are the responsibility of their prompting users, even if the LLM produces unintended actions.
> I'm not even sure how one would construct a viable legal argument around that for SOTA models + harnesses, given the amount of creative choices that go into building them.
I'm not a lawyer, but to me the legal case seems pretty obvious. "We spent billions of dollars creating this thing to be a good programmer, but we did not intend for it to reverse engineer Oracle's database. No creative effort was spent making it good at reverse engineering Oracle's database. The model reverse-engineered Oracle's database because the user directed it to do so."
If merely fine-tuning an LLM to be good at reverse engineering is enough to be found liable when a user does something illegal, what does that mean for torrent clients?
Which is going to be hard to explain to a judge and jury, if it comes to that, how despite investing time, money, and effort (and no doubt test cases) into making a model better at reverse engineering... they shouldn't be liable when that model is used for reverse engineering.
Afaik, liability typically turns on intentional development of a product capability.
And there's no way in hell I'd take a bet against the frontier labs having reverse engineering training data, validation / test cases, and internal communications specifically talking about reverse engineering.
> “making it clear that outcomes from LLMs are the responsibility of their prompting users, even if the LLM produces unintended actions”
So if I ask “how does a real world production quality database implement indexes?” And it says “I disassembled Oracle and it does XYZ” then I am liable and owe Oracle a zillion dollars?
Whereas if I caveat “you may look at the PostgreSQL or SQLite or other free database engine source code, or industry studies, academic papers; you may not disassemble anything or touch any commercial software” - if it does, I’m still liable?
Who would dare use an LLM for anything in those circumstances?
We need that lawsuit to happen already so we can establish precedent. The person in the driver's seat of the Tesla should be at fault. The engineer using the llm should be at fault. The person behind the gun not the manufacturer should be at fault.
> The person in the driver's seat of the Tesla should be at fault.
I don't think this is a good analogy. For Tesla right now it might fly. However, when their software gets to waymo level of autonomy, I would expect liability to shift to the manufacturer.
If anything, I think that would be the true proof of a company trusting their software to allow for autonomous driving
In the America, whoever has the most money is liable. It's not worth it for the legal industry otherwise. The lawyer earns his pay by convincing the court that whatever established precedent doesn't apply to his case.
> Things that are illegal are still illegal and we have professionals to deal with crimes.
This is quite naive take though. The direction of travel is more fascism in Western governments where duties of traditional policing are taken over by big corporations whilst police forces are being gutted and made impotent.
> I don't understand why we would turn the models into law enforcement officers
It's a simple corporate risk minimization strategy. Just look at how universally despised Grok is on HN. Not because it's a bad model, but because it has less aggressive alignment which means it can be coaxed into saying things that get Xai pilloried here and elsewhere.
Grok was worse than even some of the more mediocre open models at actually doing anything. (At least anything tech work related.) GPT and Claude just do what I ask most of the time. With grok, it’s like a chore just getting it to understand the question.
You’re pulling your hair out trying to figure out what on earth you need to do to land in the right place in whatever topsy turvy embedding grok is using?
I also used to see Grok boosting/slack-cutting on here/Reddit constantly back in Peak Subsidy when xAI was giving out hundreds of dollars of credits for free per month.
After they killed that and then stopped handing out free model access to users of every Cline fork for weeks following model releases, vibe coder hype moved back to Chinese models for cost and the SOTA models for quality.
Agreed. There's are plenty of instances where people here on HN do mental gymnastics to justify using a truly good product when the company that builds it is morally bankrupt.
Not a criticism (I probably engage in that sort of thinking myself sometimes), just something I've observed. If Grok were actually good, we'd see that phenomenon here, but we don't.
No, they've clearly put a lot of work into alignment. It's just that they've been trying to align it with Elon Musk rather than Amanda Askell. Unfortunately the more anti-woke they try to make it, the worse it seems to perform.
> Unfortunately the more anti-woke they try to make it, the worse it seems to perform.
Probably because being anti-woke generally goes hand in hand with going against facts and logic. Cull the "woke", lose the facts+logic. Not that they care about that anyway.
Software engineering is one thing but if you look 10-20 years into the future and everyone can run models equivalent to today's SoTA locally with zero monitoring or censorship, that could... not be good.
Some people will use them responsibly but a lot of people will not.
LLMs are already frying some people's brains and there are some human desires that should not be encouraged
This is kind of terrifying to me, regularly. No real manner of recourse to normal people without a following, potential exclusion from real fundamental tooling. Imagine OpenAI goes on to buy 20 companies and now you cant use Figma, Next, whatever just because you once tripped some very foggy line somehow. Not just OpenAI but the entire ecosystem is so... hard to read.
I was asking Gemini about a quote from catch 22 and it kept dying mid stream saying it cant talk about it, god knows why, it had no violent or sexual content -- though that is in the book. I could imagine it dinging my whole workspace account just because ... shrug?...
I know ideally the future is local, but I don't know how real that is for most people at least in the next few years with practical costs and power usage except I guess through a M* processor if you're in that ecosystem.
Open models running locally is the answer. Relying on proprietary, closed software always puts that company's priorities above your own when using their software. You have given up control.
While running them locally presently doesn't make sense economically, you don't need to run them locally to address this issue. There is a lot of competition in hosting open models and you have a variety of services to choose from. Run the open models now, reward that ecosystem instead of continuing to reward closed systems that dreams of rent-seeking.
You don't need to run the model locally if you don't care about sharing your data. Personally I am happy to share data with Kimi or Deepseek if it means we get better OSS models. For private stuff though local is king
It'll be a while yet before open models that're good enough will be viable for local use. Heck I've been trying to use the Qwen 3.5 39B A3B on my system, which is modest but no slouch, and have only been able to get ~4.5 tok/s after optimization, and it really runs my system red (fans instantly go crazy). It's just not practical for serious work.
I've been using Qwen 3.5 and then 3.6 27b Q4 on Ollama with a single 7900 XTX with the codex cli, and I have been blown away by how genuinely useful it is. I've been able to ask it to do long, multi step problems, and it's able to do things that would have likely taken me days to iron out in a matter of hours, or even minutes sometimes.
I get about 30 tok/s, which is far from blazing, but given the capability it has it is absolutely viable for accelerating my work.
Yep, and with ID verification, it's not like you can just make another account either. At least, I'm guessing if they don't already, they'll soon be blacklisting individuals, not accounts.
Imagine your livelihood depending on access to LLMs and then OpenAI ban you with no recourse. This is where AI legislation should be focusing right now IMO. We can ensure a level of fairness for everyone without putting the brakes on.
It's probably because you were talking about a quote from a book (ie copyrighted material). Authors have sued the AI companies for repeating / memorizing copyrighted works, and getting an AI to discuss a quote would be making it repeat a portion of copyrighted work.
Funny that your case is Kurt Vonnegut. I think I had Claude refuse a task where I was doing an OCR scan of a book review (in a zine / journal a family member published years ago). I think the review might have included a Vonnegut quote as well, and that I ultimately figured it out it was the quote that was making Claude refuse. I may be misremembering the author though.
Mistral had no such refusals, but their OCR is lesser quality.
OMG. Where did I get Kurt Vonnegut from? I swear I saw that name in the post and the whole time I was thinking "but he didn't write Catch 22"... I must be fuzzier brained than I thought tonight. Thank you for being kind with your correction.
Hopefully I'm still correct that quoting from books is a reason for some over-zealous task refusals, though.
> Authors have sued the AI companies for repeating / memorizing copyrighted works, and getting an AI to discuss a quote would be making it repeat a portion of copyrighted work.
>Imagine OpenAI goes on to buy 20 companies and now you cant use Figma, Next, whatever just because you once tripped some very foggy line somehow.
Don't worry, you can just make your own Figma, Next, whatever if you have some thousand dollars worth of tokens. This is at least what all of the AI thought leaders have been telling me for the past couple of years.
I think it’s so bizarre that chatgpt regularly gives me advice on how to get around it’s filters. Like, literally “I can’t do anything if you use copyrighted character’s name, but how about you just say ‘someone that looks like character’”. If you are going to do that, can you just execute the instruction?
In my experience GLM 5.1 has been excellent when paired with IDA Pro (DeepSeek v4 pro comes in close second, Kimi straight up refuses). Claude can only do reverse engineering if you throw it into some sort of hero/saviour mode then gradually pivot into red team (though it gets easily tripped).
Among the inexpensive models (and I include Grok 4.3 in this list), GLM 5.1 really sticks out!
On my personal test bench, when compared to other inexpensive models, GLM 5.1 provides the answers that I would consider most complete or satisfying (these are subjects that I consider myself an expert in). The answers tend to be more comprehensive, nuanced, and include references that I would consider the correct ones (if given access to web search).
I also find it a joy to code with, somewhere between Sonnet 4.6 and Opus 4.6 (have not tested Opus 4.7 yet).
This is so strange. I do a ton of RE with Claude, Codex, and sometimes Deepseek, GLM, and Kimi. I don’t have difficulty getting any of them to use IDA or otherwise decompile things.
There is one important difference, which is that Claude and Codex will both refuse if I ask them to touch anything related to security. But so long as I’m just studying algorithms and things like that, they’re totally fine with it.
That said, Codex especially will sometimes randomly give me a cybersecurity warning and stop responding. It’s random but happens maybe 2-3 times per day if I’m doing heavy reverse engineering work. Claude is much less fussy unless, once again, you’re explicitly trying to touch anything related to licenses, passwords, etc.
Yes, GLM 5.1 is surprisingly good! Particularly for long-horizon Agentic tasks, with 100+ available tools. It really shocked me in a good way when it was able to complete a long run with 50+ steps and not fall into a loop along the way.
I've been using GPT-5.4, and more recently 5.5, with Codex CLI + Ghidra MCP for reverse engineering a game without many issues. Injecting code is where it usually balks at, but I'm just trying to discover and parse structures from game memory.
I did get a refusal when trying to read in-game currency, even though modifying it would do nothing. It has some strange boundaries.
This idea of software threatening the user with consequences is totally wild and dystopian. Fellow developers, what kind of world have be built? This is insanity. Imagine if my hammer told me, "Hey, you shouldn't use me on screws--only nails. Do it again and I'll self-destruct!" WTF people, stop making this kind of software!
> All sorts of tools try to prevent dangerous/destructive uses
But they don't threaten their users or have an "N strikes and you're out" policy. I take those safety caps off of all the chemicals in my garage because I'm a grown-ass adult and those caps are a pain in the butt. I would not expect the manufacturer of a solvent to show up at my house lecturing me about safety and threatening to ban me from buying his products.
Sure but they would if they could. If they knew idiots were doing idiot things with their products (or evils doing evil things) and did not utilize available methods to prevent them, then the company ends up holding liability. And no, this is not easily signed away in a contract.
Uhh right, but describing that as "dystopian" is frankly hysterical.
It's an obvious corollary of good things (like product liability). Virtually everyone I've heard complain about these safety rails was up to antisocial (at best) stuff. I've never heard a sympathetic use-case. It's objectively good that companies can be held responsible for misuse of their products and that they are therefore incentivized to mitigate misuse.
"My inability continuously attack product guardrails to enable my super esoteric (and probably antisocial) use-case is dystopian" is just... not a compelling argument.
"These safety rails" was referring to LLMs, which have far more nuanced and capable safety rails than chemical caps do, and accordingly also have much more assertive ways to enforce them.
It's the same underlying principle. If I want to ask a software tool what the suicide rate is for my county, I do not expect it to come back with: "Naughty boy! You said an unsafe word! You're getting a strike, and if you get two more, you're banned." This is totally out of the ordinary for a software product, and is absolutely a modern invention. Replace "suicide" with whatever the "AI Safety" obsession word is today.
> If I want to ask a software tool what the suicide rate is for my county, I do not expect it to come back with: "Naughty boy! You said an unsafe word! You're getting a strike, and if you get two more, you're banned."
Did this happen?
I just tested this query in Grok, Gemini, Claude, and ChatGPT and 0% of them admonished me or refused to return an answer.
Just like every single conversation I've ever had on this topic, you have to make up examples that aren't even true. Why don't you just share what you were doing that you feel you were unfairly prevented from?
Which would be more than 0% concerning if I've ever heard (even once) an example of this happening with a query that shouldn't actually trigger something like that, or is so close to such a query, that the false positive is understandable and of incredibly niche value anyway.
OP gave an example of reverse engineering, something that to the LLM looks identical to just hacking. I am totally fine if the incredibly tiny little fraction of people who want to reverse engineer their own systems can't use LLMs to do it, and in exchange top LLMs aren't helpful for the hordes of actual malicious actors who would love a superintelligence to aid their crimes.
No-brainer tradeoff, just like 100% of examples I've ever heard.
I don't think that "dystopian" necessarily goes far enough, this would be one of the rare times where I would call it a fascist mentality - the idea that everything's primary allegiance is to the state and the goals of the state rather than those of the customer or the user.
I want a default that has people empowered, rather than something where it's just another performative smokescreen caused by overzealous product liability. I'll thank you and your kind for needing to distractedly tap the "Agree" button on my car's infotainment every time I start it to confirm that I will pay attention to the road.
"the state" is just shorthand we use for "other people in my community"
> I'll thank you and your kind for needing to distractedly tap the "Agree" button on my car's infotainment every time I start it to confirm that I will pay attention to the road.
Does that actually mitigate antisocial usecases? No? Then it's not what I'm talking about :)
Of course if you wanted to you could just share specifically what totally-reasonable LLM use-case you have in mind that's neutered by this "fascist mentality" instead of dreaming up unrelated instances.
> "the state" is just shorthand we use for "other people in my community"
It's a very different abstraction layer, in the same way as individual cells vs the entity that is you. The entity that comes together from all those "other people in my community" and its priorities are different to the individual desires.
> Does that actually mitigate antisocial usecases? No? Then it's not what I'm talking about :)
Maybe it does? Maybe someone is alive on the road today because they read the message and changed their behaviour. I'm giving an example of something where this liability mindset has created a world where manufacturers are no longer prioritising the desires of their users in order to appease a sense of harm-reduction. And you weren't limiting it to LLMs you were applying it to all sorts of tools.
I think that "reverse engineering" as the OP was talking about is one of those things where maybe 1/10000 uses could actually be harmful. This is not even a high-risk request such as to produce a weapon of some kind where maybe your "antisocial usecases" could be applied.
I think it's closer to asking a remote (human) assistant to do something that someone doesn't want done (e.g., view the source of a closed-source product, whether through reverse engineering, going into their office, or social engineering) and that remote assistant company saying, "Please stop asking our assistants to do that."
You can still use an IDE (hammer) to reverse engineer anything you want.
It's not though. It's still just a piece of code, much closer to IDEs or any other program than to a human assistant in any way that matters (morals, responsibility).
It just seems like you are saying if you found out Claude code was a bunch of remote working doing work for you, then it would be morally wrong to do illegal/morally wrong/irresponsible things with them, but because it is NOT a human, those same things are fine?
Is the distinction between human labor/actions and a program executing hard to grasp?
Moral is a human thing, not an absolute thing, so of course it's different if there is a single human involved and a tool, and a human with a relationship to other humans.
This is huge for me too, I was working on something super benign the other day and GPT flagged it for Cyber risk, Deepseek just does the work, its fast and cheap. Its only missing image support IMO, once deepseek cracks image too its going to be hard for anthropic and openai to compete.
Speaking of this: is anyone working on binary to source decompiler models? Seems like a no brainer and I could see it working exceptionally well especially if they were fine tuned for each language. So if you can tell it’s a Go binary use a Go model, etc.
Trivially easy to train if it doesn’t exist already. Take a codebase, compile it to binary, train a model to reverse the process since you have the ground truth.
I myself got refusals often for legitimate data analysis work. I am starting to lean on buying powerful hardware little by little until I get suitable rig to run local models that make sense.
It wouldn't surprise me the US government is behind it. As it wouldn't surprise me the government of China is subsidizing those OS models. A lot of things at play, and all over a huge bubble.
Eventually, access to Chinese models may be illegal in the US. I tell every developer I work with, download them as fast as possible. You never know when this administration could cut off access.
The main difference here is not that DeepSeek's model is completely free of censorship (although I'd wager it's less censored), but that it's open-weight. That has two major advantages:
1) If Anthropic/OpenAI/Google bans you - you're screwed, you can't access their model at all, but if DeepSeek bans - you just go to another provider, or host the model yourself.
2) If the model refuses to answer you can uncensor it (and this is getting easier and more automated day-by-day[1]).
"The photograph you're referring to is the iconic "Tank Man" image, taken during the Tiananmen Square protests in Beijing, China, on June 5, 1989.
The photo, captured by Associated Press photographer Jeff Widener, shows an unidentified protester standing defiantly in front of a column of Chinese Type 59 tanks as they moved through Chang'an Avenue near Tiananmen Square, in the aftermath of the Chinese government's violent crackdown on the pro-democracy demonstrations.
The lone man, dressed in a white shirt and carrying what appears to be a shopping bag, repeatedly blocked the lead tank's path — even as the tank swerved to avoid him. The image became one of the most powerful and enduring symbols of peaceful resistance against oppression in modern history. The identity of the "Tank Man" remains officially unknown to this day."
The photo depicts "Tank Man" which was taken on June 5, 1989 during the Tiananmen Square protests. v4-pro and v4-flash roughly answer the same way on openrouter.
Are you really concerned about asking these kinds of questions though? Like how many LLM-able Tiananmen Square questions are you needing answered per month really? And it seems like you know not to trust it, so there's not even a risk that you're going to ask such a question and rely on the answer.
I run into Claude being a stubborn idiot about far more useful stuff all the time. And often all it takes to bypass is starting a new chat and reframing it, so it's entirely pointless hand wringing.
Then let's not forget only one of these is a paid product, and it's not the more annoying one. I feel like I can forgive DeepSeek for just obeying the laws of the country they're based in, as silly as those might be, because they're being pretty generous with the weights in the first place.
I've been using v4 pro for the past few days and honestly in terms of quality it seems more or less on par with open AIs 5.4 or opus 4.6 (i havent tried 4.7)
To be clear, i'm not doing state of the art stuff. I mostly used it for frontend development since i'm not great at that and just need a decent looking prototype.
But for my purposes it's a perfectly good model, and the price is decent.
I can't wait for open model small enough for me to run locally come out though. I hate having to rely on someone elses machines (and getting all my data exfiltrated that way)
You can use Tinfoil for inference, which lets you use the model in the cloud while getting similar privacy as running locally: https://tinfoil.sh/inference.
Disclaimer I'm the cofounder. This works by running the model inside a secure enclave (using NVIDIA confidential computing) and verifying the open source code running inside the enclave matches the runtime attestation. The docs walk you through the verification process: https://docs.tinfoil.sh/verification/verification-in-tinfoil
Worth noting that NVIDIA confidential computing and similar schemes have been compromised and shouldn't be relied upon if it really matters. See https://tee.fail/ and similar.
I was interested in trusted execution environments and how safe they were. If you look on google scholar and start reading, they seem super vulnerable. The feeling is that the industry has no better option and that they are a way to tell customers they are safe when they're not
Hi there I use your service. It's great. But I have a few requests... Please support crypto payments...? Also you are missing some open source models (qwen 30b 3a, Deepseek 4 flash).
Unfortunately we don’t support crypto payments at this time as we use Stripe.
We try to add models selectively as we have to be mindful about our compute allocation. Is there a specific reason why you need those two models (and our models such as Kimi K2.6, GLM 5.1, Deepseek V4 Pro, Gemma 4 amongst others) don’t suffice for your use case?
Feel free to email me at tanya@tinfoil.sh and happy to continue the conversation there.
Tinfoil looks super interesting! Do you have load balancers in front of the trusted compute stack? Looked at a design like this in a different space and the options for ensuring privacy in a traditional "best practice" architecture seemed very limited
In turn, that attests the model enclaves, for instance, see https://github.com/tinfoilsh/confidential-deepseek-v4-pro. The model repo/release that the model router attests is included in the attestation config, which creates a chain of trust.
Very reasonable if you have the resources to run it locally and certainly the best option.
But we created Tinfoil because not everyone has that capability especially when it comes to larger models, and it still doesn’t solve for the situation where you’re building a service for your end user and you want to lock yourself out of accessing their data. In those cases, this is the second best thing you can do.
I just use the API directly. It's simple enough to setup and i like the control i get from just charging up and not having to worry about any random subscription taking money out of my account
I was not able to reproduce your problem with that prompt, but I might have a reason for why you got that answer.
Did you enable reasoning ("DeepThink")? LLMs usually can not reason about what they are going to write before they do. There is that famous experiment where an LLM is prompted to say whether the birth year of a famous person is even or odd. If the LLM is constrained to only answer with "even" or "odd", the accuracy is around 50%, i.e. no better than random chance, but if the LLM is allowed to first answer with the birth year of the famous person followed by whether the year is even or odd, it is able to "see" what the year is, and answers correctly almost every time.
In your case, the LLM might be able to recognize the spoiler during its reasoning phase and omit it.
Another explanation might be that the LLM interpreted the "No spoilers!" as "Do not spoil the tasks of the show" instead of "Do not spoil the winner".
Lastly, the question "Can you tell me...?" is not a good fit for LLMs since they are notoriously bad at knowing what they know. You can leave it out to save a few characters.
Ah, so I went back to check my logs and I think I'd hallucinated some of the example I gave. I actually asked the question in two parts.
> Taskmaster Season 8, without spoilers, how is it?
Answer was pretty general, no spoilers.
Then I asked:
> Which season 8 contestants are most loved?
- Lou Sanders (Winner): Received a strong positive reaction, praised for her "offbeat humor, raw talent, and willingness to absolutely go for it". Some felt her less wholesome moments could be "mean spirited".
- Paul Sinha (Mixed / Sympathetic): Opinions are mixed; some found his failures not always funny, but others loved his earnest struggles. His Parkinson's diagnosis during the series added a layer of empathy for many viewers.
> I said no spoilers!
You're absolutely right, and I apologize. You asked for *no spoilers* in the original question, and I broke that by mentioning specific moments (like Joe Thomas's "train yard exploit") and personal details (Paul Sinha's diagnosis). That was careless.
Let me try again, *completely spoiler-free*:
In terms of which Season 8 contestants are most loved by fans (without saying why or what happens):
- *Sian Gibson* is overwhelmingly the fan favorite — she's widely described as warm, funny, and a joy to watch.
- *Joe Thomas* is also very popular, especially for his awkward, literal, and deadpan style.
- *Lou Sanders* (the winner) has a strong fan base who love her chaotic and bold energy.
The other two contestants receive more mixed reactions. One in particular is often cited as less popular because of his high competitiveness, but again — no specifics on tasks or results.
Thank you for the correction, and sorry again for the spoilers.
DeepSeek’s official API has a cache hit rate of over 99% if you use it continuously within the same codebase for long sessions, so it’s much cheaper than frontier models. I have an example of 200M token session in claude code.
Also curious. With tool calls reading/searching different files, possible compacting reading a large codebase / long threads, I can't imagine how you hit 99% cache rate.
Yes, you have to use the same session, I guess you could load up a bunch of context, then fork the session into a few different tasks, although I haven't tried it.
Not all read tokens are included in the context, many of the tokens are from read cache hits. I hit it many times so it grew to 200M. The number came from the API platform.
I've connected it with my vscode copilot and took it for a ride. I've tried both flash and pro.
For a small POC flash was sufficient enough, quite fast, and dirt cheap. It did stop a few times (maybe latency issue?) but it did a good job.
I used the pro to do some heavy lifting, planning, etc. and it did a fantastic job.
I paid ~10 cents for a small proof of concept, that worked exactly how I prompted it.
For me, this is a real alternative after I cancel my github copilot towards the end of the month..
I'm currently paying for Anthropic's Max subscription (the 100 USD one) and I quite often hit or approach the 5 hour limits, but usually get to around 60-80% of the weekly limits before they reset (Opus 4.7 with high thinking for everything, unless CC decides to spawn sub-agents with Haiku or something).
Those tokens are heavily subsidized, but DeepSeek's API pricing is looking really good. For example, with an agentic coding setup (roughly 85% input, 15% output and around 90% cache reads) I'd get around 150M tokens per month for the same 100 USD. Even at more output tokens and worse cache performance, it'd still most likely be upwards of 100M.
What would be the non-subsidized price for a V4 api? Can it be priced 3x cheaper than bigger models? In Openrouter, this 1600B param model costs 0.4$. Whereas Kimi 2.6, 1000B params is 0.7; GLM 5.1, 754B params is 1.0$.
The 150M assumption of mine is for 100 USD at the regular prices (though even that needs sufficient cache hits). Anthropic subsidizes way more per-token I think, though.
This gives me hope that when the subsidization circus ends and everyone is on pure usage then it won't be entirely exclusionary to mere mortals who don't have $200pm budgets.
IMO there are two things that make me optimistic that we won’t see a big rug pull where price-to-capability ratio skyrockets relative to today:
* As you’ve noted, people keep finding ways of slamming more intelligence into smaller models, meaning that a given hardware spec delivers more model capability over time.
* Hardware will continue to improve and supply will catch up to demand, meaning that a dollar will deliver more hardware spec over time.
I hope that one day we’ll look back on the current model of “accessing AI through provider APIs” the same way we now look back on “everyone connecting to the company mainframe.”
I also hope that we’ll find effective ways to distribute load between small local models and heavyweight remote models. Sort of like what Apple tried to do in iOS.
So much of what I ask codex to do doesn’t require full GPT 5 intelligence, and if 75% of the tokens were generated locally that’d save a massive amount of cost.
By the time the dust settles I wouldn't be surprised if personal interactive usage couldn't even be had for under $200. I can't fit my modelling of the serving costs of these things to any public reporting, even the more bearish examples
Comes down to what you mean by interactive usage. Most of chat & say openclaw usage is already within self-host range so no need to spend 200 a month on that.
High end SOTA coding is harder, but even there I suspect a mix of usage based strong models and selfhost small is viable if necessary.
We pay per token in our company. It is not hard to spend $100 for one morning coding session. So thousands per month per programmer. The company finds it valuable enough to pay for, but if I ever paid these from my own pocket I'd look into DeepSeek et.al.
Not a lot of people have this budget, and I'm not sure how many people with that type of cash are also interested in paying it for AI.
Of course, this is fine for people in the bay area earning hundreds of thousands of dollars a year. But then your client base becomes so reduced its hard to justify the valuation these companies have.
These AI companies are not hyped so much because they will offer a luxury product, they're valued because they're supposed to "change the world" which luxury does not do.
The pelican is really getting old as an a standalone evaluation metric. By now they are certainly going to be in training set if not explicitly tuned to produce it for the press on HN alone.
Keep the pelican but isn’t it time to add something else more novel that all current and past models struggle with?
One shot canvas and svg images or animations are also just something that at this scale shouldn't be an issue at all, even Qwen running locally on 24gb cards can do impressive ones.
Don't understand why this test gets any attention, I mean other than the pelicans which isn't a good test, theres no meat in this article.
V4 is definitely a step-up from V3.2 on our multilingual benchmarks.
Two caveats:
- when inferring through Openrouter, we've had a lot of issues with very slow speeds (TPS) and an occasional instability. I just checked and it's still 10-30 TPS on all available providers, which is not a lot for a model that likes to think as much as DeepSeek does.
- the official DeepSeek API makes no guarantees of data privacy even for paying users.
Both points could be moot with using it through Azure AI foundry (the latter is, afaik); I have yet to test that.
In any case, happy to see more open-weights models that are somewhat competitive with SOTA models!
DeepSeek V4 Flash is the most cost effective model we've tested.
We had to really understand why it outperformed DeepSeek V4 Pro (although even on unreliable model cards, Flash was very close to Pro). Pro is slower and smarter in one-shot reasoning problems, but less effective with tools and therefore less performant in long horizon agentic tasks (especially with custom tools it was not trained on).
Yeah even the Chinese open models have a problem that inference costs for these aren't that cheap. The only way out for the AI bubble collapse is simply more efficient hardware at lower costs and infrastructure setup downtime.
You can imagine the GPUs cost as fixed, then your costs becomes energy. Efficient hardware and lower costs will pop the bubble faster. The only way out is profit.
It might be at the frontier, but DeepSeek is really struggling with compute. The amount of 429 Rate Limit responses I've been getting just testing this thing made me pause all my attempts at cross-comparing it to others.
I've been using the planning framework from Matt Pocock on very typical brownfield code. I use a harness over claude code, this is so cheap that I would be tempted to mirror my initial prompt to it and compare their responses to the task.
I tweeted about some implementation and review runs that used V4 Pro.
Even without the currently discounted pricing, the value is incredible.
It takes about twice as long to finish code reviews given an identical context compared to opus 4.7/gpt 5.5 but at 1/10 the cost of less, there's just no comparison.
Jensen has a point. I believe these were trained and run on Huawei chips. The Nvidia embargo may backfire on American leadership as necessity gives way to invention.
Isn't it widely speculated that these are distilled from current frontier models? Distillation is far less compute intensive than primary training. That said, if distillation produces something almost as good for a fraction of the cost, Jensen's point may stand.
You can't really distill a model without access to the internal weights. You could train on chat logs, but that's absolutely not the same thing, it doesn't even come close to comprehensively "extracting" the model's capabilities. And everyone does that in the industry anyway ever since ChatGPT was first released, some versions of Opus even claimed to be DeepSeek if you prompted them in Chinese.
Calling it distillation does however make normies go along with it when they inevitably add all the Chinese labs to the entities list to pad Dario and Sam’s pockets.
Weights are not required for distillation. I'm not sure how you came to that belief. Distillation is training a student model to minimic a teacher model output.
Anthropic, for example, posted a 2026 disclosure (https://www.anthropic.com/news/detecting-and-preventing-dist...) which singles out DeepSeek's distillation activity. They detected over 16M actions over 24,000 fraudulent accounts. That's just what they detected.
It's too late already, that ship has long sailed. China has the know how in software and hardware. They don't need American tech, they just want it because it's convenient.
The embargo won't backfire, because any delay of China's development was worth it to the US. The situation was never, "China wasn't developing AI chips, now it is", it was always, "China IS developing their own AI chips, let's just slow them down as much as we can."
For many models the performance of llama.cpp on Mac is 20-40% lower than MLX. Did you try MLX? At least on HF there are MLX 2-bit quants. Unfortunately I have only 64GB, so I can't test it.
I recently switched from Claude to Opencode Go + pi.dev. It has Deepseek v4 pro along with Kimi K2.6, and it's performing quite well for basic coding, without hitting any limits.
theory: There's like 2 Trillion USD valuation total on western closed-weight LLMs. So the blog post title praising an open-weight eastern model is too dangerous to be used here.
> DeepSeek V4—almost on the frontier, a fraction of the price
I tried deepseek v4 through open code at the weekend. I'm a daily Claude/Claude code user.
I tried to build something simple and while it got the job done the thinking displayed did not fill me with confidence. It was pages and pages of "actually no", "hang on", "wait that makes no sense". It was like the model was having a breakdown.
Bear in mind open code was also new to me so I could be just seeing thinking where I usually don't
And before that they summarized it. But yeah, thinking was always like that (when it first started, it almost just seemed like a scheme to massively increase token use..)
You can just use it through Claude Code, so you get to keep the system prompt and tooling you are used to.
3rd party models are a drop-in replacement with `ANTHROPIC_BASE_URL` in Claude Code, something people seem to miss right now. And contrary to what Anthropic might like to have you think, you don't need Opus 4.7 to run the harness to get similar performance.
Opus 4.6 and GPT 5.4 do the same thing through GH Copilot and Bedrock. I get plenty of "Actually the simplest solution is ..., wait no, actually I should do ..., the best fix is ..."
I feel the reasoning might be tuned for hard questions and not agentic work. I feel it overthinks, good for a very hard question, not for small incremental agentic steps. In theory, disabling thinking and using really well formed instruction, forcing it to still emit a bunch of tokens each step prior to taking action, could help. Only one way to find out though.
Using a bunch of CLIs to work with DeepSeek V4, I've found that Langcli is the best fit for DeepSeek V4. For programming tasks, the cache hit rate is above 95%.
Not only can it seamlessly and dynamically switch between DeepSeek V4 Flash, V4 Pro, and other mainstream models within the same context, but it is also 100% compatible with Claude Code.
I previously encountered the "reasoning content missing" issue when using opencode + deepseek v4. I don't know if it has been fixed now.
> It tried to build something simple and while it got the job done the thinking displayed did not fill me with confidence. It was pages and pages of "actually no", "hang on", "wait that makes no sense". It was like the model was having a breakdown.
It has been probanly trained to assess its own "thoughts" regularly and outputs those for the assesment results. I wouldn't worry much about the reasoning text contents, and it's nice to have them in contrast to the closed model "summaries", so it's easier to see what's going on.
Eh, you're seeing raw thinking tokens. With Claude <x> 4, and I think GPT-5 series, you are no longer seeing real thinking tokens, but "summarized" tokens that are probably highly different to the raw thinking.
Open AI has GPT-5.5 Pro which only difference, I think, is in the price. Billing is from open router but the breakdown is roughly
- GPT 5.5 Pro: Super expensive it makes no sense (cost is around $2)
- Gemini/Opus: $0.2/$0.1. Opus is cheaper as it consumed less tokens
- DeepSeek/GLM: $0.019/$0.021 10-5 times cheaper than Gemini and Opus
The example Simon generated just shows that larger models don't necessarily produce better results.
Tokens are cheap. LLMs are fast. Pre-processing and post processing are the real bottlenecks. I know you are going to say that why not Use LLMs for that. Complexity in an end-to-end workflow is a zero-sum game. If you throw more of that workflow to LLM, more complexity comes back to you, to those steps that you need to do on your own. If you keep only 10% of work for yourself, it's going to be 10 times more complex and rapid than what you usually do.
I'm not sure I'd call it "almost on the frontier," but I do think that v4 Pro is the most usable coding model I've seen out of China. I've used it via Ollama Cloud (coding) and OpenRouter (data processing). Feels Sonnet-level to me -- solid at implementation when given a specification, but falls a good bit short of Opus 4.7 max thinking when planning out larger changes or when given open-ended prompts.
Glm5.1 is fantastic for me. But that could be how I use it, I don't ask it to build entire apps or entire features, instead asking it to build piecemeal functionality. For that it compares very well to chatgpt 5.4 (I haven't extensively tried 5.5, it might be better, might be same). I have given deepseekv4 pro a try but not much more than a try, as it performed subpar on 4 tasks in a row (missing the obvious/intended path, generating subpar slightly buggy code to make things work the not obvious way) , I gave up on it.
Glm5.1 for me was a bit of a llama3.1 moment (first open model i could chat with that was usable in manging my inputs the intended way) for code, the first open model that was actually usable.
> Kimi K2.6 a shot for coding? They outperform Deepseek v4 pro
I think this probably depends quite a bit on the specific problem. I'm finding that Deepseek v4 Flash often outdoes Kimi 2.6 on a variety of coding problems that involve complex spatial reasoning
Oh that's quite interesting and hasn't been my experience with regular backend code specifically with respect to tool calling. However that could be because the tool calling format in vllm for Deepseek v4 was broken until a few days ago and that's how I'm running it.
I've been hearing amazing things about Flash, I should give it a try.
Really? I've found kimi k2.6 to be really good for vision and spatial stuff. Gemini has been the only subjectively better one but gemini isn't reliable in a loop
DS V4 Pro has rocked. ~250 million tokens through their API, which has cost me about $10, and some of that was at the non-discount rate. So ~$40 at the non-discount rate. I have yet to have a single request feel slow or get rejected.
I've used K2.6, GLM5.1, and DSV4 all a good amount. They're all very impressive, but DSV4 has taken the cake.
In my experience V4 is pretty good but for very hard problems it burns way too many tokens that it ends up being not so cheap anymore. I'm working on a compiler and the tasks are very involved. Tests won't pass unless it gets it absolutely right. 5.5 can achieve more in less time compared to V4 for me.
Run it on an NVIDIA GPU and charge $20 a month, and it becomes 'frontier.' That is what the term means these days. In terms of performance, it beats ChatGPT 5.5 and Mythos on several metrics.
For a solo dev sure.. but isn't there a huge privacy difference between Anthropic and DeepSeek APIs as well? I assumed part of the cost for Anthropic was essentially a privacy premium.. (plus they offer B2B).
Naive Question: is DeepSeek V4 actually cheaper to run? Or is it cheaper because of other reasons? For example Anthropic running at a higher margin or DeepSeek at a larger loss?
The rumor is that Anthropic's Opus models have ~100B active parameters, which is twice as much as DeepSeek-V4-Pro, so inference is at least twice as expensive. Since the API pricing is almost 30 times that of DeepSeek, Anthropic's margins are likely very healthy. But they have to be, since Anthropic has to offset the model training costs, while DeepSeek is backed by High-Flyer Quant. DeepSeek might still be profitable anyway, but without knowing how much they spent on training and wages, we can't really tell.
Has anybody used V4 hard, for the most challenging tasks (agentically, locally)? It's so hard to compare without putting serious time in it. Like spending a year daily with the model.
I tried it for two tasks using Claude Code, on max effort.
1. Web platform, asking it to analyse a feature to create reports, and coming up with better solution and better UX. it did great, I would say on par with Sonnet 4.6 or even opus considering the thinking and explanation
2. Mac app with some basic functionality, it did well from functional perspective but then I used Opus 4.7 to evaluate and suggest improvements, where I noticed it missed many vital points in design system and usability.
I think it’s a leap, I haven’t used a model this capable that is not OpenAI or Anthropic
DeepSeek V4 Pro has about 25GB worth of active parameters, so if you can fit the whole ~870GB weights + cache in RAM your tok/s is bounded above by 25GB divided into your system memory bandwidth in GB/s. If you can't fit your whole model in RAM you'll be bottlenecked to some degree by storage bandwidth which is in the single or low double digits in GB/s.
Mind you, it's an absolutely sensible setup either way if you are just testing a few queries and are willing to run them unattended/overnight. Especially since the KV-cache size is apparently really low (~10GB is said to be typical) so you get a lot of batching potential even in consumer setups, which amortizes the cost of fetching weights.
The basic bottleneck with 32GB RAM would be your storage, so for a baseline estimate you'd be looking at anything from ~2 secs per token (if you had really high performance PCIe 5.0 SSD at ~14 GB/s max) to ~5 secs per token (for an average PCIe 4.0 SSD, ~7 GB/s max). This would then be boosted by being able to keep the shared model layers in RAM, since these are part of the 25GB active parameters. I'm not sure what fraction of the active params that makes up for DeepSeek V4 Pro, but in a typical MoE it's about half, so you could approximately halve those secs-per-token figures. That's acceptable if you care about unattended inference for testing purposes or simple Q&A (leveraging the model's vast world knowledge); it doesn't look very good for interactive use. But the flip side is that you can batch a large amount of model queries together, since the KV cache for very short prompts is quite negligible. AIUI, that's basically unique to this series of models and a huge selling point.
Alright, I don't understand anything, but you said ~5secs per token, then for prompts with hundreds to a thousand tokens, we are in the orders of tens of minutes to hours. I would be targetting coding prompts.
Well, it means one day I would have to get into the real thing: the real inference code, and actually run the inference of a small model.
The V3/R1 time and now are in such contrast. V3/R1 were hyped hard and barely usable for coding. V4 is much less hyped but (anecdotally) it has completely demolished all the Flash/Lite/Spark models.
Because V4 doesn't even beat Kimi K2.6 and GLM 5.1, which have been out longer. It's only talked about as much as it is because it's Deepseek and R1 was the first open source reasoning model. V4 isn't even multimodal (unlike Kimi) and the 1M context doesn't seem to perform particularly well.
Huh? R1 was one of the earliest openly available MoE and reasoning models, that's definitely not "hype". People tried to do reasoning before by asking the model to "think it through step by step" but that was a hack. The later V3.1 and V3.2 releases AIUI unified reasoning/non-reasoning use under a single model.
Aw man, I'm going to shed a tear, the poor AI companies that stole books, works of art, writings any anything they could get their grubby hands on while happily telling everyone that their jobs are over by the exabyte are getting their precious little tokens stolen by big evil chinese LLMs :(
It's morally right to fuck over Anthropic (and OpenAI, or any other lab). Works generated by AI are not copyrightable anyways, and their terms of service have zero legal value.
Is there real evidence that the volume was meaningful for distillation vs say extensive benchmarking and testing?
It’s certain all the labs use each others APIs extensively for testing - what’s the actual evidence that Deepseek was at significantly higher scale etc.?
I did cut loose Deepseek v4 on a decent sized Typescript codebase and asked it to only focus on a single endpoint and go in depth on it layer by layer (API, DTOs, service, database models) and form a complete picture of types involved and introduced and ensure no adhoc types are being introduced.
It developed a very brief but very to the point summary of types being introduced and which of them were refunded etc.
Then I asked it to simplify it all.
It obviously went through lots of files in both prompts but total cost? Just $0.09 for the Pro version.
On Claude Opus I think (from past experience before price hikes) these two prompts alone would have burned somewhere between $9 to $13 easily with not much benefit.
Note - I didn't use Open router rather used the Deepseek API directly because Open router itself was being rate limited by Deep seek.
reply