AI at a Crossroads: Why Wall Street Is Saying 'No' to ChatGPT and Claude
Author: Jeff @IOSG
Why Private AI Is Needed
Over the past two decades, enterprises have adopted cloud software based on trust established at the protocol level—and it worked. Each SaaS vendor only saw a slice of enterprise data and generally had little incentive to use customer data to enhance its core products. Salesforce saw sales pipelines, Workday saw HR data, Jira saw development workflows, and AWS provided the foundational storage and compute infrastructure. However, today’s AI workflows advocate uploading an organization’s entire dataset at once—including the structured context linking departments—to maximize productivity. Regardless of good intentions, upstream service providers can now leverage this data to build new features rather than letting it sit idle on servers.
Some enterprises acted early. In February 2023, within three months of ChatGPT’s launch, major Wall Street banks had already restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company banned generative AI across its entire network. In response, OpenAI launched ChatGPT Enterprise in August that year, promising not to train on commercial data and introducing a zero-data-retention (ZDR) policy—now a standard requirement for enterprise procurement.
But contracts only lock down corporate accounts. IBM found that by 2025, shadow AI—employees feeding company data into unauthorized AI tools via personal accounts—was involved in one-fifth of all data breaches, with heavy shadow AI usage adding an average of $670,000 to breach costs. In a 2025 survey by security training firm Anagram, 40% of employees said they would violate AI usage policies to complete tasks faster.
A court order in May 2025 forced OpenAI to retain consumer-grade chats even after users deleted them. In November, a judge ordered OpenAI to hand over 20 million of those messages to The New York Times’ lawyers as part of discovery. Then came criminal cases: ChatGPT logs from the defendant in the Palisades wildfire arson case were admitted as evidence, and an affidavit in a double homicide in Florida cited the suspect’s queries about how to dispose of bodies. In a July 2025 interview, Sam Altman acknowledged that ChatGPT conversations are not protected by legal privilege, and that OpenAI ‘may be required to turn over’ user chat logs in litigation.
Open-source models that run on self-hosted or verifiable environments are catching up fast, but the strongest still lag behind cutting-edge closed-source models by roughly four months in general capabilities. This leaves token-maximizing enterprises and individuals at a crossroads: either sacrifice a few months of model quality for privacy, or continue uploading sensitive material to Anthropic’s servers—because that’s exactly how competitors are gaining productivity advantages.
There is currently no perfect solution on the market. This report outlines various attempts to close the gap and assesses how far provably private, frontier-level AI remains from being delivered into the hands of enterprises and everyday users.
Private AI isn’t a single engineering feat. Every mechanism currently on the market addresses the same event: a prompt leaves your device, travels across the network, lands on the machine running the model, and returns a response. The differences between mechanisms lie in where plaintext exists along this path, who can read it there, and what guarantees the privacy of the response.
Protocol-level privacy
Contractual zero retention
Anonymous proxy
Every leg of machine-to-machine communication runs over TLS, which only encrypts the transmission channel—the receiving party can still read all information. Relays often use Oblivious HTTP (RFC 9458) to split this knowledge: like asking a friend to pass a note—the friend knows who sent it but can’t read its contents, while the recipient can read the message but doesn’t know who wrote it. OHTTP became an IETF standard in January 2024, and many companies now route production traffic through OHTTP relays leased from Cloudflare and Fastly.
This represents the upper limit of privacy attainable when accessing closed-source models—and the reason is arithmetic. A single flagship training run now costs on the order of billions of dollars, and these labs’ valuations—approaching trillions—rest entirely on exclusive control of their model weights. As long as capability gaps persist, so will premium pricing, which is why labs guard their weight files like state secrets.
Meta has already undergone this experiment involuntarily. When LLaMA was first released in February 2023 exclusively to researchers, its weights leaked onto 4chan as a torrent within a week. Within another week, llama.cpp enabled the smallest 7B model to run locally on a MacBook; three days later, Stanford fine-tuned the same model into a chat assistant called Alpaca for under $600. This leak reduced Llama’s operational cost to just electricity—anyone with the file could run it at home. In July 2023, Meta officially open-sourced Llama 2 under a commercial license that excluded companies with over 700 million monthly active users. Once the weights escaped, so did the premium.
Cutting-edge labs could theoretically implement attestation (remote verification) for inference on closed-source models, but attestation can only confirm which code segment read the prompt—it cannot verify what that code did with it. To determine whether a server retains data, we’d need to audit the serving code and reconstruct it to match the hash reported by the hardware. But disclosing the serving code would also reveal the batch processing and caching optimizations that underpin the lab’s profit margins—techniques that carry forward into every subsequent model generation. Apple and Meta can afford to provide remote attestation for the service stacks behind iPhone and WhatsApp because their profits come from devices and advertising; publishing their serving code incurs almost no cost.
Architectural-level privacy
Each mechanism in this category replaces trust-based assurances with proofs grounded in hardware, cryptography, or physics—but each comes with its own trade-offs for enhancing privacy, chief among them being that they can only run open-source models.
TEE (Trusted Execution Environment) confidential computing
Run inference inside a hardware enclave—a sealed compartment on the chip that even the machine operator cannot access—and have the chip issue an attestation specifying exactly which model and code segment were executed.
It eliminates readable intermediaries. The user device encrypts the prompt using the enclave’s key, so every hop along the way carries a sealed envelope that only the enclave can open.
Trust is placed in the client. The same client-side code responsible for encrypting the prompt and verifying the attestation also has the power to undermine that guarantee. Therefore, verifiable E2EE requires both a proven enclave and open, reproducible client code.
Compared to the simplicity of TEE, E2EE imposes significant engineering overhead, slowing feature integration. By turning the proxy into a blind messenger, E2EE forces all functionalities that depend on reading plaintext to be rebuilt around client-side keys—or confined entirely within the enclave.
FHE (Fully Homomorphic Encryption, and MPC variants)
The trade-off is speed. Native FHE supports only addition and multiplication, so the nonlinear operations required by transformers must be reconstructed at significant cost. Inference on encrypted data is 10,000 to 100,000 times slower than on plaintext—each token takes several seconds to minutes even on small models, compared to milliseconds without encryption.
Local inference
The obvious costs are expense and model capability. gpt-oss-120b scores roughly half as high as GLM-5.2 on the Artificial Analysis index, yet it occupies 65GB—more than the combined VRAM of two flagship gaming GPUs currently on the market. Meanwhile, full-precision GLM-5.2 can only run on an 8-GPU data center node, with GPUs alone costing over $300,000.
In fact, enclaves themselves add virtually no extra operational cost for providers. Every H100 shipped since 2023 includes native enclave support—the additional cost stems solely from the throughput loss due to encryption, not from extra hardware. Currently, Azure’s confidential H100 SKU rents for $8.90 per hour, versus $6.98 without enclaves—a 27% premium over conventional cloud infrastructure. Meanwhile, specialized enclave providers like Phala offer confidential H100 instances starting at $3.80 per hour, undercutting Lambda’s standard SXM cards priced between $3.99 and $4.29 per hour. On the managed API front, NEAR AI offers gpt-oss-120b with attestation at $0.15 per million input tokens and $0.55 per million output tokens—on par with plaintext offerings from Amazon Bedrock, Together, and Groq. Even for models requiring multi-chip parallelism, NEAR AI matches Fireworks’ pricing exactly for GLM-5.2 and undercuts it by 15% on input and 4% on output for the larger Kimi K2.6.
Although these new privacy-focused inference providers may be sacrificing margins to gain market share—a statement true of any growth-oriented company in any market—the structural trend is clear: the cost of privacy is falling for both consumers and operators.
The gap persists, but on June 30, AIA Labs—under Bridgewater—and Thinking Machines presented a compelling case: an open model fine-tuned with expert annotations outperformed frontier models in both accuracy and cost.
In their study, the team fine-tuned Qwen3-235B using Tinker, Thinking Machines’ hosted fine-tuning API service. They first procured annotated data from vendors and used it for an initial round of training. Samples where the model disagreed with annotations were then handed over to the firm’s investment professionals for re-annotation. The training employed reinforcement learning (GRPO), enhanced with three modifications: round-robin batching (each task contributes one batch in turn), CISPO loss (which caps how far any single response can shift the model), and on-policy distillation (which anchors learning to the current best checkpoint to prevent the model from regressing to weaker versions).
All tasks were drawn directly from investment professionals’ daily workflows: determining whether a news article matters to C-suite-level investors, assessing if a central bank document hints at future interest rate direction, or identifying where boilerplate language begins in a document or email. Evaluation was conducted on an independent test set. Frontier models scored roughly 50% on average with simple prompts and only reached 78.2% even when augmented with expert-crafted prompts—still below the 80% threshold set by investment staff. In contrast, the fine-tuned Qwen achieved 84.7%, which, according to the paper, translates to 29.8% fewer errors than the best frontier model and inference costs 13.8 times lower.
This case demonstrates that open-source models can prevail on both accuracy and cost—but the training process itself wasn’t fully private. The expert annotations used were Bridgewater’s proprietary data, processed through Tinker’s third-party service, placing it at the same trust level as agreements under the ZDR framework. The fund also rented compute capacity, meaning the entire training ran on machines it never controlled. Buyers who want this recipe but refuse to accept such trust assumptions have few options today: renting bare-metal GPU clusters leaves the training process readable to the cloud operator, while purchasing dedicated clusters solves the data custody issue but sends costs soaring.
A solution with attestation has just emerged. In March, Workshop Labs and Tinfoil released Silo—a post-training stack running inside a Tinfoil enclave on a single 8-GPU node, with cryptographic keys held exclusively by the client. According to the paper, enclave overhead adds only 11 minutes to a two-hour training run. Moreover, by freezing base model weights and training only small adapters on top, the stack can accommodate trillion-parameter models like Kimi K2 Thinking. The main challenge lies in reinforcement learning, which requires frequent data transfers between components—precisely where enclaves incur the highest performance costs.
Less than a month after Silo’s launch, Workshop Labs was acquired by Thinking Machines. Now, all components needed to run a Bridgewater-style RL loop inside an enclave fall under a single company.
Privacy at the Harness Layer
Another issue looms beyond all private inference mechanisms. These approaches only secure the path from prompt to model, yet every external tool call initiated by an agent creates a new channel entirely outside the inference layer’s control. The recent surge in harness engineering has magnified this problem: every tool, memory store, and data source connected to the model becomes another destination that reads its slice of the workflow in plaintext. Calendar servers see meeting details; database servers see queries. Even a fully local agent, if it needs anything beyond its training data, must send search terms in plaintext to a search engine—because if the server can’t read the plaintext, it can’t answer the query.
The prevailing approach still defaults to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to manage all tool traffic, stripping personally identifiable information (PII) before requests leave the user’s device. The gateway also decides which servers receive traffic—blocking unvetted ones—and logs the destination and content of every call for potential forensic use. Even if these controls carry independent audits (e.g., SOC 2), tool servers must still read queries in plaintext to respond; whether they retain copies depends on their own data retention policies, multiplied across every tool in the harness. Moreover, the gateway itself becomes an additional trusted party along the data path—one that reads, rather than verifies.
Architectural-level solutions target the middle layer. For example, Phala hosts MCP servers directly inside a Trusted Execution Environment (TEE), covering wallets, code execution, and data sources. Users can verify privacy claims via an attestation rather than trusting the operator. However, TEE-hosted tools ultimately still hand queries in plaintext to service providers—the enclave secures only the messenger, not the destination.
Only a few destinations have learned to respond without reading the query—but only for structured queries. Apple offers private information retrieval on iPhone, enabling phone numbers to be checked against spam databases without exposing the number. Microsoft uses the same technique for passwords in the Edge browser. MongoDB’s Queryable Encryption allows clients to encrypt fields before they leave the device, enabling servers to perform exact-match and range queries using only ciphertext.
Encrypted search is possible—it’s just not yet commercially viable in terms of performance and cost.
Outlook
Demand for private AI is growing. Venice AI recently surpassed 3.5 million registered users and processes 1.3 trillion tokens per month, followed by a new Series A funding round valuing it at $1 billion. Proton is its direct competitor; its chat product Lumo hit 10 million users within a year of launch. On the infrastructure side, Phala already runs 2–3 billion tokens daily on OpenRouter. Duck.ai routes models like gpt-oss-120b and Gemma into Tinfoil enclaves, offering verifiable privacy beyond user agents. And that doesn’t even count self-hosting—which may well be the largest channel for private inference, since running models on one’s own hardware leaves no usage traces.
Current hosting solutions aren’t perfect either. Users seeking maximum privacy via end-to-end encryption (E2EE) must wait for new features to be rebuilt entirely outside service providers’ visibility. Private harnesses still rely on protocols at the service layer. Affordable post-training fine-tuning still requires trusting third-party vendors for optimal results. Self-hosting eliminates all service providers at once—but running the most powerful open-source models locally can cost more than the house housing the hardware.
Yet the key value capture lies beyond these compressed pricing layers. Privacy is nearly free where it already exists—but it hasn’t yet reached mainstream agentic workflows. Operators focused solely on renting enclaves control just a switch on standard chips, not a moat; protocol-layer gateways compete directly with traditional middleware. Defensible ground lies in the unsolved half of this report: training loops locked inside enclaves, end-to-end sealed tool calls, and search indexes that never see actual query terms. Whoever delivers any one of these first will sell something immune to commoditization by price wars. Capital chasing private AI should buy the gap—not the switch.
开源模型怎么赢？
尽管性能开销在压缩，前沿模型与 SOTA 开源模型之间仍有一段肉眼可见的差距，一个追求生产力最大化的主体想留在最前排，仍得信任前沿实验室不窃取自己的 IP。
差距还在，但 Bridgewater 旗下 AIA Labs 与 Thinking Machines 在 6 月 30 日给出了一个案例：一个用专家标注微调的开放模型，在准确率和成本上同时击败了前沿模型。
研究中，团队在 Tinker (Thinking Machines 的托管微调 API 服务) 上微调 Qwen3-235B。他们先从供应商处采购标注，用这批数据训练第一轮,再把分歧样本转给公司的投资人员重新标注。训练跑的是强化学习 (GRPO)，外加三处修改：round-robin batching (各任务轮流出一个批次)、CISPO loss (限制单条答案能把模型拉动多远的上限)、on-policy distillation (锚定当前最优 checkpoint，确保模型不会向更弱的副本学习)。
任务全部取自投资人员的日常工作流：一篇新闻对 C-suite 级投资专业人士是否重要，一份央行文件是否暗示未来利率变动的方向，一份文档或一封邮件里的模板套话从哪里开始。评分来自独立测试集，前沿模型在简单 prompt 下平均得分约 50%，配上专家 prompt 也只冲到 78.2%，低于投资人员设置的 80% 门槛。而微调后的Qwen拿到 84.7%，按原文的口径,这相当于比前沿最优少犯 29.8% 的错，推理成本低 13.8 倍。
这个案例证明了开源模型能在准确率和成本上取胜，但训练过程仍然不是私有的。过程中使用的专家标注是 Bridgewater 的私有数据，途经 Tinker 的第三方服务,落在与 ZDR 协议同一个信任层级。基金还租用了算力，整场训练跑在它从未掌控的机器上。想要这套配方又不想背信任假设的买家，今天的选择很少。租裸 GPU 集群，训练过程对云运营方可读。买下集群解决了数据托管问题，但成本一飞冲天。
带 attestation 的路线刚刚到来。3 月,Workshop Labs 和 Tinfoil 发布了 Silo，一套跑在 Tinfoil enclave 里、单个 8 卡节点上的后训练栈，钥匙只由客户掌控。文章给出的 enclave 成本是，两小时的训练多花 11 分钟，而且这套栈通过冻结基座权重、只在其上训练小型 adapter，可以装得下一个万亿参数模型 (Kimi K2 Thinking)。难点在于，强化学习需要在各组件之间来回搬数据，而搬数据恰恰是 enclave 成本所在。
Silo 发布后不到一个月，Workshop Labs 就被 Thinking Machines 收购，在 enclave 里跑下一个 Bridgewater 式RL循环所需的部件，如今都归到了同一家公司名下。
Harness 层的隐私
还有一个问题横在所有私有推理机制之外。这些机制各自管的是 prompt 到模型的路径,而 agent 发起的每一次外部工具调用，都开出一条推理层根本碰不到的路。最近的 harness engineering 风潮把问题成倍放大，接在模型周围的每个工具、记忆库和数据源，都是又一个以明文读取自己那份工作流切片的目的地。日历服务器读到日程，数据库服务器读到查询。一个完全本地的 agent 如果只要想要训练集之外的任何东西，还是需要把搜索词以明文形式递给搜索引擎，服务端读不了明文，就回答不了问题。
主流解法仍默认落在协议层。Runlayer 和 MintMCP 这类公司用一个中央网关管控全部工具流量，在请求出门前遮蔽个人身份信息(PII)。网关同时决定哪些服务器能接到流量，把未经审查的挡在门外，并记录每次调用的目的地和内容以备取证。即便这些管控挂着独立审计 (SOC 2)，工具服务器还是得读明文查询才能作答，它留不留副本取决于自家的留存条款，并且要乘上 harness 里的每一个工具。此外，网关本身也是路径上多出来的一个依赖信任的读取方，而不是验证。
结构级的方案打到了中间那一层。比如 Phala 把 MCP server 直接托管进 TEE，目录覆盖钱包、代码执行和数据源，用户可以凭一份 attestation 验证隐私声明，而不是信任运营方。然而 TEE 托管的工具最终还是要把查询以明文交给服务提供方，enclave 封住的只是信使，不是目的地。
只有少数目的地学会了不读也能作答，但仅限结构化查询。Apple 为 iPhone 提供私有信息检索，让来电号码比对垃圾电话库时无需暴露号码,Microsoft 在 Edge 浏览器里对密码用了同一种方案。MongoDB 的 Queryable Encryption 让客户端在字段离开前就加密，服务器仅凭密文就可以完成等值和范围匹配。
但对于开放式搜索，今天最好的答案止步于信任，可验证的加密搜索还没走出实验室。Brave 在自家 400 亿页的索引 (而非 Google 的) 上承诺零数据留存，可它依然落在协议层。Exa 建了一套神经索引，把用户的关键词嵌入成语义，按语义匹配给结果排序，但嵌入这一步仍在 Exa 的服务器上从明文算起。MIT 2023 年的 Tiptoe 论文在 3.6 亿网页上完成排序而不暴露查询，但每次搜索都要烧掉大量服务器算力，排序质量与不加密的搜索有差距。Apple 2024 年的 Wally 论文通过把真实查询藏进一堆诱饵里,使得通信成本最多压低 31 倍，但这套数学要到数百万并发查询才变得便宜，而这个规模，今天没有任何私有搜索系统拥有。
加密搜索做得到，只是性能和价格都还没到商用可行的地步。
展望
私有 AI 的需求在增长。Venice AI 最近突破 350 万注册用户和每月 1.3 万亿 token 的吞吐，随后完成新一轮估值 10 亿美元的 Series A 股权融资。Proton 是它的直接竞争对手，其聊天产品 Lumo 上线一年内用户破 1000 万。基础设施方面，Phala 目前在 OpenRouter 上就日均跑 20 到 30 亿 token。Duck.ai 把 gpt-oss-120b 和 Gemma 路由进 Tinfoil 的 enclave，给用户代理之外的可验证隐私。这还没算自托管，它很可能是私有推理最大的渠道，毕竟模型跑在自己的硬件上，不留任何使用痕迹。
然而放在主流 AI 的大浪潮里，隐私 AI 仅占极小一部分，而这个差距只有在前沿实验室有意满足这份需求时才会合拢。5 月，Google 全线产品处理了 3200 万亿 token，照此计算，Venice 一个月的吞吐约等于 Google 的 18 分钟。去年 11 月，Google 上线 Private AI Compute (PAC)，把部分 Gemini 驱动的功能放进与公司自身隔离的密封 TPU enclave 里跑，并且设计由 NCC Group 独立审计。但问题在于，PAC 只覆盖个性化推荐、录音摘要这样少数 Pixel 功能，并不覆盖数亿人在用的 Gemini 应用。Google 敢把设计交给审计方，是因为这些功能靠设备和广告变现，不靠卖 token。
当下的托管方案也不完美。想通过 E2EE 拿到最高隐私的用户，得等新功能在服务商读不到的地方重建一遍。私有 harness 在服务层仍然依赖协议。价格合理的后训练，想拿到最好的微调结果仍得信任第三方供应商。自托管一次性甩开所有服务商，但在本地跑最强的开源模型，花的钱可能比插着它的那栋房子还贵。
缺陷归缺陷，私有 AI 已经是一个真实且负担得起的选项，剩下的缺口也在收窄。对普通消费者,在 Lumo 和 Venice 上，无日志承诺下的开放模型私密聊天分文不花，Venice 或 Tinfoil 的 18 到 20 美元订阅则把同样的聊天封进 enclave，不比一份 ChatGPT 订阅贵。对企业工作流，带 attestation 的端点如今比明文路线甚至更便宜。NEAR 的 E2EE API 这样的端点已经能把加密的上下文带进 enclave，记忆、文件上传、自定义指令今天都能在 E2EE 之上运转。至于带 attestation 的后训练，NVIDIA 即将推出的 Vera Rubin NVL72 会把机密计算从 Blackwell 的 8 卡节点扩展到 72 卡机架，让前沿 RL 循环在不暴露 IP 的前提下更加可行。
然而关键的价值捕获，则落在这些价格压缩的层级之外。隐私在它已经存在的地方近乎免费，但还没覆盖主流的 agentic 工作流。专注租售 enclave 的运营商握着的是标准芯片上的一个开关，不是护城河，而协议层的网关则跟传统中间件同场竞争。可防守的阵地，是这份报告里还没被解决的那一半：关在 enclave 里的训练循环、端到端封死的工具调用、看不见词条的搜索索引。谁先把其中一件做出来，卖的就是任何价格战都无法商品化的东西。追逐隐私 AI 的资本，该买的是缺口,不是那个开关。
所以，信任还是验证?对于重执行、重 agent 的任务，选信任，因为每一次工具调用本来就把明文交到了 enclave 封不住的目的地，而前沿模型在这类循环里配得上它的价格。至于把一家公司同对手区分开来的高阶思考，选验证。战略、规划,以及多年专业经验提炼出的判断，恰恰就是争议中的那份 alpha。往前的路,是在公司自控的边界内，用这些专有洞见微调开源模型。在一家公司 alpha 所在的领域里，专家调校的开放模型已经在准确率和成本上同时击败前沿，而在隐私环境下构建它的基础设施，正一个节点一个节点地到来。
Risk Disclaimer: The above content only represents the author's view. It does not represent any position or investment advice of Futu. Futu makes no representation or warranty.Read more
Comment (1)
to post a comment
1