Technology

1. OpenAI addresses distillation
2. AI ‘kill switch’ bill
3. IBM’s warning
4. Polymarket’s fake sites
5. AI scams are rising

A deeper look at Kimi K3’s capabilities, and why AI could cause the collapse of the global tax system.

Now that Kimi K3 has been out for a week, we know a lot more about how the Chinese AI model performs in the real world. Compared to other models from the frontier labs, the early technical read is that it’s kind of “meh.”

It doesn’t hold a candle to the top AI models on finding cybersecurity vulnerabilities. And while it looks efficient on paper, it burns through crazy amounts of tokens compared to US counterparts.

So how can it pose a business risk to US AI labs, or to national security, if it’s not better or cheaper than those offered by the frontier labs?

One simple answer is that it doesn’t. Kimi follows a familiar pattern that we’ve seen with other Chinese open-weight model releases: They look great on paper and cause a huge stir, but the performance doesn’t live up to the hype. Because benchmarks used to evaluate AI models are essentially take-home tests, it’s fairly easy to cheat on them. (Open-weight models also tend to work great for simpler, higher-volume tasks where speed matters more than quality.)

But even if it turns out that Kimi K3 isn’t as groundbreaking as the American government — and AI labs — fear, that doesn’t mean there aren’t real safety risks. Since the models are open, they can be stripped of any guardrails, including propaganda mandated by the Chinese government, and could become a valuable tool for hackers.

Safety advocates also argue they could help terrorists develop bioweapons, since it’s difficult to get closed-source models to help with these things. Try asking Anthropic’s Fable, for instance, to explain even simple questions about biology, and it will force you to use a less powerful model.

Fortunately, today’s open-source models from China don’t appear capable enough to do much damage yet. But they could in the future. Banning them may sound ideal, but enforcing it is futile. The better option is to start preparing now for a world in which powerful AI models can be downloaded for free and used without guardrails.

OpenAI’s Brockman: Distillation is a technical problem

OpenAI and Anthropic have pushed the US government to address the distillation of their top models by Chinese companies, framing the practice as a threat to national security and prompting Trump officials to threaten sanctions on Chinese companies.

But there’s also a way to use technology to solve the issue, OpenAI President Greg Brockman told a roomful of reporters on Thursday. “AI is going to be an important priority for national competitiveness, so it is reasonable for the public to really care about these questions, but in terms of solving it, ultimately, there is a technical question.”

Brockman said the company has systems in place to prevent distillation. It’s a process the company outlined in a letter to Congress in February, and includes using machine learning and human review to identify when users are scoring the models’ responses, attempting to extract reasoning, and generating large amounts of synthetic data — all warning signs for distillation.

Brockman also used the OpenAI briefing, which centered around GPT-Live and new voice capabilities, to push back on the idea that it’ll be undercut by Chinese models on price. “It’s not the case that open source models are magically cheap,” he said. “Everything is running on the same hardware.”

OpenAI is building out that hardware business — chips and data centers — as its moat, he said. “Now that people actually care about price, which was not the case three months ago, we are delighted,” he said. “Our goal is to always be the cheapest model for any task you want, and we have extremely advantageous conditions to deliver on that.”

— Rachyl Jones

The limits of an AI ‘kill switch’

US lawmakers introduced a bill requiring a “kill switch” for advanced AI, after an OpenAI agent went rogue and hacked into another company’s database. The bipartisan legislation, which would require companies to shut down their models at short notice, may sound good, but it would be pretty tricky to implement.

Clever AI models know they might be switched off, and can take steps to resist it, including copying themselves across the internet, hiding their motives, or disabling the switch; they have already been caught trying all three.

Kill switches would be effective against some AI systems, like if the White House felt people were misusing an AI model and wanted to shut it down, but not to stop “rogue” AI models, which would get around the kill switches by the time anyone noticed — if they notice at all.

For regulators worried about AI, a more effective use of government dollars might be new scientific research into understanding how AI works. With little insight into what LLMs are actually doing, it’s harder to figure out how to safely deploy them.

— Reed Albergotti

Mad scramble for AI dollars

IBM’s big earnings miss, previewed last week but somehow coming in even worse than suggested, boiled down to customers buying less of their AI-adjacent stuff so they could buy more of other people’s AI-critical stuff. The episode shows the mad scramble for dollars in AI, which has been a giant sucking sound in the economy for a while now. It’s also a sign the “let 1,000 flowers bloom” phase of this cycle is falling away under the harsh glare of priority-setting and thinning shareholder patience.

“Deals are slipping a few weeks here, deals are slipping a few weeks there, because people are in this process of evaluating what AI can do and what AI can’t do,” Vice Chair Gary Cohn told Semafor.

The cost to implement AI — tokens, engineers, McKinsey advice, etc. — “is occupying a huge majority of [companies’] budgets, and if they don’t get more budget, they’re forced to not buy something else,” Cohn said. Now, we’re in the “educational friction” era, as Cohn puts it, where lessons are being learned on the fly and budgets are reworked accordingly.

— Liz Hoffman

Polymarket’s fake bets training

Newly surfaced internal videos from Polymarket reveal how far the company’s employees were willing to go to build its user base and legitimacy, instructing social media creators how to stage fake bets on a mock version of the platform earlier this year.

One presentation, titled Getting Started with Fakecharts: Setup and Deposits, was shared in a Discord server that Polymarket used to communicate with dozens of social media creators. In the video, viewed by Semafor, Polymarket employee Tobin Tang walks creators through the company’s “fake charts platform,” demonstrating how to use fake money to wager on whether President Donald Trump would acquire Greenland before 2027. He enters $1,000 into a betting field, clicks a button labeled “Buy Yes,” and declares, “Boom, trade complete, just like that.”

Tang was an undergraduate student at the University of Waterloo in Ontario when he left to work for Polymarket Chief Marketing Officer Matthew Modabber, who Politico reported sent at least $350,000 to influencers via PayPal for these kinds of videos.

The videos, along with interviews with current and former Polymarket employees, provide a deeper look at how the company trained what The Wall Street Journal described as a “social media army” to stage fake bets.

— Jake Angelo

An AI-powered marketing scam

As companies like Meta push increasingly realistic AI-generated videos, the worst parts of social media and AI are converging. In what appears to be a new dropshipping scam flooding Instagram and TikTok, accounts with hundreds of thousands of followers are posting AI-generated videos depicting teens getting bullied for trying to sell Christian apparel alongside links to buy the goods. It’s part of a growing phenomenon of fabricated accounts that are using AI to stoke outrage and then drive sales.

It’s not clear who’s behind these specific accounts and if the products actually get shipped (many of the products on the linked websites also appear on other online retailers’ sites). And the videos exclude labels required by Meta and TikTok to disclose that AI is used. But after Semafor reached out to TikTok and Meta, some of the accounts were no longer available online.

— Rachyl Jones

Former Surgeon General Vivek Murthy is more worried about your phone than your body. On this week’s episode of Mixed Signals, Murthy joins Max and Ben to make the case that America’s loneliness crisis is a public health emergency — and that social media is a bigger driver of it than almost anything else we’re paying attention to. Plus, why he chose to focus on isolation rather than cancer or heart disease after his tenure as Surgeon General, whether the science on social media harms is as definitive as he suggests, and what he’d say to people who think the health risks of drinking alcohol are worth the social benefits.

One result of the rise of AI could be the collapse of the global tax system. If AI means capital and corporate profits, rather than salaries, become the main source of value in the economy, then states’ revenue-raising machinery — built around wages — becomes inadequate, a Bloomberg columnist argued. If, as after the Industrial Revolution, the upheaval leaves millions out of work for years, governments will have huge demands on their social safety net and a withered tax base to meet it. A separate issue will be the democratization of tax advice: High-end minimization, previously only available to major corporations, could be deployed to find loopholes for smaller taxpayers, although it could also be used by governments to find those loopholes.