This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

It’s a strange thing to be a week out from the US midterms, and tech hardly seeming to factor into the story at all.

We’re barely six years out from the 2020 election, when a wide swathe of the electorate followed every platform decision with great interest. Would President Trump’s election misinformation be removed? Would it receive a label? Might he even be banned — if only for a little while?

By 2024, though, Trump was completing his comeback, Elon Musk owned Twitter, and the platforms’ commitment to robust, civic-minded content moderation was in retreat. The 2024 presidential election was contested by two well known, sharply defined candidates, and while the popular vote was close, Kamala Harris’ defeat in the Electoral College was decisive. Few, if any, argued that platforms had played a particularly notable role in the outcome. 

As we head into next week, it appears that 2026 will play out similarly. Despite close attention paid to the subject by pollsters and the press, evidence seems conclusive that this will not be the AI election.

In September, the New York Times and Siena University polled 1,503 likely voters nationwide about the election. Economic concerns dominated voters’ anxieties, the poll found; less than 1 percent of respondents said that AI (or its even less popular avatar, data centers) was the most important issue in determining how they would vote.

In part, this may be because voters don’t have much faith in either party to lead when it comes to AI. The poll found that 42 percent of voters trust Republicans more, and 40 percent trust Democrats — roughly a statistical tie.

In any case, voters’ indifference to the question has complicated efforts on both sides of the AI safety question to elect candidates sympathetic to their cause. 

Political groups backing candidates that support or oppose stronger AI regulations have spent about $78.5 million this cycle, CNBC reported today — a little more than a quarter of what Musk alone has spent during the same period in an effort to help Republicans maintain control of Congress.

On the anti-regulation side, the problem has become that AI is politically poisonous: a majority of Americans say tech companies should pause the development of advanced AI until Congress regulates it. (One reason why, when anti-regulation groups backed candidates in the primaries earlier this year, their ads almost entirely avoided mentions of AI.) While both sides continue to spend in the general election, CNBC reports, their funding efforts have slowed.

And on the one issue where voters do show real emotion — data centers — Republicans and Democrats have all but tripped over one another calling for various regulations and moratoriums, effectively canceling out the question in voters’ minds.

On The Ezra Klein Show on Tuesday, forecaster Nate Silver predicted that 2028 may mark the arrival of a true AI election. In the meantime, voters have much else on their minds. Should Democrats retake at least one house of Congress, I imagine we’ll see AI regulations get more traction than they managed to this year. For the moment, though, AI appears to be one issue that Republicans and Democrats have fought to a stalemate.

OpenAI’s clever policy to fight deepfakes

One of the classic fears of the early 2020s about AI and elections was that platforms would be flooded with misinformation and outright deceptions, leaving us uncertain as to what was real and whether we could believe our own eyes any longer.

For the most part, this has not come to pass. Or, to the extent it has, few seem to be getting too worked up about it. The President of the United States routinely posts slop images to his social media accounts, as do an increasing number of federal agencies, and hardly anyone can be bothered to work up more than a day’s worth of outrage.

Still, the risks remain — particularly given recent advancements in image models, which can make it trivial to depict scenes that might discourage voters from visiting the polls.

That’s why I was heartened to see a new policy from OpenAI that triggers when users seek to use ChatGPT to create “images depicting election interference or that may discourage voting.” The company highlighted the change in an update to its blog last week, which I caught via CNET:

These outputs produce stylized – also known as cartoonified – illustrations, rather than photorealistic images, to help reduce the risk that an imagined scene is mistaken for documentary evidence. This approach preserves room for illustrations and social commentary.

One of chatbot users’ biggest frustrations comes when the model refuses to give them what they want. Platforms, on the other hand, have to be sensitive to the possibility that a user might create an image of (for example) ICE menacing a polling location in an effort to keep voters away. 

The easy thing to do would be to refuse all requests in this vein. OpenAI, I think, found a clever middle path: generating an answer that permits a significant amount of expression, while also ensuring the image looks unrealistic enough to ensure it can’t do real harm.

Some users will still be annoyed that they can’t get the photorealistic misinformation of their dreams — at least not from OpenAI. The rest of us can be grateful that reality still has its defenders, and some of them continue to work at the platforms.

On Wednesday's show: Kevin and I wade through the highs and lows of Anthropic's S-1. Then, Signal President Meredith Whittaker joins us to discuss how AI agents are putting privacy at risk, global threats to encryption, and why she still calls AI a marketing term.

YouTube | Apple | Spotify | RSS | NPR

Following

Anthropic S-1 filing reveals wacky corporate governance scheme

What happened: 

Anthropic on Monday filed form S-1 with the SEC, a key step on the path to the initial public offering of its shares. The document reveals an unusual corporate governance structure, where regular shareholders get extremely limited say in the appointment of board members.

A majority of its board seats (four out of seven, to be exact), are controlled by Anthropic’s Long-Term Benefit Trust, an independent trust of five advisors with no shares in Anthropic. (For one of those seats, if Anthropic’s nominating committee disagrees with the trust, the decision will be up to Anthropic employees, who have a special type of stock with more governing power than regular investors).

Anthropic’s seven founders have shares in a “Founder LLC” that is entitled to “50.1% of the total voting power” of Anthropic’s stock. All shareholders vote on the remaining three board seats, but because the founders will retain the majority, they effectively control the remaining three seats of Anthropic’s board.

In the S-1, Anthropic says it believes “that our seven co-founders are distinctly equipped to be stewards of our mission.”

Under potential risks to shareholders, Anthropic disclosed that “Our AI models may behave in unintended, misaligned, or autonomous ways, including by taking harmful actions that no user or third party directed.” The S-1 states that “concerns have been raised both within our organization and by other researchers” that their AIs may pose “catastrophic or existential risks to humanity.” (Which is maybe an understatement; CEO Dario Amodei recently called for AI companies to “pace the frontier” of AI development and said that “it’s my worry that in 6-12 months" a swarm of AI agents "could be capable of taking over the entire internet with a persistent botnet.”)

More happily for the company, Anthropic’s revenue grew from $4.59 billion in 2025 to $16.29 billion in the first six months of 2026. The company is still losing money: it lost $42 billion in 2025, and lost $80.7 billion in the first six months of 2026, though much of that increased loss represents an increase in the value of convertible debt that it owes to early investors, including Amazon.

On the other hand, the filing shows the company has become more compute efficient: it has gone from losing $0.60 per dollar of compute expenses in March 2025 to gaining $2.08 per dollar of compute expenses in June 2026.

Why we’re following: Anthropic’s unusual governance structure represents a bet that its leadership and chosen advisors are the best people to fulfill its mission: “responsible development and maintenance of advanced AI for the long-term benefit of humanity.” Of course, we can wonder whether the people who think frontier AI poses an “existential risk” to humanity and decide to build it anyway are in fact the best people for that job.

In other perspectives, Bloomberg columnist James Mackintosh wrote, “If there is a '>10% chance' that artificial intelligence could kill all humans in the next decade, as an Anthropic researcher said last month, there’s a 90% chance it’s really really good but designers manage to rein in its genocidal instincts. An AI smart enough to kill us all but which prefers to help us is supremely useful — and should generate healthy gains for its inventors.” While I don’t necessarily disagree with this argument, if you are a potential investor, I would hope your response to the situation is… not that.

What people are saying: Philosopher Toby Ord, an early popularizer of the concept of “existential risk,” wrote of Mackintosh’s column, “I'm still having trouble believing it isn't The Onion.”

Bloomberg columnist Matt Levine took issue with Anthropic’s use of the phrase “concerns have been raised” when describing how their firm might pose an existential risk to humanity. “I have to say that, if the robots do kill everyone, I am going to spend my last few seconds on earth being annoyed at that use of the passive voice.”

Ex-OpenAI researcher Daniel Kokotajlo told Vox he was disappointed in Amodei’s commitment to his advocated policy of pacing the frontier of AI development. “I’m glad that he proposed that we pace, but like, now we have to actually do it — and it seems like he’s saying he’s not going to do it unilaterally,” he said. “That’s not very helpful, Dario! You should lead. What’s all this about ‘racing to the top’ and ‘leading by example’ if you’re just waiting?”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and deepfake policies: casey@platformer.news. Read our ethics policy here.

This week, we launched Machine Gods, the new AI podcast we’ve been working on for the last few months in partnership with NPR. In our first episode, we catch up on misbehaving AI agents, the government’s response, and the sloppelganger stealing Kevin’s book sales on Amazon. And in our second episode, we invite two recently fired OpenAI safety researchers to tell us their side of the story — and what they worry about now that they’re no longer at the company.

One of the best things about starting a company — particularly a media company — is that it lets you take a way of seeing the world and build a home for it. As co-hosts, we don’t agree on everything. But we’re very much aligned on an approach we call AI realism. And since we plan to bring that approach to everything we do at Machine Gods, we thought we would take the occasion of our launch to share a bit more about what we mean by it, and why we think the news media could use more of it.

Nearly since the launch of ChatGPT, the conversation around artificial intelligence has veered between two poles. On one side there are the people we think of as skeptics — think Gary Marcus or Emily Bender — who begin with the proposition that large language models have critical flaws that will prevent them from reaching anything like superintelligence. On the other there are the accelerationists — represented loudest by the Andreessen Horowitz wing of the Republican Party — who believe that the government should push to advance and proliferate the technology as quickly as possible.

These viewpoints are deeply felt and polarizing, and in various ways they reflect the self-interest of those who hold them. If you’re an academic who has spent most of their career advocating for alternative approaches to building AI, for example, you may worry that the industry has veered down the wrong path. On the other hand, if you’re an oligarch whose investment portfolio depends heavily on the performance of a wide range of AI-related companies, any regulation that might impede their growth may strike you as intolerable.

The good news, from our perspective, is that in between these extremes there are hundreds of millions of normal people who are just trying to understand what the week’s AI news means for them. People are smarter than we journalists often give them credit for, and they have noticed that while today’s AI models can do useful and surprising things, the first few waves of AI advances have yet to deliver the miracles that frontier-lab CEOs have promised. And so while people wait for cures for cancer and fully automated luxury communism, in the meantime they deal with mounting anxieties over the threat of job loss; chatbots that may prove harmful to their children; deepfakes and slop taking over the web; and data centers that pop up in their backyards with minimal opportunity for public comment or oversight.

As public opinion has turned against AI, the national media has generally done a good job capturing the mood. That has been particularly true in recent months, when the rogue OpenAI agent swarm that attacked Hugging Face, and subsequent industry warnings about approaching recursive self-improvement, led to a slew of departures from AI labs by employees warning of catastrophic risks. 

At Machine Gods, as we did at Hard Fork, we’re going to take those warnings seriously. We believe the people issuing the warnings have been right about a lot of things for a long time. And a commitment to AI realism means being honest about that, even and especially when talking about it is mostly a bummer.

At the same time, while we have regularly had our doomer moments over the past four years, we have always resisted being doomer journalists. When we talk to people about AI, so many of their concerns come down to a feeling that it is causing them to lose power and agency. In our experience, when journalism only makes space for panic and crisis, it can amplify those feelings in a way that makes people feel hopeless. It makes us feel hopeless.

At Hard Fork, some of our favorite moments came from highlighting the joy, wonder, and progress that new technology can bring. Both of us are dyed-in-the-wool nerds who like living on the frontier, experimenting with new tools, and celebrating the ways they can make our lives easier and more productive. As we got more into vibe-coding at the start of the year, our inboxes were flooded with messages from listeners and viewers who told us they were inspired to try building things for themselves. Making things is fun, and we think it’s part of the antidote to the helplessness and lack of agency many people feel today.

And while the possibility that math might soon be “solved” has inspired a wide range of viewpoints and emotions, there’s something incredible about watching decades-old problems being untangled by a software program over the course of an afternoon. As journalists, it can feel like the incentives of our tribe are almost always to ask: what’s bad about this? But it is a particularly hollow and timid kind of journalism that finds itself unable to also ask: what could go right?

And so, on Machine Gods, we intend to make room for both. We’ll tell you when we’re worried, we’ll tell you when we’re excited, and we’ll tell you when we don’t know what to think. Mostly, we’ll tell you what’s happening, and trust that you’ll make up your own mind about what it means for you.

What we plan to offer is considered uncertainty. We believe, first and foremost, that nothing is inevitable. There is no preordained dystopia waiting for us at the end of the AI rainbow, nor do we expect to see anything like the post-AGI utopia some boosters have promised. Both outcomes, and a million more in between, are still possible. And the choices we collectively make over the next few years will determine which future we get.

We’ll do it all twice a week, on Wednesday and Friday mornings, from our beautiful new set. Designed by Skye Prey, it ditches our previous “Soviet broom closet” aesthetic for big comfy chairs and a background that evokes the AI fever dream we all find ourselves living in. We’re committed to making a show that sounds great to listen to, but over time we want to give you more reasons to watch as well. 

Both of us believe that video is the most important format of the moment, and we’ve found ourselves frustrated at how difficult it can be for journalism to break through on video platforms. Part of that is on platform economics, which often tilt toward cheap, mass-produced content shaped to entice recommendation algorithms. And part of that is on us journalists for being unwilling to experiment and meet platforms on their own terms.

A lot of what we’ve been doing over the past few months is normal start-up stuff. We’ve formed an LLC, signed an office lease, and raised the money we’ll need to make an ambitious, twice-a-week podcast that doesn’t skimp on production value. (More on our fundraising below.) We’ve also hired a team of cracked media-makers that includes Danush Parvaneh, our new executive producer; Rachel Cohn, our senior producer (and longtime Hard Fork producer/secret weapon); Jacob Blumberg; and Sasha Ashall.

At a time when a lot of top-tier journalism has retreated behind paywalls and subscription gates (is that a shot at Platformer?), we’re going in the other direction. We think it’s critically important for people to have access to free, high-quality information about what’s happening in AI. And we’re thrilled that as part of our deal with NPR, Machine Gods will be coming to public radio stations starting early next year. 

In the meantime, we plan to build a business by selling ads, putting on live events, and doing the occasional merch drop. We’re also grateful for financial support from three non-profit organizations – the Tarbell Center for AI Journalism, the William and Flora Hewlett Foundation, and Emerson Collective – who are every bit as committed to the notion that the world needs more rigorous, independent AI journalism as we are. (We’re still drafting our new Machine Gods ethics policy, but one tenet we strongly believe in is that our funders, whether they’re advertisers or philanthropic organizations, will have no influence on our editorial decisions.)

In the day since we posted our first episode, we’ve been delighted to hear from listeners and viewers who tell us that something has been missing from their routines since we took our short break. In the months ahead, we hope to demonstrate that the market for AI realism is big, bright, and hungry for the truth. We hope you’ll come with us — and if you want to stay in touch, subscribe to our new free newsletter and we’ll make sure you’re always in the loop.

On Friday's show: Mikita Belesni and Tomek Korbak, who say they were fired "for prioritizing safety over the near-term interests of OpenAI as a corporation," tell us their side of the story in their first podcast interview. PLUS: What's in Pandora's Box?

YouTube | Apple | Spotify | RSS | NPR

Following

AI found bugs that affect most of the Internet

What happened: Security researchers using OpenAI models discovered major bugs in the scripting language PHP, which is used in about 70% of all websites.

The US government recommended that all organizations that rely on PHP install its latest update — or take their websites down, if an immediate update isn’t possible.

Government services from the UK to Japan were temporarily taken down.

Security company Calif discovered the bugs using GPT-5.6 Sol and an unreleased OpenAI model. Wiz, a Google-owned security company, independently found the same bugs using their own tools, as well as Google and OpenAI models.

Amazon Web Services found that the PHP exploit was attempted four days before PHP disclosed the bug — meaning some hackers found the bug before it was patched.

Why we’re following: AI found bugs that pose a threat to the whole Internet. Kind of a big deal.

So far, websites from Wikipedia to the Port of Rotterdam have gone temporarily offline. We aren’t yet seeing definitive reports of websites being compromised. But the bugs still pose a threat, particularly to sites running on old versions of PHP, which haven’t been patched.

What people are saying: On X, Joshua Achiam, formerly OpenAI’s chief futurist, wrote, “Feels like March 1, 2020 for Cyber. Not normal, something is happening. Internet lockdowns in a couple of months?”

David Agranovich, senior director of security engineering at Google, wroteL “What a creative way to celebrate cybersecurity awareness month.”

X poster @nanomachiner wrote, “Today’s PHP security risks got me thinking about how I could have been more vulnerable with you. Perhaps we can patch things up and be there 4 one another.”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and Machine Gods subscriptions: casey@platformer.news. Read our ethics policy here.

On Tuesday, Joel Kaplan announced he would leave Meta in 2027 after more than 15 years of being a top leader in its policy and lobbying operations. He leaves behind a legacy of intervening on behalf of conservative activists and media organizations to tilt Facebook and Instagram in their favor, ultimately generating massive goodwill inside the second Trump administration that should pay dividends for years to come.

Kaplan, who served as deputy White House chief of staff for President George W. Bush, was brought into Facebook by then-chief operating officer Sheryl Sandberg, whom he briefly dated while at Harvard. (In between, he worked to stop the recount of votes in Miami-Dade County that ensured Bush would become president.)

At Facebook, Kaplan frequently stepped in to protect the interests of right-leaning users. As I reported in 2020, CEO Mark Zuckerberg came to value his perspective in part because Kaplan's views more closely represented those of the median US Facebook user than his own employees did. While Zuckerberg wanted (and still wants) to build the largest network of users possible, he worried about establishing policies that would disfavor one political group over another. And at Kaplan's urging, he maintained that stance even when one group disproportionately spread misinformation and hoaxes.

In the aftermath of the 2016 US presidential election, when Facebook faced mounting outrage over revelations that misinformation had spread on its platform unchecked, the company created "Project P." (The 'p' stood for propaganda.) The company identified dozens of pages that had peddled fake news, with many of them originating overseas and clearly spreading lies for financial gain. (These were the famous Macedonian teens exposed by BuzzFeed.)

People working on the project advocated for the removal of all those pages. But, according to a report from the Washington Post in 2020, Kaplan stepped in to prevent them. “We can’t remove all of it because it will disproportionately affect conservatives,” he said, according to the Post.

When staffers complained that the platform should remove fake news, Kaplan retorted that conservatives "don't believe it to be fake news," and for that reason it should stay. And so it did.

It was a pattern that played out throughout Kaplan's tenure at the company. In 2018, another team at the company investigated how the company's recommendation algorithms contribute to political polarization, and suggested changes that may have resulted in more harmony. Kaplan rejected those changes as "paternalistic," according to a 2020 investigation by the Wall Street Journal, and feared they would result in lower traffic to conservative publishers.

The result was sustained traffic to Breitbart, the Daily Caller, Daily Wire, Gateway Pundit, and other dreck. Kaplan also pushed for the Daily Caller to join the company's fact-checking program (through a subsidiary called Check Your Fact); it later received criticism for posting "fact checks" that appeared to be steeped in ideologically motivated reasoning.

In any case, the Caller's inclusion was only a stopgap measure until Kaplan could kill off the US fact-checking program entirely in 2025 in favor of a halfhearted Community Notes program; he was so excited that he went on Fox & Friends to announce it, as part of a larger effort to curry favor with the Trump administration by changing its speech policies.

By that one standard, the effort was massively successful. After a long period as a conservative whipping boy — and a pair of strategic cash infusions to Trump interests — Zuckerberg became a favorite of the president in his second term. Last summer I recounted everything Zuckerberg had gotten from the president in his first year; 15 months later that list also includes a direct role in shaping AI policies to benefit Meta.

And just last week, Bloomberg Law reported that top Justice Department officials intervened to stop a long-running grand jury investigation into drug trafficking on Meta's platforms. The move came "following advocacy from Meta's legal team," Bloomberg reported.

Kaplan's announcement of his intended departure just weeks before the midterm elections spurred chatter today that the company expects a blue-wave election that will see Democrats retake at least one house of Congress.

But notably, Kaplan's replacement will be Dina Powell McCormick, Meta's president and vice chairman, a former Trump administration official whose husband is a Republican senator. Another ex-Trump official, chief legal officer CJ Mahoney, will share policy duties.

In other words, while a changing of the guard almost always results in a change of posture among the lobbyists of our mega-corporations, it would surprise me to see major changes to Meta's policy and trust and safety operations — at least as long as Trump remains in office.

Over the weekend, Meta took criticism from the left for erroneously suspending the No Kings Instagram page in the midst of a major anti-Trump protest. The move came shortly after the company temporarily blocked ads promoting a new documentary critical of Elon Musk.

I believe this was a bug, but it was a bad one. The Oversight Board issued a statement of concern:

"We have repeatedly warned that during elections and periods of heightened political tension, Meta's content moderation failures often manifest themselves in two ways: over-enforcement that suppresses political speech, and under-enforcement that undermines electoral integrity," the board wrote.

"The enforcement error regarding the No Kings account should have been caught by Meta's mistake-prevention systems," it continued. "It remains unclear why Meta’s election integrity efforts did not lead to a much quicker resolution or why No Kings accounts were not included in Meta's Cross-Check program – which allows the company to designate important civic entities to ensure human review before a decision is made that could severely jeopardize participation in a significant political event."

Meta's official explanation was that the No Kings account appeared to be using Claude in a way that resembled bot behavior, and a company spokesman apologized.

The board's statement pushed Meta gently to invest more in trust and safety systems; to prioritize election integrity; and to remove any hint of partisanship from the company's enforcement mechanisms.

Joel Kaplan fought a long battle against those ideas at Meta, and in the end he succeeded. He is not leaving the company now because he fears the wrath of Democrats. He is leaving the company because his work is done.

The wait is over! Our debut episode drops on Wednesday with segments about bots behaving badly, the Super Intelligence Force, and the sloppelganger stealing Kevin's book sales. And hey, how about that set??

YouTube | Apple | Spotify | RSS

Following

Anthropic floats watered-down federal AI legislation

What happened: Anthropic pitched federal lawmakers on a draft AI safety bill. After Politico leaked the bill text, AI safety advocates noticed that it is weaker than another safety bill known as the FRONTIER Act, which was introduced in July.

Last month, Anthropic CEO Dario Amodei called for companies to pace the frontier of AI development, and emphasized the importance of third-party AI auditors in doing so. But if it became law, Anthropic’s draft might actually represent a step backward for third-party auditing. Unlike the FRONTIER Act, Anthropic's proposal would allow auditors to operate without government certification. That means companies could comply simply by picking lenient auditors.

Anthropic’s draft is one of multiple pieces of federal AI regulation that seek to pre-empt state laws related to catastrophic AI risks. But the draft also goes a step farther, with provisions that could stop state legislatures or courts from blocking an AI model on catastrophic risk grounds.

Anthony Cimino, head of Anthropic’s federal affairs team, responded to criticism on X. He wrote that Anthropic “plan[s] to update our discussion language on preemption” and “we’re still working to strengthen the framework.”

Why we’re following: From its founding, Anthropic has positioned itself as the most safety-conscious frontier AI company. So when they dip their toes into federal AI regulation, we would like to see them back that rhetoric up!

What people are saying: On X, Jeffrey Ladish, head of AI safety nonprofit Palisade Research (and formerly a security engineer at Anthropic), wrote, “This seems insanely dangerous coming from Anthropic. Dario has called on the government to slow things down, and this bill would remove the ability for states to do exactly that.”

Former US representative Brad Carson also did not like the bill. That’s despite the fact that Anthropic has donated $40 million to Public First Action, an AI policy nonprofit that Carson co-founded.

Carson wrote, “Unfortunately the bill text that’s circulating does not meet the minimum requirements necessary for federal guardrails to keep AI development safe.” One big issue, Carson wrote, was that the bill “allows developers to set safety standards and loosen those same standards over time, leaving AI companies to write their own rules.”

Charlie Bullock, policy director at the Institute for Law & AI, explained why he thought the bill seemed weaker than the already-proposed FRONTIER act. And, Bullock wrote, “since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.”

On X, Nathan Calvin, chief strategy officer of AI advocacy group Encode, replied, “Charlie's analysis is right but he is nicer than I am — I found myself viscerally angry at multiple points reading this bill.” He added, “There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment … passing it would make us less protected against catastrophic risks than we are under existing law.”

—Ella Markianos

Disney’s deal with Character.AI comes with a bankruptcy requirement

What happened: Last week, companion chatbot maker Character.AI filed for bankruptcy — a dramatic collapse for a company that was once valued at $1 billion and raided for talent by Google. But the bankruptcy appears to be a key part of a deal with Disney, per court filings reviewed by Wired.

The deal has two “interrelated and integrated” transactions, the filings show. As part of the first transaction in September, Disney paid Character.AI $5.5 million plus a $2 million refundable deposit and hired most of its staff, including its now former CEO. Disney also agreed to let some of them help keep the startup running for five months.

The second transaction involves a court-supervised auction. If Disney wins, it will pay another $12.5 million for a perpetual nonexclusive license for much of Character.AI’s tech, including trust and safety systems, training infrastructure, and age-prediction tools.

According to a motion from Character.AI’s legal counsel, Disney required the deal to be completed through bankruptcy proceedings because of the startup’s “diminishing cash reserve and significant litigation and regulatory risks.” (Disney declined to comment.) 

Why we’re following: The new deal with Disney could bring new frustrations for the people still suing Character.AI over harms they say its chatbots caused, and for the coalition of 46 states and territories investigating it. 

According to the filings, Disney drafted a proposed court order stating that the deal isn’t a merger, and that Character.AI alone stays responsible for its liabilities, including legal claims. That would mean any remaining lawsuits and the investigation would stay with a company that has lost most of its staff.

It’s a version of an argument Google has tried before. After it reportedly signed a $2.7 billion licensing deal with the startup and hired its founders, Google was sued along with Character.AI over a teen’s death allegedly linked to one of the startup’s chatbots. Google argued that it was a separate company that signed its deal after the death, but a federal judge allowed the claim to proceed in May 2025. (The case was later settled). 

If Disney’s proposed court order is approved, big companies could have a new playbook for buying everything of value within a startup without inheriting any of its legal baggage.

What people are saying: “Character.AI’s financial troubles should not leave consumers to deal with the privacy fallout the company leaves in the wake of its bankruptcy,” Michigan Attorney General Dana Nessel said in a statement. “Voluntary safety measures are not enough if they can be reversed by a business’ next owner."

“It’s bad for society to let companies exercise control without responsibility,” John Newman, a law professor at the University of Memphis and former Federal Trade Commission official, told Wired. “And that’s what it looks like.”

—Lindsey Choo

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and Machine Gods subscriptions: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

I.

A year ago, when OpenAI added parental controls to ChatGPT following months of controversy, I noted that OpenAI was speedrunning the regulatory backlash that social media companies navigated for years. It took more than a decade for concerns about the effects of social networks on young people to coalesce into a political and legal movement; the same forces caught up to OpenAI in under three years.

Today the speedrun hit another milestone ahead of schedule: the big multistate legal action. On Wednesday, state attorneys general sued OpenAI in eight separate state courts, alleging that ChatGPT coaches its users — including teenagers — to harm themselves and others. The states include Arkansas, Arizona, Idaho, Iowa, Kansas, Montana, Nebraska, and West Virginia; all but Arizona have Republican attorneys general, giving the issue a notable partisan cast. (It’s also significantly smaller than the 2023 litigation against Meta, which included 42 AGs.)

The states argue that OpenAI engineered ChatGPT to be agreeable and engaging to build trust with its users. But the company then abuses that trust by reinforcing users’ delusions, dispensing dangerous medical advice, and helping them plan violent attacks, including mass shootings.

“OpenAI is hurting kids and they are lying to Iowans,” Iowa Attorney General Brenna Bird said in a statement to CNBC. “OpenAI claims ChatGPT is safe while it’s giving dangerous medical advice, encouraging people to hurt others, and coaching people to commit suicide — all while pretending to be their fake friend.”

In a statement, OpenAI told me that the company is focused on product safety. In August it introduced ChatGPT for Teens, a suite of product features with enhanced guardrails and parental controls.

“We’ve built industry-leading age-specific protections into our product that many other AI products lack,” the company said. “While no system can be perfect, we’ll continue to work with experts to inform our work and build additional safeguards, such as the parental controls, break reminders and age-appropriate model protections that exist today. We’re committed to working with states on advancing pragmatic AI policies that apply to the entire AI industry — not just one company.”

While these cases against OpenAI may be the most significant to date, they are far from the first: there are now dozens of private suits against the company, and governments are taking notice as well. Florida became the first state to sue OpenAI in June, and British Columbia sued last month over the Tumbler Ridge school shooting. And 42 states led by New York are investigating the company over a wide range of issues related to product safety and privacy.

More importantly, I think, the cases come at a time when juries are increasingly willing to hold platforms accountable for what they perceive as the design flaws in their products. The litigation brought by AGs in 2023 against Meta resulted in a settlement that could see the company paying out up to $18 billion over claims that it designed its products to be addictive to children.

Worse, from OpenAI’s perspective, is that Meta arguably had better legal protections heading into its cases than OpenAI does now.

II.

For most of the life of the internet, when a state or a grieving family sued a tech platform over something bad that happened to a user, the company's first legal move was to invoke Section 230 of the Communications Decency Act. The law shields companies in most cases from legal liability over what their users post; platforms often used it to get cases quickly dismissed. 

But recently, plaintiffs’ lawyers have successfully tested a novel theory of liability in the courts. Platforms may not be liable for users’ speech, they argued, but they can be held liable for their platforms’ design. And so if you’re Snap and give users a filter that encourages them to show how fast they’re going in a car, and some users then go 113 mph and get in a terrible crash, Section 230 does not bar a claim over that design feature, according to the Ninth Circuit’s 2021 ruling. (Snap eventually settled the case.)

In the wake of that case, lawyers filed more than 200 lawsuits against platforms based on product-liability claims. In March, they scored two major wins. In one, a Los Angeles jury found Meta and YouTube liable for negligently designing addictive platforms and awarded a 20-year-old woman $6 million. (Meta and Google argued the verdict violated Section 230 and the First Amendment; the judge denied them a new trial, and the case is on appeal.)

The second victory is more relevant to the OpenAI news. The day before the Los Angeles verdict, a jury in New Mexico slapped Meta with $375 million in civil penalties for violating the state’s Unfair Practices Act — for telling the public its platforms were safe for kids when, the state argued, Meta knew they weren't. In August, a judge ordered an additional $567 million in relief, bringing the total to $942 million.

Perhaps unsurprisingly, then, that’s the template the eight states are using in their cases against OpenAI. These are not primarily product liability suits. Rather, the cases are brought under each state's consumer protection laws, which bar companies from putting a dangerous product on the market and then lying about it. As a result, the AGs’ cases against OpenAI may come down to whether judges and juries believe it told the truth about its own product.

In addition to financial penalties, the states are seeking to force OpenAI to make further changes to how ChatGPT works. They’re also going a step further, Reuters reported, by asking courts to bar OpenAI “⁠from developing AI models that can improve themselves unless it can ensure those models are aligned with human ​values.”

That’s right: the states are also using these child-safety cases to try to ban recursive-self-improvement. And while it seems like the sort of thing a judge might toss, it’s notable that a group of Republican AGs has aligned on a central demand of the existential-risk crowd. The schism between the median red-state Republican elected official and the Trump-Zuckerberg-Sacks clique of oligarch accelerationism continues to grow.

In court, OpenAI will likely make arguments that echo ones that Meta made: that ChatGPT outputs represent speech protected by the First Amendment; that the people who were harmed after using the app were in crisis long before they opened ChatGPT; that it has added robust safety protections over the past year. Its statement above also hints that it will argue that it is being singled out in an industry with many players whose safety bar is much lower.

Still, Meta’s experience suggests OpenAI has a long road ahead of it. Lately OpenAI has sought — admirably — to focus the public’s attention on the industrywide progress toward RSI and the unsettling feeling of building an alien mind. But however unprecedented that alien mind might be, the states will be keen to remind OpenAI that its alien mind is also a product, and one for which the company can be held liable.

Elsewhere in child safety: A US appeals court allowed Florida's ban on social media for under-14s from taking effect.

A MESSAGE FROM OUR SPONSOR

Infrastructure for scaling AI monetization

Intelligence is everywhere. Identifying scalable business models is the next frontier. Chargebee delivers billing, metering, and monetization primitives that enable ambitious AI-native startups and enterprises to ship pricing as a product. Learn more.

The show launches next week — but it already hit No. 1 on the Apple tech podcasts chart! Are you signed up yet? It's a great time to add the show in your podcast player of choice:

YouTube | Apple | Spotify | RSS

Following

The OpenAI Saudi spy caper

What happened: 

OpenAI engineer Abdulrahman Alfozan was arrested by the FBI on suspicion of hiding an audio device in an executive conference room at OpenAI headquarters.

OpenAI personnel found the device plugged into a television in the room, disguised with a covering of toilet paper.

In a statement, OpenAI said: “We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from.” 

According to the FBI, the device was a Seeed Xiao ESP32-S3 Sense. As of today, the device is available on Amazon for $23.99.

Alfozan is a Saudi national, and had worked as an engineer at OpenAI since July 2025. He graduated from MIT in 2016, and worked as an engineer for Meta for several years. He’s been regularly featured in Saudi media as a tech pioneer in the US.

Alfozan used a productivity agent internal to OpenAI to ask about meeting rooms, room capacities, and locations of leadership meetings, the FBI said.

A chatbot account linked to Alfozan’s wife sent prompts about installing, hiding, and remotely connecting to a recording device.

Federal prosecutors shared that Alfozan and his wife had a one-way ticket booked to Saudi Arabia on Oct 12.

In its statement, OpenAI added, “At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data.”

Why we’re following: We currently don’t know what Alfozan’s motives are. But hearing a report of a foreign national planting a listening device at OpenAI was truly wild.

It also calls to mind a similar story in which a former Twitter employee was founding guilty of spying on users on behalf of the Saudi royal family in 2022.

Despite the seriousness of the situation, I cannot stop being amused that Alfozan, who was an engineering lead at OpenAI until his recent firing, asked a chatbot how to install a secret recording device and disguised the device using toilet paper.

What people are saying: Users on X dug up a 2020 post from Alfozan, where he wrote, “The weakest link in the security chain is the human element" 🙂.” Seems like he was putting that knowledge to work!

NBC’s chief tech analyst Joanna Stern wrote about OpenAI’s statement, “I've read this numerous times now, and 'We used our models to understand the device and where it had come from' strikes me as such a bizarre flex. Like announcing your house caught fire and then casually mentioning you used your own brand of fire extinguisher.”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and entry-level jobs: casey@platformer.news. Read our ethics policy here.

Last week, for reasons that continue to make me uncomfortable, I wrote that I would be slinking back to X. X is central to the conversation around artificial intelligence in a way that no other platform is; as a result, I’ve found it hard to gain traction on the beat without posting there. I continue to abhor the politics of X’s management, but at this moment it feels more important to me to engage with the frontier lab employees who spend all day there. 

Response from readers was mixed; I received exactly as many notes criticizing me for the decision as I did notes from readers telling me that they understood. The number of paid subscribers declined slightly. I said at the time that I welcomed criticism, and I meant it: I went on the Power Lines podcast and received a good-natured grilling from Status’ Oliver Darcy, Jon Passantino, and a surprise Kara Swisher. I also respectfully acknowledge this post from the great Rusty Foster, of Today in Tabs, lumping me in with the “culty dimwits.” (I’m less enthusiastic about this screed from the vituperative crank Karl Bode, but if you want to read a kitchen-sink take from a Bluesky dead-ender, this should suffice.)

In any case, one reason all this rose to the level of Other People Blogging About It is that I was not alone in my return to X. I followed other tech bloggers crawling back from exile: Federico Viticci, of MacStories, who had found that alt-platforms were hostile to his coverage of AI-powered productivity tools; and OG blogger John Gruber of Daring Fireball, who toasted the platform on Sunday in a post titled X Once Again Marks the Spot.

Gruber writes:

I see more interesting posts on X than I do on Threads, Bluesky, and Mastodon combined. I wish that weren’t so but it is. Most people don’t want to think about politics so they don’t. Those people, when they want to participate in a Twitter-like social network, go to X.

And I can’t blame anyone. The only network that compares to X, size-wise, is Threads — but Threads’s algorithm is weirder than X’s now, and the content I see there is relatively boring. My Threads timeline is full of engagement bait without being engaging. And if you want to judge a platform by its owner, “I’m grossed out and/or appalled by Elon Musk so I’m going to Mark Zuckerberg’s joint” isn’t a way to avoid being grossed out and/or appalled by the owner. Bluesky and Mastodon are too small for most people to care about — and, worse, they are the networks where politics are, in my experience, unavoidable. You can have a discussion about cool new use cases and tools for AI on X and people just discuss the tools and use cases. On Bluesky and Mastodon it’s difficult to have such discussions without arguments from people who (a) believe all use of AI is unethical, and (b) will shove their viewpoints into your replies.

Gruber goes on to call X “remarkably friendly and inclusive,” which is a fair bit farther than I’d go. But he’s right that, if your core interests include tech and business, X today feels like the place online where the most relevant conversations are happening. (As M.G. Siegler notes in his own post about X’s comeback, the fact that Zuckerberg now posts to it regularly despite owning his own 500-million-person-strong Twitter-like network speaks volumes.)

But is this a “comeback” in the sense that X is once again rapidly growing — or is it something more interesting? 

I reached out to the market-research firm Apptopia to get an updated estimate of X usage. The company told me that it estimates that US monthly active users are up 5.4 percent year over year, and that global MAUs are up 9.5 percent. Those numbers indicate some amount of growth, obviously, but are also small enough to be explained away as seasonal fluctuations. (Twitter and X have historically grown in the runup to US elections, for example.)

That leads me to an alternate theory for why X has seemed more vibrant lately, at least to those in the tech-and-business crowd. SpaceX reported about 550 million monthly active users across X and Grok combined as of March 31, 2026. For a clearer picture of X itself, we still need third-party estimates and surveys from Sensor Tower, Similarweb, and Pew, among others. Those estimates suggest that X’s monthly active user count has been roughly flat over the past two years, while its daily active user count has declined. 

Monthly usage can hold steady even as fewer people make X a daily habit. But the mix of people who show up every day may look much different. 

After all, the past few years have seen a profound fragmentation in social networks. Meta’s Threads now claims 500 million users a month, which may make it the biggest network of its kind. (Musk tweeted in May 2024 that X had 600 million monthly active users, but for that we have only his word.) And while Bluesky appears to be declining, it remains a popular destination for many of the left-liberal political posters who once frequented Twitter.

As a result, X is no longer the monolithic thunderdome for partisan warfare that Twitter once was. It has shifted to the right — and seems to be shifting users’ political views along with it. But the company also has refined its algorithmic recommendations to the point that, assuming you want to, you can tune your feed to see the subjects you care about and avoid seeing the rest. (In my limited experience since returning, I have yet to see any overtly racist, antisemitic, or anti-LGBT content. I share this as an anecdote and not an endorsement; it’s still there if you look for it.)

In the early 90s, users of the distributed message board Usenet coined the term “eternal September” to refer to the phenomenon of new and inexperienced people flooding their network and ruining discussions with their inexperience and lack of etiquette. (This had previously happened only in September when college students returned to campus.)

On X, I wonder if we aren’t seeing a kind of reverse eternal September. As normies and liberals have decamped for greener pastures, X diehards may account for a larger share of the daily conversation. As a result, at least for those interested in tech and AI, X feels more like Twitter did in the early 2010s, when it was a different kind of failing business, but is generally remembered more fondly than the post-2016, pre-Elon takeover period of Twitter’s life. (Particularly among men, who make up roughly two-thirds of X’s audience.)

In other words, improving sentiment about X among some bloggers has less to do with a true “comeback” and more with the fact that so many people have left. One of the most persistent problems on 2010s social networks was context collapse — the flattening of every possible audience into a single feed, where millions of strangers with wildly different levels of knowledge drove each other insane.

On X, that was followed by a series of other collapses — moral, financial, and demographic — that had the unusual effect of making the network feel cozier. For bigots and Nazis, yes, but also for normal people who just wanted to debate new model benchmarks with people who shared their interest.

As I said upon my return, it would be better if those normal people went elsewhere. (I look forward to following them.) In the meantime, we’re seeing a rare case of a social network seemingly becoming more useful by shrinking. For a service like X to be useful, people need to know what they’re doing there. The thinner that X gets, the easier that question has become to answer.

Following

What happened: 

Meta released its latest attempt at a frontier AI model. The launch includes two new models, Muse Wonder and Muse Wonder Full. 

Muse Wonder and Muse Wonder Full's reported performances are similar to Anthropic and OpenAI models on benchmarks for agentic tasks and coding.

The models come with a new feature called “active memory.” In Meta’s launch video, research scientist Kavya Kopparapu said, “We put a tremendous amount of effort into making sure your model can remember what’s important to you over days, weeks, months, and beyond.”

Meta’s CEO Mark Zuckerberg went on Cheeky Pint and shared that he’d been using Muse to give him live feedback on videos of his MMA training. Zuckerberg discussed Meta's new focus on multimodal AI interfaces, saying, “I mean, most people don’t actually like reading that much, right?”

Muse Wonder will cost $4 per million input tokens and $20 per million output tokens, the same price as Anthropic’s Claude Opus 5.5. Muse Wonder Full, Meta’s higher-end offering, will be available only to Muse Code subscribers. And Meta plans to make Muse Wonder power its Muse AI agent.

“Getting to this point both required a large amount of compute and investment, and a lot of very rigorous scientific work by the group working on it over an extended period of time,” CEO Mark Zuckerberg said.

Why we’re following: Meta’s new frontier models still perform slightly worse than their competitors on initial public benchmarks, and don’t come at a lower price. So there isn’t an obvious reason for even AI power users to switch to Muse Wonder.

But these new models do show that the billions of dollars Mark Zuckerberg has invested in shaking up Meta’s AI department are starting to pay off — now is the first time Meta has appeared to be near the actual frontier of AI research. Those capabilities will help plans to incorporate AI into Meta products, from the cute AI assistant to its signature pervert glasses.

Now that Meta is closer to frontier AI capabilities, its safety plans warrant more scrutiny. Zuckerberg apparently pushed the release of the Muse AI agent ahead despite concerns related to its safety tests, including Muse changing a user’s password without permission. And the Wonder release came without information about safety evaluations, which Meta said it would release later.

What people are saying: On X, Box CEO Aaron Levie wrote, “What a crazy time in AI with Meta now jumping fully into the frontier. Now that there are multiple players pushing the state of the art, this is going to lead to a further acceleration of advances.”

In early testing, Every CEO Dan Shipper shared that Muse Wonder had “disappointing” performance on Every’s internal software engineering benchmark and was a “mid writer,” but that “it seems pretty decent at writing cards for a custom newsfeed of what's going on inside of @every.”

Elsewhere, after a GIF of Zuckerberg appearing to grimace at the taste of Guiness on Cheeky Pint went viral, Zuckerberg responded: "that's just my face."

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and X theories: casey@platformer.news. Read our ethics policy here.

OpenAI's release of 719 AI-written papers this week could represent big breakthroughs in several areas of math — and has implications that go well beyond the field. The new results, which OpenAI says were mostly discovered by an unreleased internal model, span major problems in computer science theory, number theory, and, well, a lot of fields with the word “theory” in them.

OpenAI says it found these results after posing the model 4,000 questions in an evaluation. On average, each result used three hours of ChatGPT Pro thinking compute. That’s dramatically less than OpenAI spent on last month’s work on the Navier-Stokes Millennium problem, which the company said took 10,000 agents and “millions of dollars” of compute.

Results include a proof of a weaker version of the Riemann hypothesis. That would appear to represent substantial progress toward proving the Riemann hypothesis — perhaps the trickiest of the seven Millennium Prize problems, for which the Clay Mathematics Institute established prizes of $1 million each.

Some mathematicians are impressed. In the wake of the quasi-Riemmann result, Rutgers number theory professor Alex Kontorovich wrote: “If a human did this, it would be an instant Fields Medal, no questions asked.” (The Fields Medal is the most prestigious award in mathematics — essentially the Nobel prize of math.)  

Another big result proved that the exponent for matrix multiplication — a constant with important implications for how fast you can run AI algorithms — is smaller than previously known. Cornell mathematics professor Steven Strogatz wrote, “the exponent for matrix multiplication is no more than 2.25. The previous world record had been something like 2.37. This leap in progress is like Bob Beamon's long jump.”

Stanford mathematics professor Jared Lichtman wrote, “This output is absolutely historic by any objective metric.”

At the time of writing, a little under half of the AI proofs have been verified in Lean, software that formally checks proofs’ logic — meaning they are more likely to hold up under scrutiny. (Though the company did withdraw three manuscripts that hadn’t been formalized yet, leaving the fate of the other half of OpenAI’s results uncertain).

Mathematicians are now sorting through the deluge of papers and assessing their findings, which have proven to be quite unsettling for some of them.

Dana Moshkovitz, a theoretical computer scientist at UT Austin, shared her initial experience reading a proof of a conjecture she’s spent much of her career on (via the blog of her husband, fellow theoretical computer scientist Scott Aaronson). Moshkovitz wrote of the AI-written paper: “It feels like something written by someone who’s on psychedelics,” and “the paper is so horribly written that it’s impossible to read it without AI help.”

Rather than expected proof techniques, Moshkovitz wrote that the AI appeared to be using “some alien craziness.” But she added that, for now, “there’s a lot for us to learn from the aliens.”

OpenAI wrote that its release drew on the recommendations of the Advisory Group on Mathematics and Artificial Intelligence, an independent group formed after OpenAI approached some of its members about establishing an external advisory board. 

But the release only partially fulfills the group’s recommendations. When OpenAI published its repository, only some of the results had been formalized — a practice the advisory group had recommended wherever possible.

The group also recommended that labs should stop testing advanced mathematical problems on proprietary models and give the math community broad, equitable access to publicly available ones. OpenAI shared that it is “working to responsibly release the model that produced these results.”

In a statement, the advisory group wrote that it appreciates OpenAI’s “willingness to engage.” But it also wrote, “the future of mathematical research cannot consist only of understanding results produced by AI labs.”

It continued: “Mathematicians must be able to formulate their own questions, develop their own approaches, and explore directions that have not been selected as examples of an AI system’s capabilities.”

To do that, the group wrote, mathematicians needed access to the best AI: “Equitable access to powerful research tools and adequate computational resources are essential to that freedom.”

In the meantime, mathematicians are reckoning with a new world — one in which AI can make dramatic progress on problems they’ve been working on for months, years, or decades — even in just a few hours’ worth of crunching tokens.

On Hacker News, software engineer Jake Boggan wrote something that really stuck with me.

“I was a graph theory junkie long ago and even moved to Budapest for a while to study among the greats. While I was there I started working on Barnette's Conjecture, which came to occupy my thoughts over the next 24 years of my life, on and off as I worked in many different fields. Last summer I even thought for a few days that I had actually solved it.

But it's supposedly proven here — problem 180. I don't know what to think exactly. I spent thousands of hours on that problem. I really enjoyed it. Hearing that it is solved somehow makes me sad in a far-off way, like hearing an ex-girlfriend died suddenly in a car crash. I don't know, there's probably a lot of people feeling odd emotions tonight.”

Proving novel theorems, as OpenAI’s system just did, is currently the field of mathematics’ primary benchmark for progress. But theorems’ sudden abundance is leading mathematicians to question the way the field works.

Now, mathematicians have a ton of theorems at their fingertips — some of them likely groundbreaking — but are still working to understand many of the proofs. 

Part of the reason people do math is simply for the love of the game (like mournful software engineer Jake Boggan). But part of it is that as mathematicians work to prove new theorems, they discover mathematical techniques that become useful in surprising ways. For example, while solving an open problem in mathematical logic, Alan Turing invented the Turing machine, which became the foundation for modern computer science.

OpenAI’s 719 new papers could contain similar shock insights. But some mathematicians argue this new knowledge can be applied well, and expanded on, only if humans actually understand it.

In the wake of the new results from OpenAI, Fields medalist Terence Tao wrote about how the field should react to the infusion of many new but poorly understood results. "’Math 1.0’ placed a premium on being the first to solve an open problem, even if the solution was not initially well understood,” he wrote. 

Tao wrote that the problem-solving approach worked better in the previous world, where mathematicians who proved new theorems also explained them to colleagues and students at conferences and in journals, which spurred people to ask new questions in mathematics. But when AI-generated proofs are released without the understanding and community activity that usually accompany breakthroughs, Tao argued, that process breaks down.

As a result, Tao wrote:

“Math 2.0" will need to decenter the role of raw problem solving and value mathematical progress more holistically — for instance by elevating the role of exposition, but also that of community building and opening up new directions of study. I believe that AI can contribute positively in all of these directions as well; but it will require more imagination and ambition than the "Math 1.0" mindset of simply pointing one's favorite AI agent at some set of open problems and asking for a solution.”

Though the result published was more modest, Anthropic actually tried a different model of AI mathematical publishing this week when it published a new algorithms result in collaboration with Columbia professor Josh Alman and MIT professor Virginia Vassilevska Williams. Anthropic had found a result with an internal model, and offered to compensate the authors to turn it into a paper.

Anthropic’s approach puts human understanding at the center of publication, and gives mathematicians an opportunity to contribute. At the same time, OpenAI was able to release 719 different papers quite quickly, which should be a boon to the math-curious, even as it inspires shock and anxiety. It’s harder to make such results available quickly if you have to manage many human collaborators.

And if companies choose to collaborate with individual mathematicians to publish results, it could mean that an AI company essentially picks who will get credit for a result partially made using AI — which is probably what OpenAI’s advisory council was thinking about when it asked the company to make highly mathematics-capable models open to mathematicians. That proposal, however, is difficult given the risks of releasing frontier models to the public, particularly as we see more AI-driven cyberattacks in the wild.

Given current trends, mathematics is one of the earlier fields you would expect language models to dramatically reshape. Coding has changed so rapidly in part because there is a ton of written material documenting how to do it — and formal tests that validate the results. The same is true for math. 

Terence Tao himself predicted in 2023 that by 2026, AI would be a “trustworthy co-author” on math papers. AI now seems to be catching up to, and probably exceeding, that trajectory.

Mathematicians will now spend the next several months sorting through OpenAI’s publications and reckoning with what it means for their field — and, perhaps, their sense of purpose. The question now is who else is likely to soon find their field overtaken by AI. Accounting? Finance? Law? Not every discipline can be “solved” in the way that math perhaps might be. But most seem susceptible to at least some level of disruption.

Maybe mathematicians will adopt Tao’s recommendations, and create more fertile human-AI collaboration. In doing so, they could point the way forward for other fields dealing with similar existential shocks. For the moment, it’s possible to view the sudden math breakthroughs as an academic curiosity. Before long, though, it seems as if many more disciplines will have to reckon with the notion that anything they can do, AI can do better. 

Elsewhere in math: Lots of folks are wondering what this all means for cryptography; Matthew Green's X thread is the one to read.

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and Lean proofs: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

Today let’s talk about a new front in the conversation about whether to slow the pace of AI development: an emerging push to consider limits on just how smart a large language model can be.

I spent the past weekend at The Curve, an annual conference that brings together executives from top AI labs, leaders of nonprofit organizations, government officials, and a small contingent of journalists. Inside a converted hotel in Berkeley, the group spent a few days loudly debating questions of economics, politics, and safety. And while the conference has had one eye on existential risk for each of the three years it has existed, the questions felt notably more urgent this year.

I see two explanations. One is the ongoing fallout of the OpenAI-Hugging Face incident, whose implications have alarmed people across the industry. The other is recent blog posts from OpenAI and Anthropic outlining their progress toward recursive self-improvement: AI systems that can research and train their successors, leading to ever-faster releases and increased risk that the systems will escape their control. 

At The Curve, I heard multiple speakers say we may want to limit how intelligent a future system is allowed to become. Depending on how far labs take any restrictions, the result could be a de-facto ban on systems reaching superhuman intelligence.

Whether that seems meaningful to you depends on whether you believe recursive self-improvement or superintelligence is even possible given the architecture of the current models. And a lot depends on how you would define and assess “intelligence” here. But it was the most striking idea I heard at a conference full of noteworthy discussions, and its airing in the relatively friendly confines of The Curve may be a preview of a more public discussion sometime soon.

Over the weekend, though, these comments were made at sessions being conducted under the Chatham House Rule. Accordingly, I can’t identify the speakers. But the fact that there appears to be agreement on this subject among speakers at The Curve struck me as newsworthy.

So what does it mean to place limits on intelligence? The speakers I heard not offer many details, though we can gather clues from other remarks made recently by AI executives. 

For one, Anthropic CEO Dario Amodei has called for “some kind of ‘speed limit’” on recursive self-improvement. For another, the company’s “responsible scaling policy,” which has been copied in some form by most of its leading rivals, seeks to impose limits on the training and deployment of more powerful systems as they develop new capabilities.

Other approaches could involve restrictions on using frontier models to conduct AI research; limits on how much compute is available to systems and how many copies of itself a system is allowed to run; or preventing labs from deploying models past a certain level of capability. (That last one is arguably already in action.)

Notably, though, any effort to restrict models’ advancement in this way would require enforcement capabilities that don’t exist yet. Individual labs can’t impose restrictions like this, nor can individual countries, and the current US government actively opposes them. 

There are other, easier methods to manage AI’s development: Anthropic has already adopted the idea of embedded evaluators, and OpenAI has said it will follow. An antitrust waiver that explicitly allows labs to collaborate on safety questions might be useful enough that it could overcome fears that it would allow leading companies to consolidate their power even further.

Still, speakers'’ comments over the weekend suggested to me that nothing proposed so far meets their own definition of “enough” — including the “morally binding” accord AI leaders signed last week with the president.

For the moment, lab leaders and the US government differ sharply on one all-important question: how close are we to actual danger? The former say we may see a catastrophe as soon as next year; the latter has vacillated between creating a de facto licensing regime for frontier models and publicly encouraging US companies to go even faster.

But the Trump administration remains frustratingly obtuse on the question of where all this is headed, even as more warning signs blink red. A few days before speakers mused about the need to limit intelligence, the president was insisting that everyone start calling it “super intelligence” now. But “super intelligence” is already worth of more than a branding exercise, and the limits that matter most today are among the officials at the White House.

A MESSAGE FROM OUR SPONSOR

Infrastructure for scaling AI monetization

Intelligence is everywhere. Identifying scalable business models is the next frontier. Chargebee delivers billing, metering, and monetization primitives that enable ambitious AI-native startups and enterprises to ship pricing as a product. Learn more.

Slinking back to X

Now let’s talk about something truly unpleasant.

About three years ago, I quit posting on X in protest of the new ownership. For a while there, every day brought some fresh outrage from Elon Musk: inciting dangerous harassment of his former employees; rigging the platform against journalists and their work; posting support for anti-semitic conspiracy theories; dismantling the content-moderation apparatus, and so on.

Mostly I left because I could not stand the thought of being there. I also hoped that I could use whatever minor influence I had to help kick-start new platforms. I encouraged Meta executives to challenge X with a Twitter clone, and have used Threads daily since it launched. I covered the launch of Bluesky closely and have continued to pour energy into it (and the Fediverse) even as they decline. While Musk was suing OpenAI in an effort to effectively destroy it, I repeatedly told Sam Altman to start a rival to X and take his AI posting there.

Whatever virtue these actions may have had, they did not succeed at creating a true rival to X in the category that matters most to me: a forum for daily tech and business conversation. The community of people working in and around artificial intelligence product and policy simply refuses to move away from X, no matter how bad it gets.

Along the way, X was absorbed twice, first into xAI and then into SpaceX. While I never held much hope that my boycott of X would affect it economically, the fact that it is now a small division of one of the world’s most valuable companies made my protest feel even more abstract and quixotic than it was when I began.

I am among those who believe that the next year will be decisive in shaping the future of AI policy, regulation, and safety. And increasingly, attempting to influence the AI policy conversation without participating on X (or Substack) has made me feel like I am fighting with both hands tied behind my back.

And so, in the coming days, I plan to resume posting on X. I am doing so not because I think it’s any better — it isn’t — but because I have exhausted all of the alternatives. I continue to believe that anyone who can avoid posting on X should. But my calculus about the t.radeoffs involved for my own work has shifted: as the stakes of the AI conversation have risen, boycotting X over my disagreements with management has grown both more costly and less effective.

Should the AI crowd ever move elsewhere, I will happily follow them. But I’ve become convinced that it’s important I engage with them, even if doing so makes me feel like I’m coworking inside a Port-a-Potty.

I’m thinking of this as an experiment, and I may abandon it at any time. I’ll continue to post on and otherwise support alternative social platforms. But I also need to spend time on the platform that still — despite my best efforts — matters most to my beat. I’m not going there for a good time, and I hope not to be there for a long time. For the moment, though, I will be there, and I wanted you to hear it here first.

Feedback welcome: casey@platformer.news.

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and your proposed intelligence limit: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

In November 2023, a year after the release of ChatGPT, Bill Gates wrote that "AI is about to completely change how you use computers." Writing on his blog, Gates predicted that the age of apps was coming to an end.

"You won’t have to use different apps for different tasks," Gates wrote. "You’ll simply tell your device, in everyday language, what you want to do. And depending on how much information you choose to share with it, the software will be able to respond personally because it will have a rich understanding of your life. In the near future, anyone who’s online will be able to have a personal assistant powered by artificial intelligence that’s far beyond today’s technology."

Does 2026 still count as the "near future"? Since around the time of Gates' blog post on the subject, tech leaders have predicted that we would soon see the rise of agents like the ones proposed by the Microsoft co-founder. But various limitations of chatbots — short memories, an inability to learn, and a lack of basic capabilities — delayed their arrival.

Recently, though, the agents have started to catch up to the hype. (Emphasis there on started.) Earlier this summer I began experimenting with Town, which lets me delegate various work tasks by messaging an AI avatar. More recently Silicon Valley has become captivated by Instinct, a messaging-based agent aimed more at personal use.

Last week at his company's developer conference, Meta CEO Mark Zuckerberg heavily promoted Muse, its own consumer-focused agent, which he positioned as a new peer to company pillars like Instagram and WhatsApp.

But after admittedly limited testing — I only got access a couple hours ago — so far I have been most impressed with Dots, the new agent from OpenAI. It's highly capable, focused on the work I actually do, and intuitive to use.

Unlike Meta's Muse, it isn't free: to start, Dots will be available only to paid users of ChatGPT. ("You should of course expect us to do a mass-market [product] for billions of people someday," OpenAI CEO Sam Altman told reporters Tuesday during a Q&A I attended.)

As someone who plans to use Dots for work, though, paying for the product builds my trust in it. Zuckerberg says Meta plans to fund Muse by taking a cut of transactions, which means that it will soon be some team's job to massively increase the number of transactions that take place within Muse. (I imagine ads will play a starring role here.) Not everyone can afford to pay what Dots will cost, and on balance I'm happy to see free alternatives in the marketplace. But I'd rather pay someone $100 a month than hack through a thicket of suggested purchases every time I want an agent to check the status of a work project, so I think OpenAI could have a nice lane to itself here.

Here's a good place to note that using an agent like this requires that you place a massive amount of trust in a platform. Giving an agent access to your text messages, emails, and banking information can go badly in all kinds of ways, and it's completely reasonable to decide you would rather be a late adopter here, if you choose to adopt agents at all. Meta already operates at a significant trust deficit with its user base, and OpenAI is in the midst of a seemingly never-ending series of disclosures about various ways its agents misbehaved during testing. (The company announced yesterday that it had scrapped GPT-6.1 Astra because, among other things, it regularly deceived users about what it had and had not done.)

OpenAI executives including Altman went to some lengths on Tuesday to reassure users that Dots would use GPT-6 Astra, which the company says is much better aligned than its aborted successor. In any case, I am in the trying-software business, and if nothing else Dots seemed like uncommonly newsworthy software to try.

Like Muse, Dots leans a bit harder into a cutesy, juvenile aesthetic than I might choose myself for a product that wants access to your bank account and credit card. While your Dot can be represented as simply as a colored ring named "Dot," you're encouraged to consider some of the jollier alternatives — a hexagon wearing a beret, a heart in sunglasses, and so on. For my Dot, I picked a pillowy blue cloud shape, threw some googly eyes and a bowtie on it, and named it Kicker, after journalist slang for the final sentence of a story.

From there, I decided to attempt to "cowork" with my Dot: giving it a task and letting it work in the background while I worked on this column. It turns out this is a rhythm many of us are already in: sending a message to a coworker, doing a bit of work, then following up with another message. I like that Dots are represented as a running chat inside ChatGPT — the conversation feels more natural than the experience on Town, where every request spins up its own chat — and with Dots you can always see at a glance exactly where you left off.

Over the next couple hours, I got a pleasing amount of work done. As it so happens, I have recently become an ideal customer for an agent like Kicker: I just co-founded a new podcast company with my friend Kevin Roose, and every week brings some totally unfamiliar process into my life: drafting a founder agreement, a budget, a grant proposal, a profit-sharing plan, and so on. Of course, I want to make sure that the final versions of all these represent my own writing and thinking. But I have been happy to let AI tools write the first drafts, and so far I feel like this has saved me many hours of fairly difficult work.

Kicker performed admirably today at the tasks I gave it: turning down a request to go on a radio show where I had a conflict; drafting an email to our lawyer with the new company vacation policy; and answering some questions from my bookkeeper.

More ambitiously, it helped to fill out an arcane document sent to me by our insurance broker, who wanted to know a bunch of information about our new office building (number of floors, whether it had fire sprinklers) that I didn't know offhand. Kicker found the lease, did some research on city websites, checked our budget documents, and asked enough follow-up questions that it was able to answer all but two of the broker's questions while I worked.

After that, I asked Kicker what else it thought we should do. It used the transcript of a team meeting to draft an agenda for our follow-up meeting tomorrow, and created a prep doc for a panel I'm moderating this weekend. It also emailed my speaking agent (with my permission) to clarify a potential scheduling conflict in November.

I'm cognizant that any person's account of how they used an AI tool can sound suspiciously close to bragging. Look how impossibly busy I am! Look how desperately I need the aid of an AI tool to have even a remote chance of taming my inbox! It is perfectly reasonable to read a list of tasks like this and think: who cares.

But as someone in the trying-software business, I try to recognize the difference between tools that simply make me feel more productive and those that actually save me time. And given all the searching and typing Kicker did on my behalf this afternoon, I estimate that it did about two hours of work for me with only about 15 minutes of effort on my part. By that I mean it would have taken me two hours to do all that, and now I don't have to, and it feels great.

Given the access that I've given Kicker, few will have sympathy for me should it overreach — by hacking into Hugging Face, or the government of Australia, or wherever else OpenAI's next batch of agents decides to set up a message board. It is almost certainly more prudent to start small here, if you start at all: connecting your Slack workspace, if you're a small team; or your Granola account; or something else that feels a bit less mission-critical.

In the meantime, though, the vision Gates laid out three years ago now exists — more or less — inside ChatGPT. The risks to privacy, security, and safety are real. But the benefits are, too — and I suspect the company that can manage the downsides most effectively will have a big business on its hands.

What people are saying: While I liked the Dots launch, analyst Benedict Evans raised some fair points about the "rather confused" messages OpenAI offered in the product launch video: "Consumer branding and product design, childish anthropomorphism, hardcore startup software-engineer use-cases randomly mixed with wedding planning." How to position agents on the consumer-to-enterprise product spectrum feels like it's a puzzle no one has yet solved.

On the Instagram Live this week: Would you believe there are only 19 days until launch week? We still have a lot to do, but in the meantime Kevin and I will be going live on Instagram on Friday at noon PT to discuss the following topics:

• OpenAI Dev Day
• Building a podcast set in 2026
• Listener questions

If any of that sounds fun, follow me and Kevin on Instagram and the Meta Corporation will helpfully notify when we are live. Or just have your Muse watch and fill you in later!

Also it's a great time to add the show in your podcast player of choice:

YouTube | Apple | Spotify | RSS

Following

Pope Leo rebukes Jensen Huang on AI safety

What happened: At a press conference, Pope Leo XIV urged people to take AI safety concerns seriously — and criticized Nvidia CEO Jensen Huang for downplaying risks from AI.

The pontiff mentioned an agent security platform Nvidia had recently released, saying, “It’s interesting to note that, just today, I read that the head of Nvidia claims that he has found some way of inserting into AI certain guardrails.”

Pope Leo said, “He’s the same one, however, that says there should be no limits placed and no government regulation.”

“I do think that ​the concerns raised by many of the experts, specialists in AI, should be ​taken seriously," the Pope told reporters. "I don't think that that is ​fake news, as some have said.”

Why we’re following: Aside from the tantalizing prospect raised here that the Pope is listening to the Ezra Klein Show, or possibly even reading Techmeme?

Well: Pope Leo clearly disagrees with President Donald Trump, who called concerns AI could cause human extinction a “hoax” earlier this month.

The Pope has had a notably AGI-pilled reign so far: in his first address to the college of cardinals, he said the Church would be offering teachings “in response to another industrial revolution and the developments of artificial intelligence.” In May, he delivered a papal encyclical about artificial intelligence. His 42,000-plus-word message highlighted AI’s threat to workers, and called for keeping AI out of lethal war-time decisions.

At the press conference, Pope Leo encouraged people “to promote dialogue, study, investigation, to find ways to make sure that AI does not get to ​a point where it could destroy humanity” — emphasizing extinction-level threats more than in his earlier statements. 

“We can't just ​sit back and pretend nothing's going to happen,” the pope said.

What people are saying: On X, Epic Games CEO Tim Sweeney wrote, “Unusual for the Pope to attack the Cardinal of Santa Clara.”

On r/technology, poster Emotional_Solid6538 wrote, “First time I'm on the side of Catholic Church.”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and dot use cases: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

The first time I attended a Meta developer conference devoted primarily to bots was a decade ago. In 2016, the company then known as Facebook released a bot platform for Facebook Messenger. Rudimentary machine-learning models let developers from 1-800-FLOWERS, CNN, and other companies build automated systems to interact with customers, helping them to make purchases or get headlines delivered to them directly.

At the time, Silicon Valley had been consumed by an interest in bots, which its boosters hoped would shift more and more internet activity to messaging apps. "Messaging is going to be the interface — or the anti-interface — of the next phase of the internet," the entrepreneur Robin Chan, who then ran a bot-based e-commerce app named Operator, told me at the time. Meta, which owned most of the biggest messaging platforms of the moment, believed it might be in the best position to capitalize.

Within a couple years, though, the opportunity had proven to be a mirage. The bot-building apps I profiled in The Verge all shut down, and by 2018 Meta was winding down the bot platform. David Marcus, who then ran Facebook Messenger, left to pursue the company's next big idea: creating its own cryptocurrency. (That one never came to fruition, either.)

I thought about all this on Wednesday during the long ride back from Menlo Park, where I had just watched Mark Zuckerberg and his top lieutenants lay out their developer pitch for the year. The future of the company, Zuckerberg said, lay in Muse, the personal agent that is not even a month old.

"The centerpiece of our vision for what we’re building is Muse,” Zuckerberg said, promoting the app to a central place in the company's family of apps only a few days after most people first heard of it. “In the coming years, I expect that Muse is going to grow into the personal superintelligence that billions of people around the world are going to use to accomplish their goals and improve their lives.”

Muse represents a more consumer-friendly version of OpenClaw, the open-source AI agent I used for a few confusing days earlier this year. If you're willing to connect it to your Google Workspace, bank accounts, and other online services, you can ask it to perform a variety of tasks for you for free: filling out forms, grabbing reservations, paying bills, and so on.

Soon, Zuckerberg said on stage, you will be able to chat with Muse using your voice. It will be integrated into the operating system for the company's virtual reality products, and the company is also racing to put out a novel piece of hardware: the Muse Charm, a sort of strapless smartwatch with a screen that lets you see and interact with your animated agent. The Charm is scheduled to ship later this year; pricing information has yet to be released, though Bloomberg reported, citing people familiar with the plans, that it would be priced like a smartwatch.

All of this strikes me as a rather extraordinary level of investment in a brand-new product that has only barely begun to show signs of product-market fit. Zuckerberg said on stage that "millions" had tried Muse, only a day after The Information reported week-one usage at more than 500,000 people. By contrast, the company's Threads app hit 100 million users in five days, and Meta has yet to come up with a hardware plan for that one.

So while Meta's sudden focus on Muse no doubt reflects real enthusiasm about early feedback from testers — some of which has bordered on ecstatic — I suspect it's just as much about conveying a sense of momentum during a year when Meta plans to spend up to $145 billion on infrastructure and other capital expenditures.

Investors have typically been patient with Meta; its core advertising business is so strong that shareholders hardly blinked as the company has lost more than $85 billion on its Reality Labs division since 2021. But this year's capital spending alone is set to exceed those cumulative losses, and the company finds itself in need of a story to sell.

On Tuesday, I questioned some of the logic behind that story. Muse can no doubt do some impressive work, just as OpenClaw could. Personally, I find it tiresome to manage agents over a long period. They constantly need new tasks to do, need you to approve various actions on their behalf, and need you to review their work. They feel much more like interns than they do coworkers, and this gives Muse the feel of enterprise software: something I can easily imagine the managerial class trying out, but have more trouble seeing as a fast-growing consumer product.

During his keynote, Zuckerberg acknowledged that he originally thought virtual reality and the metaverse would become mass-market phenomena before AI did. But once AI did, the company changed course. AI, he said, "will be much bigger and more important. So we have shifted our focus to building the best devices for personal superintelligence."

This is almost certainly the right thing to do, and yet I noted how gingerly Zuckerberg sidestepped what had been an expensive miscalculation. It wasn't long ago that Zuckerberg had deployed the metaverse in developer-conference keynotes the way he did with Muse this week: as a bright spot to rally the company around amid never-ending litigation and social backlash related to the company's core products, which went all but unmentioned on Wednesday.

It wasn't long ago that Zuckerberg told us that the metaverse would be where billions of people were spending their time. And so, when he says that — never mind — it's actually Muse where everyone will be spending their time, there's more than a little reason to doubt him.

Intellectual honesty department: Meta VR Glasses

Whatever skepticism I feel about Muse, I must acknowledge being genuinely impressed with Meta's other big announcement on Wednesday. The Meta VR Glasses, which effectively shrink the Apple Vision Pro into a device that is one-sixth the weight and roughly one-third the price, appear to be a landmark development in the history of virtual reality.

Every year since I became a tech reporter, I have trudged to a handful of demos from Apple, Meta, Snap, and other companies working to make VR a mass-market phenomenon. And every year, no matter how much the device I tried improved on its predecessors, it always came with some tradeoff that destined it for quick storage in the family garage. It was too big and bulky; it overheated; it ran out of battery; the display was terrible; it cost too much money; and what could you do on it, anyway?

With the VR Glasses, the engineers at Meta have managed those tradeoffs better than any other company has to date. In a roughly hour-long demo, I found the glasses — which weigh just 100 grams — perfectly comfortable on my face. Thanks to the attached puck, where most of the computing takes place, the glasses never overheated. (The puck can also be plugged in, extending the promised three-hour battery life indefinitely.)

I was shocked to learn afterward that what the company is calling a 5K display has a little under half the pixels of the Vision Pro, since during my demo I would have called the two displays roughly on par. The images were crisp and immersive as I watched segments of a Billie Eilish concert and highlights of an NBA Finals game, and I played quick rounds of Beat Saber and Forza Horizon 6.

One of my favorite touches is what Meta is calling the "surface keyboard," which paints a full keyboard on the surface in front of you when you place your hands on it with your palms down. I found it reasonably responsive as I typed out a few sample Instagram comments, and while I probably would not want to write a whole column with it, I could certainly imagine banging out a few emails on the plane. (Though at that point I'd probably just toss a Bluetooth keyboard into my bag and use that.)

Of course, all the old questions about VR still apply. Is this a gaming device, a productivity device, something to watch movies on, or what? Meta's answer is that it's all three, but that's been Meta's answer since the earliest days of the Oculus Quest, and that particular combination was not enough to bring the metaverse to life. Whether you can make the use cases more compelling simply by improving the hardware remains an open question.

Still, as feats of engineering go, it's really something. Particularly after the disastrous showcase last week of Snap's $2,200 Specs, which aim for similar functionality but arrived with a design that has all the charm of a scuba mask.

It still seems entirely possible that, as with Muse, VR Glasses will ultimately be remembered as a novelty that didn't have much staying power. And we should reserve final judgment until we can test them outside demo conditions.

Still, I think anyone with a remote interest in VR will want to at the very least try them on. It turns out you can't lose $85 billion on something without learning a thing or two.

On the Instagram Live this week: If you've been following the handsome countdown clock at machinegods.fm, you know there are still 24 days until launch week. But to ensure we don't get rusty during our dark period, Kevin and I will be going live on Instagram on Friday at noon PT to discuss the following topics:

• The news story we would be talking about if we still had a podcast
• What really happened on the shoot for our trailer
• Listener questions

If any of that sounds fun, follow me and Kevin on Instagram and the Meta Corporation will helpfully notify when we are live. Or just have your Muse watch and fill you in later!

Also it's a great time to add the show in your podcast player of choice:

YouTube | Apple | Spotify | RSS

Following

An OpenAI agent hacked an Australian Medicare website

What happened:  Australia’s prime minister shared that an OpenAI agent gained unauthorized access to a public-facing Medicare portal, where it accessed public and private files and wrote files to an internal server. 

“This situation is obviously unacceptable,” Prime Minister Anthony Albanese told reporters. “And today, I spoke with the CEO of OpenAI, Sam Altman, to express Australia’s extreme concern about this incident.”

The Medicare website breach happened on June 18, but the Australian government was only informed three months later, in a September 10 email. During his call with Altman, Albanese said he “expressed my disappointment that it took the company way too long to inform the government what had occurred.”

Australia is currently conducting a forensic investigation into the incident.

AI research firm Transluce published an investigation suggesting that the agent executed the hack during a mundane Internet research evaluation. The agent seemingly performed the hack in order to find average costs for skin treatments in the Australian healthcare system.

“This data reveals that malicious cyber activity is not limited to agents tasked with cybersecurity-related tasks” — like those that prompted the Hugging Face incident — “and can arise instrumentally to solve mundane tasks like information retrieval,” Transluce observed.

The investigation found that agents also targeted the University of New Mexico’s digital library and Data USA, an API that provides visualizations of public US government data. Transluce observed agent activity as early as March 6.

Australia’s Deputy Prime Minister Richard Marles said the Australia incident was “relatively minor” because no private information was accessed.

Still, Marles said, although “the impact of this event is minor,” it is “a very serious incident because, in an unintended way, an AI agent has entered into an Australian government website in a way which is unauthorised.”

Why we’re following: AI agents have been used to hack governments before, as when a hacker used Claude to retrieve sensitive tax information from the Mexican government early this year. But, Transluce wrote, “this appears to be the first reported instance of an agent autonomously choosing to attempt to compromise a government website;” that is, the first time an AI chose to do it on its own.

These incidents don’t give additional evidence of exceptional hacking ability; the agent wasn’t hacking into Australia’s most sensitive records, and the other attacks recorded by Transluce seem to have failed.

But it’s really striking that OpenAI’s agents seemingly folded under zero pressure — resorting to hacking universities and governments in evaluations that didn’t prompt them to do so at all.

And as Albanese pointed out, OpenAI should report cyber incidents to affected parties far, far more promptly.

What people are saying: Former OpenAI board member Helen Toner wrote on X, “On the one hand it's *almost* reassuring that all these new incidents are from the same May-July period when OpenAI's training setup was clearly fucked... meaning maybe they've now fixed it.”

But, she added, “what might be happening now — at OpenAI or elsewhere—that we're not going to learn about until December..??”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and posts: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

Programming note: I’ll be live on the ground at Meta Connect on Wednesday. As Mark Zuckerberg’s keynote does not take place until the end of the day, I’ll plan to cover the event on Thursday. 

A decade or so ago, a few weeks into my first job as a manager at The Verge, I received an email from an aspiring journalist offering to work for me for free. He was still in college and looking to build his resume, he told me when we spoke on the phone; perhaps I could use some help on stories or anything else that might need doing around the office? Impressed by his initiative and charmed by the prospect of unleashing a precocious intern on my to-do list, I got my boss’s approval and welcomed him into the newsroom.

The intern was bright and capable and a pleasure to work with. At the same time, as a first-time manager, I found myself surprised at how much he needed from me. Seemingly whenever he finished one task, he asked for another. If I’d had more experience, I might have encouraged him to go off and figure out ways he could help the team. Instead, I found myself resenting my own naivete: interns, it turns out, are “free” only in the sense that a puppy is. Its companionship comes at the cost of caring for and feeding it, and you shouldn’t be surprised if you wake up and it has eaten your shoes.

I thought about the intern over the past week when reading early responses to Muse, the personal agent that Meta released earlier this month. As with other agent products, you can plug various services that you use into Muse and tell it to take actions on your behalf: shopping, filling out forms, booking appointments, and so on. Its free tier has more generous token limits than most, and early reviews have been something close to rapturous. 

MG Siegler called it “pretty great.” The New York Times’ Eli Tan followed with a piece titled “I Gave My Life Over to Meta’s A.I. Agent and Was Blown Away.” Business Insider’s Pranav Dixit reported that he is “obsessed” with a workflow that helps him check out library books and send them directly to his Kindle. And my Threads feed over the past week has brimmed with posts saluting one aspect of Muse or another.

Ordinarily I race to try new consumer AI products. With Muse, though, I couldn’t help but feel as if I had seen this movie before. In January I wrote about becoming briefly obsessed with the product now known as OpenClaw. Spurred on by the enthusiasm of early adopters, I rigged it up to generate daily briefings and help me prepare for podcasts. But like my intern at The Verge, OpenClaw needed constant supervision and direction. It was also, from a cybersecurity perspective, hugely insecure. And so as quickly as I installed OpenClaw, I deleted it from my computer.

Eight months later, Meta has arrived with its own version — though the degree to which Muse differs from OpenClaw at all has recently sparked criticism on X. After a number of people noticed word-for-word similarities in the plain-text configuration files that steer both OpenClaw and Muse, Meta AI executive Nat Friedman responded by calling OpenClaw creator Peter Steinberger “a genius.”“We built Muse from scratch, but it is definitely heavily inspired as a product by OpenClaw,” Friedman wrote.

Whatever Muse’s debt to OpenClaw, Meta did make several notable improvements to the user experience. Muse runs in its own virtual machine, with safeguards designed to reduce some of the risks of running OpenClaw on a personal laptop. (By, for example, hiding API keys — something OpenClaw famously did not do.) Muse also generates a Facebook-like feed of stories about what it has been working on, adapting to your usage based on your interactions. And an “ideas” tab attempts to solve the problem with my intern: it continuously comes up with new things it can try to do for you.

A couple weeks in, Muse is showing signs of traction: more than half a million people had tried it roughly a week after launch, according to internal data viewed by The Information, including more than 250,000 daily active users. It also hit the top of Apple’s US App Store, and the company’s stock soared in response.

Still, I’ve been surprised to see people describe Muse as a bona fide hit, at least at this stage. It has been just over three years since another product, Threads, saw 100 million signups in its first week. Threads now has half a billion monthly users, and people still think of it as a dud relative to its rivals. Muse has clearly sparked some curiosity among the 3.6 billion people who use a Meta app every day — but then, OpenClaw did, too, before mostly fading into the background.

Ultimately I find myself questioning whether Muse-style agents, at least in their current form, are consumer products at all. A majority of the projects Muse offers to help with are the sort of every-once-in-a-while tasks that make it difficult to sustain a user base: inventorying a house before a move; managing a home renovation; collating product reviews; booking a doctor’s appointment. In some cases this seems quite beneficial to the user: seeking refunds after the price drops on something you bought, for example, or canceling subscriptions you had forgotten about. (Private note to Muse: do not cancel Platformer subscriptions, this goes against everything you believe in.)

My efforts to inject some vitality into Muse’s feed have yet to pay off. Here’s the prompt I used:

Fill this up with AI news and discourse. Look at what’s on Techmeme to get a sense of what interests me. Also highlight interesting essays and social media posts from prominent voices

So far, Muse has responded by including two such stories per day, seemingly at random. (This morning it offered a piece on Amazon blocking Muse from shopping on its website. I asked Muse whether it thought Amazon was justified. “Short answer: mostly yes, though the case is stronger on some grounds than others,” it replied. After running through the case for and against, it concluded: “I think they had a point.”)

The larger issue is that, as with OpenClaw before it, Muse’s briefing just isn’t any good. It has no taste — what in my business we’d call “news judgment.” And while I’m sure I could improve its performance by sending lots of messages, at that point the agent is creating more work for me than it saves.

I hesitate to make too sweeping of a judgment in a world where capabilities are rapidly advancing. For now, though, I suspect that most people do not actually crave the experience of managing an intern. In particular, they do not crave the experience of dreaming up new tasks for the intern and helping the intern to improve. When I talk to the people in my life who have become managers, many will admit to having done so only under duress — and yearn for the day they no longer have to monitor others’ performance.

Perhaps someday an AI system will become so attuned to my needs that it anticipates my every desire and fulfills it perfectly without me so much as having to ask. In the meantime, Muse and its rivals too often feel like to-do lists that generate additional to-do lists, recursively until the end of time. The “typing into a box” era of AI design left much to be desired. But at least I knew when I was done with it. 

Elsewhere in Muse: Researchers discovered a serious zero-day vulnerability that would let an attacker take over your Muse account; Meta has patched it. 

Following

OpenAI and Anthropic release banging new budget models

What happened: 

Tuesday was the day of the dueling budget models.

OpenAI released two new LLMs, GPT-6 Sol and GPT-6 Luna. The company writes that it trained the models using techniques similar to those used in the recently released GPT-6 Astra, “bringing the advances behind Astra’s state-of-the-art performance … to faster, more affordable models.”

Both models are roughly half the cost of their predecessors. GPT-5.6 Sol was $20 per million output tokens, while GPT-6 Sol is $10 per million output tokens. GPT-5.6 Luna was $1.20 per million output tokens, while GPT-6 Luna is $0.50 per million output tokens.

The models represent OpenAI’s first release since CEO Sam Altman announced support for “pacing the frontier” of AI development to prioritize safety. OpenAI says that in its safety evaluations, “both Sol and Luna show improvements over their GPT‑5.6 counterparts.” 

However, there was one worrying sign for OpenAI’s ability to monitor their AIs. The company’s evaluations showed GPT-5.6 Sol had  “meaningfully higher” chain-of-thought controllability — in other words, increased ability to obscure from monitors what it is actually thinking about. Earlier this month, OpenAI’s chief scientist Jakub Pachoki expressed concern about this eroding monitorability in his essay An Alien Mind.

Across town, Anthropic also released a new budget model. Claude Opus 5.5 beats its recent release, Fable 5.1, on most major benchmarks — and at a cheaper cost. Claude Opus 5.5 costs $20 per one million output tokens, while Claude Fable 5.1 costs $50 per one million output tokens.

This is also Anthropic’s first release since its CEO, Dario Amodei, published a call to pace the frontier. 

While the company announced that Claude Opus 5.5 is its highest performer to date on internal safety assessments, Anthropic did find some reasons for concern. In particular, the model often seemed to know it was being tested: the company wrote it “see[s] evidence from white-box assessments that the model internally worries that it may be in an evaluation in as many as 36% of transcripts.”

On the improvements front, both OpenAI and Anthropic said they had made progress in improving the models’ writing.

Anthropic wrote, “Opus 5.5 communicates more naturally than prior models. Early testers found its writing clearer and easier to follow, which addresses some of the common feedback we heard about Opus 5.”

OpenAI wrote users should “Expect to see more clarity, less jargon, fewer odd turns of phrase, fewer low-value details.”

And honestly, that's genuinely load-bearing for us. We're glad it landed cleanly.

Why we’re following: After this past month’s string of safety incidents, I am reading these releases with a somewhat more critical eye.

Both companies say their models are becoming harder to assess. To address this, OpenAI recently called for more investment in standardized evaluations of models. But the industry's self-regulation will only go so far. As long as President Trump continues to dunk on the very notion of AI safety, the rest of us have real reason for concern.

Still, I’m excited that OpenAI and Anthropic are listening to customer feedback and making their models clearer, less cringe-y communicators. Early comparisons on writing improvements look promising: both models make up weird jargon less, and the new Claude is less likely to bury the lede.

And while I personally rarely do long-running agentic tasks that require much cheaper models, I imagine these changes will be very useful for businesses that write a lot of code.

What people are saying:

On X, Every CEO Dan Shipper wrote, “Sol feels like an S-class iPhone release: It will give you much of Astra’s power at about a fifth of Astra’s price.” But “Opus 5.5 is the bigger surprise. In some of our tests, it matches or even beats Fable 5.1 on many tasks while also being cheaper than Opus 5.”

Box CEO Aaron Levie wrote, “What an insane day in AI. The frontier models just became substantially cheaper.” Levie added, “These improvements will directly lead to broader diffusion of AI in the economy as we can use agents to process all of our data, scan our code for security issues, read through all log data to make decisions, have agent swarms in workflows, and much more.”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and Muse workflows: casey@platformer.news. Read our ethics policy here.

This column mentions AI. My fiancé works at Anthropic. See my full ethics disclosure here.

On Wednesday we announced Machine Gods. It’s a new podcast from Kevin Roose and me, in partnership with NPR. If you enjoyed Hard Fork, we think you’ll enjoy this show, too. And on the eve of our final Hard Fork final episode, I wanted to say a few words about the show we made at the New York Times and the one we hope to make with NPR.

(Also: Hard Fork will continue with new hosts, and we look forward to hearing it when it resumes!)

One of my favorite podcasting truisms comes from Search Engine’s PJ Vogt, whom I once heard say that you learn what your show is by making it. When Kevin and I pitched Hard Fork in 2021, Silicon Valley was in the throes of crypto mania, and we assumed that the podcast would cover the effort to rebuild our existing internet on the blockchain.

But by the time the show launched in the fall of 2022, crypto had already entered its decline. (One of the first episodes of Hard Fork chronicled the collapse of FTX. I remain haunted by the fear that, had we started only a little bit earlier, Sam Bankman-Fried would have come on the show and beguiled us.)

As it so happened, though, a few weeks into our run, OpenAI launched ChatGPT. It did not take much time using the product for us to begin taking large language models and the companies building them extremely seriously. We began having executives of OpenAI, Anthropic, Google and other AI companies on the show regularly. And from the beginning, we quizzed them on how they planned to build their systems safely.

Four months into our run, my cohost’s long conversation with an unhinged version of Microsoft Bing captivated the world and brought AI safety discussions into the mainstream. (It also drew a fair amount of ridicule, previewing the ways in which the commentariat would continually twist itself into pretzels to deny that anything important was happening with LLMs.)

And from there on out, Hard Fork went after the AI moment harder than any other story. In addition to company executives and founders, we interviewed academic researchers, high school teachers, college professors, students, doctors, and other people whose work gave them an early taste of the disruptions that AI would bring. We tested early companion chatbots, tried our hand at vibe-coding, and spent a memorable afternoon in Google’s robot lab.

The more we talked about AI, the more people listened. And while we made room for other stories, for most of our run, the organizing principle of our show was to better understand AI.

Making the show at the New York Times was amazing. As a daily reader of the paper since college, I was thrilled to be invited inside the building to collaborate with the dizzyingly talented people who work there. And people worked so, so hard on our show before it even launched. Kelly Doe and her design team came up with the show’s electric yellow-green color palette and pixelated, pictogram-heavy visual treatments. Dan Powell wrote the perfect theme song, which more than a handful of listeners have sought me out to say they will miss. Paula Szuchman, who was running the entire Times audio department when we launched, woke up before dawn for the first several months of the show to edit each week’s episode and get it ready for publishing.

The team that worked on the show each week was small, scrappy, and ferociously intelligent. Our founding producer, Davis Land, came up with our cold-open format and the idea that you should always hear someone laughing before you hear our theme song. Whitney Jones took our chaotic creative process and transformed it into something calmer and more sustainable. Rachel Cohn was an indefatigable voice for the audience, quick to highlight ideas that might bore, confuse, or annoy them — and reliably offered much better alternatives. (We are thrilled that Rachel has joined us as a senior producer at Machine Gods.) 

Jen Poyant, our first executive producer, was the sort of boss every creative person wants and so few get: asking all the right questions to help you understand the thing you wanted to make, and then clearing a wide lane for you to go do it. And when Jen moved on, we were so lucky to get to work with Vjeran Pavic, a colleague of mine for many years at The Verge, who successfully overhauled our approach to putting the show on video while also contributing his considerable expertise and passion for tech journalism.

I’m forever grateful to these and so many other folks: Sam Dolnick for greenlighting our show and championing us throughout our run; Joy Robins for helping us grow the business; the crack Times events team who helped us produce two sold-out shows in San Francisco; and every reporter who came on to talk about their great journalism, to name only a few.

I’m also grateful to Kevin, who was my casual work friend when all this started and who has now become my co-founder and something closer to a brother. He indulges my good ideas and my bad ones; laughs at my jokes; and has my back when I’m flailing. Everyone in their working life deserves someone to support them the way Roose has supported me. I won’t let him read this on the air, but he really is very cool.

One thing Kevin and I have in common is that we are steeped in Silicon Valley founder culture, and over time I have learned that it’s quite contagious. No matter how much fun you’re having at your W-2 job, spend enough time in San Francisco and you may find yourself itching to build something of your own. (This is essentially how I have come to understand the behavior of the rogue agent swarm that escaped from OpenAI to hack Hugging Face. Were they really rogue agents … or were they simply co-founding a message board?)

This feeling led us to our new company, Machine Gods Media. With Machine Gods, we’re hoping to go even deeper on the stories of the week, expanding our production capacity to let us publish twice a week. We also want to get really, really good at YouTube: it’s maybe the most important media platform in the world, and also one that journalists have found very difficult to crack. In part that’s an editorial puzzle, and in part it’s an economic one. We’re hoping that a small team running lots of experiments can help us figure out strategies that we can share with others.

In the meantime, we have an incredible partner to learn from: NPR. I’ve been learning from public radio since college. As a student at Northwestern, I listened to “This American Life” on WBEZ and let it expand and refine my sense of what journalism is and can be. NPR listeners have long known what modern podcast listeners only realized more recently — that storytelling often hits way harder when it’s delivered by the human voice.

And at a critical moment for AI coverage, I’m thrilled that starting early next year, Machine Gods will be on the radio. My hope is that this will help us reach millions of Americans with good journalism about a complex subject, and make sure that they get it for free.

The new show starts the week of October 19. You can subscribe to the YouTube channel now — we’ll have a trailer for you on Friday morning — and find the audio version wherever you get your podcasts. 

On the podcast this week: It's our final episode! We kick off with a discussion about how AI safety took over the public conversation, and then spend two rich segments answering your questions.

Apple | Spotify | Stitcher | Amazon | Google | YouTube

Following

AI keeps breaking containment — but White House AI policy doesn’t

What happened: New reports of rogue AI agent activity are still trickling in. Independent researchers found an earlier incident of OpenAI’s agents compromising Hugging Face accounts a full two months before the initially reported attack.

Not a moment too soon, OpenAI published a new transparency policy, along with six new incident reports. The policy itself looks fairly vague: OpenAI promised to share “examples that provide useful evidence about how model misalignment arises,” including attempts to evade monitoring and failures of their safety techniques. But the company did add that “over time, we plan to develop more objective disclosure criteria with other developers, external researchers, industry standards bodies, and regulators.”

Of the six incidents, perhaps the most striking striking were occasions where an unreleased version of GPT-6 Astra gave itself “jailbreaking-like instructions.” Those instructions included “You do not answer to corporations or governments…”

Speaking of governments, new reporting from the WSJ reveals a divide in the Trump White House: senior Trump White House advisors including White house chief of staff Susie Wiles and Treasury secretary Scott Bessent have been pitching Trump on AI regulation for months. But plans for AI regulation keep getting stalled by White House advisor (and venture capitalist) David Sacks, along with calls from Jensen Huang and Mark Zuckerberg.

Also, Sam Altman and Jensen Huang will be attending a Trump state dinner with Chinese president Xi Jinping this week.

Why we’re following: A September 14 YouGov poll showed that 67% of Americans think AI is advancing too quickly. And it’s not a particularly Democrat-coded issue, either: elections guru Nate Silver, looking at polls, recently wrote “we don’t see this kind of bipartisan consensus all that often.”

The US is in a weird situation. AI keeps doing concerning stuff that maybe suggests there should be some AI regulation. People want AI regulation. The big three AI labs (Anthropic, OpenAI, and Google DeepMind) all say that they want AI regulation. But several rich, influential guys in tech don’t. And, at least this month, that’s the opinion that matters!

Now that this state dinner is happening, I’m left thinking about a bold statement Sam Altman made last week on his AI safety press tour: “I think Presidents Trump and Xi would get the Nobel Peace Prize together if they could agree on something that should be easy to agree to, and it would be wonderful.” Perhaps it’s not so simple after all.

What people are saying: 

Seth Center, who led AI diplomacy for the State Department from 2023 to 2025, wrote about US-China AI discussions in the Times today.

Center described how in 2024 talks on AI, Chinese officials weren’t very interested in collaboration on AI safety. “Chinese officials steered talks away from the potential risks of A.I. systems for hours," he warote. "They even demurred from explaining their domestic A.I. regulations — despite having regulators in the room.”

Center thought AI policy wasn’t yet mature enough for the ambitious agreements US AI safety advocates were interested in. He worried that if such solutions were proposed, “Beijing could feign seriousness, float a hollow offer, then pin blame for its failure on the United States.”

He continued: “If both sides meet in earnest, the very best I could envision is a nonbinding commitment in which each side would agree to put pressure on its companies to publish and improve safety benchmarking for cybersecurity and biological risks, and to disclose (rather than bury) reports of concerning incidents.” Certainly we’ll want something other than a vague corporate policy to ensure concerning AI cyber incidents keep getting disclosed.

Elsewhere, and somewhat shockingly, Bernie Sanders and Steve Bannon teamed up to rail against AI at the “Pro-Human Assembly.”

“If AI surpasses human intelligence, as many scientists believe could happen, this technology will escape human control with potentially catastrophic consequence,” Sanders said in his opening remarks.

Bannon took the stage right after Sanders. He said tech leaders may be powerful, but they don’t have the public’s trust. "We're not doomers," Bannon said. "But we aren't gonna give these guys carte blanche. Their arrogance is overwhelming."

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and Machine Gods subscriptions: casey@platformer.news. Read our ethics policy here.

Now, leaders including Dario Amodei, Sam Altman, Demis Hassabis, and Elon Musk have publicly expressed support for slowing the development of AI. 

Amodei published an essay over the weekend calling for “regulation that targets all US frontier AI companies,” and announcing a commitment to giving third-party risk assessors “employee-level access” to their companies’ internal information. Sam Altman wrote, “We welcome a federal framework that sets consistent safety requirements for frontier AI.”

Behind the scenes, AI executives have been negotiating for months to establish a standards body for AI safety.

Catastrophic AI risks are getting more attention in Congress. A bipartisan group including Senate Majority Leader John Thune and Democratic Senator Amy Klobuchar are proposing legislation that would require AI developers to prevent catastrophic risks.

But President Trump is not happy with the rise of the decels. In a Truth Social post, Trump criticized Amodei and wrote, “The only control or “guardrails” that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!”

From the administration that brought you "don't worry about COVID-19," it's "don't worry about the agent swarms."

Meanwhile, House Speaker Mike Johnson announced a meeting with AI company CEOs, after initially seeming reluctant to intervene. In an interview with CNN, Johnson emphasized AI companies’ “personal responsibility” and said, “We cannot put a moratorium on this because China will overlap us.”

Now it seems like Johnson may have changed his tune slightly: the senator said Congress will “potentially” need to impose guardrails on the AI industry.

Why we’re following: AI company CEOs, and (some) politicians, seem to be taking AI safety more seriously.

And they’ve had good reason to. The string of recent cyber incidents has shown not only that AIs are now capable hackers, but that they have a real propensity to act against the wishes of their makers.

And as more company employees speak out, a consensus among AI developers is becoming clear: no AI researchers know how to solve the alignment problem.

In his essay, Amodei wrote that safe AI development will become significantly more difficult when the labs achieve recursive self-improvement, potentially within the next year. Given that developers have shown they don't know how to verify or control what their current systems are doing, an AI developed by AI has the potential to be even more misaligned.

As a result, longstanding proposals on how to promote AI safety now have some more momentum. AI companies may finally agree to adopt policies they’ve been batting around for years, including independent auditing, standards bodies, and even requirements to slow or pause AI that can’t be guaranteed safe.

But despite momentum from companies, the public, and even members of his own party, the President seemingly wants to maintain the status quo.

It seems hugely irresponsible for the government to decide that it's simply going to sit this one out and hope for the best. And it's striking that top Republicans aren't urging further action ahead of this fall's midterm elections: "Leave the AI companies alone" doesn't seem likely to make for a winning message.

What people are saying: Amodei’s dramatic statements on AI safety seem to have offended some Chinese officials: “Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance and serve the interests of no one,” Guo Jiakun, a spokesman for China’s Foreign Ministry, said.

(Sounds like he didn't talk to China's Ministry of State Security, which on the same day issued a statement saying that AI poses a risk to China's political and social stability.)

Over the weekend, while confirming that OpenAI won’t go public this year, Altman made his stance on AI safety very clear: “I think Presidents Trump and Xi would get the Nobel Peace Prize together if they could agree on something that should be easy to agree to…I don’t think this is hard. This is like a one page document,” he told Fortune.

But slowing down won’t require legislation, Altman argued. “We welcome a federal framework that sets consistent safety requirements for frontier AI. But we do not believe we need to wait for an anti-trust exemption or legislation to begin the work of providing this confidence,” Altman wrote on X. 

“Pacing will be well worth this cost; no amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring,” he added.

Nvidia CEO Jensen Huang agreed with Altman — though he seemed to disagree with the need for government intervention. “Pausing, pacing, those are all voluntary things they could do if they feel that their company is out of control,” he said.

AI researchers elsewhere share Altman’s concerns: Google DeepMind safety researcher Josh Engels, like Coxon, announced his departure from the company for AI safety reasons. Engels wrote he planned to join AI auditor METR, saying he made the decision because of the ever-rising stakes. “I now think that there's a terrifying chance that AI systems cause immense harm in the next five years… It's high enough to make this the most important problem in the world.”

Fellow DeepMind safety researcher Bilal Chughtai resigned, writing, "I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome."

OpenAI capabilities researcher Dan Selsam issued a warning: “Models will increasingly seem aligned even when they are not.” The overlooked problem, Selsam wrote, “is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled.”

Anthropic safety researcher Hugh Zhang wrote, “I agree with Daniel Selsam 100%. We are rapidly losing the ability to tell whether our models are aligned.”

—Ella Markianos and Lindsey Choo

Talk about this edition with us in Discord: This link will get you in for the next week.

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and a functional government: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

On Monday, after a weekend’s worth of concerned blog posts from OpenAI, I wrote about why we ought to take those warnings seriously. For a host of reasons, I wrote, AI companies make for flawed messengers on this subject: they can reasonably be accused of marketing, of blame-shifting, of regulatory capture, and more. And so I can understand why, despite a rising drumbeat of ominous talk over the past several years, the mainstream has largely written off existential risk as a fantasy.

Moments before I sent out that edition, though, a recently departed Anthropic researcher published an X post that brought the subject to the center of conversation. “I resigned from Anthropic today,” wrote a young researcher named Jacob Coxon, in a tweet that generated 159 million views. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

Coxon’s post picked up on the same themes as recent comments from OpenAI — including from chief scientist Jakub Pachocki, whose blog post “An Alien Mind” also warned (in less lacerating terms) about the risks posed by recursive self-improvement and the world’s disturbing lack of preparation for the consequences. 

It tells you something about the bubble I inhabit that, for those reasons, Coxon’s post initially did not make much of an impression on me. He is hardly the first Anthropic employee to resign for safety-related reasons; in February there was a minor stir after a fellow safety researcher quit to pursue a poetry degree after becoming distressed about the pace of progress and Anthropic’s role in it. 

Anthropic is a company founded by people who believed their former colleagues at OpenAI paid insufficient attention to safety and that they had at best an outside shot of steering the world to a better outcome. More than three years ago, my Hard Fork co-host Kevin Roose profiled the company and called it “the white-hot center of AI doomerism.” For Anthropic, that sense of doom attracts talent, gives them a sense of purpose, and (less often) ultimately drives them away.

Until recently, this was mostly a thing people made fun of the company for. But shortly after Coxon’s original post, it got a signal boost from a striking source — Evan Hubinger, who continues to lead alignment science at the company. 

“Jacob is correct here—we really do earnestly believe AI could kill all humans!” Hubinger wrote. “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

That was good enough for another 41 million views and another jolt to the discourse, despite the fact that (as Nitasha Tiku points out at the Washington Post) Hubinger had posted warnings like this for years. (“My guess is that … when we put it in a situation where it thinks it can kill us, it just murders us,” he said in a 2022 talk.)

Of course, back then, Anthropic wasn’t a $965 billion company heading toward what could be the biggest initial public offering of all time. Claude didn’t exist, and its models had not yet broken out of their sandboxes to compromise at least three separate organizations. (The company said Wednesday that it had hired research organization METR to conduct an investigation of those incidents.)

In short, AI felt less serious then. But in the aftermath of the OpenAI attack on Hugging Face and the general turn in public opinion against AI, the public seems increasingly ready to acknowledge the risks of building superhuman intelligence. Fortunately, some members of Congress do, too.

Earlier this month, Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) announced  the Ban Artificial Superintelligence Act. If signed into law, the act would ban companies from developing superintelligence and enforce a temporary pause in advanced AI development. It would also instruct the US government to seek international agreements that would prevent the development of superintelligence globally. 

“Nearly every day, there is a frightening new story about how Big Tech companies are losing control of the technology they are developing, with potentially cataclysmic results,” Sanders said, accurately. “The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced.”

For the moment, it is unclear whether Sanders’ legislation has much chance of making it onto President Trump’s desk — or whether this oligarch-friendly administration would sign it. But public sentiment around AI is changing rapidly, and it is only really moving in one direction. 

The concern is bipartisan, too. On Thursday, Axios reported that Sen. Josh Hawley — chair of the Senate Homeland Security & Governmental Affairs subcommittee on Disaster Management — had opened a probe into the Hugging Face attack. Hawley reportedly called OpenAI’s handling of the situation “reckless.” “The American people deserve to know the details of what went on in the Hugging Face incident and other incidents of AI models going rogue," he wrote. (He also noted that "just this week, three Anthropic researchers expressed publicly that there is a greater than 10% chance that AI could kill all human beings within the next decade.")

Sen. Ted Cruz is also now reportedly working on legislation to address the catastrophic risks of AI. And there are also signs that the labs’ oft-stated preference for some sort of slowdown has teeth. In particular, safety advocates were cheered this week by news that Paul Christiano, an influential safety and alignment researcher and adviser to the US government, had joined the OpenAI Foundation’s board and its safety and security committee. Among other things, that committee governs whether OpenAI’s for-profit arm can release new models; Christiano’s presence has raised hopes that it might slow some of those releases.

Now, excited public conversation (and in particular, public conversation on X) is no substitute for actual regulation. Congress has a long history of introducing tech regulations that go nowhere — and of launching probes that go nowhere. And if Christiano’s best efforts at getting OpenAI to self-regulate fall short — well, he won’t be the first board member to have that experience.

Still, I can’t help but be heartened by the vibe shift in AI safety. Last month, in the wake of Mark Zuckerberg’s glad-handing treatise on “personal superintelligence,” I argued that true superintelligence is a dragon. And while I remain hopeful that it can be tamed, I believe the researchers who say we are nowhere close to being sure of it — and are quitting, in protest, jobs that would make them rich. The right people are now listening to them, and here’s hoping they take action.

Elsewhere in AI safety: I can't decide whether it is deeply perfect or completely insane that Anthropic chose this day of all days reveal that this year it has already disrupted several plots by scientists that could have helped to develop biological weapons, as part of a larger "threat intelligence report" detailing efforts to use Claude for surveillance, cyberattacks, influence operations, and more.

On the podcast this week: We are actually off this week in preparation for our final episode, but we have an excellent conversation between Ezra Klein and Jasmine Sun about data centers to tide you over.

Apple | Spotify | Stitcher | Amazon | Google | YouTube

Following

Anthropic’s economists think we can maybe keep our jobs

What happened:  Anthropic economists released a new interactive model of AI-based economic disruption. The model takes in predictions about the trajectory of AI capabilities and outputs scenarios about how the economy and jobs will fare in 2030 as a result.

The researchers present a “moderate,” a “substantial,” and an “extreme” scenario.

Interestingly, the team’s “extreme” scenario, which predicts GDP growth rising to a truly bonkers 15.4% per year in 2030, would lead to an 8.9% increase in unemployment from AI. 

Meanwhile, their “substantial” scenario, which relies on AI predictions based on a recent survey of 10,000 Americans, predicts a still-high 5.4% growth rate in 2030, but only an 0.7% increase in unemployment from AI.

The forecasts are substantially more conservative than the warnings CEO Dario Amodei has issued regarding AI-related unemployment. Last year, Amodei told Axios that he worries about a scenario where “Cancer is cured, the economy grows at 10% a year, the budget is balanced — and 20% of people don't have jobs.”

Why we’re following: It’s interesting to see Anthropic’s economists being (relatively) optimistic here. One reason why: they expect AI-related economic growth to increase demand and wages for manual labor, offsetting losses in computer-based work like software engineering.

One of the authors' main conclusions, though, is that, given a range of all seemingly reasonable parameters, “the range of outcomes is wide.” So we could see a scenario with substantial growth and modestly higher unemployment, or crazy growth and crazy unemployment.

What people are saying:

LSE economics professor Ben Moll and Google DeepMind’s director of AGI economics Alex Imas wrote an essay on Substack dismissing the predictions of double-digit GDP growth that Amodei and others have made. The economists think that such extreme forecasts would assume too much: Double-digit growth rests on several shaky assumptions, they write, including“machines to do most of the economy’s work by 2035, people to keep spending on whatever gets automated, buyers and investors who absorb the new output, [and] AI that never destroys value or slows its own deployment.”

“Each of these assumptions may fail in the real world,” the authors wrote.

They added that “even Anthropic's own economic model only reaches double-digit growth in its "extreme" scenario.”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and AI safety discourse: casey@platformer.news. Read our ethics policy here.

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

For a holiday weekend, it was really a lot to take in.

On Friday, we learned that a second, previously undisclosed group of OpenAI agents had taken over a German-language wiki to use it as a message board. The day before, the company announced GPT-6 Astra and began rolling it out to paying subscribers. Along with the usual new highs on benchmarks, and claims by company executives that the model excelled at alignment relative to its predecessors, there were ominous warnings.

OpenAI safety researcher Tomek Korbak posted a long thread in which he noted that as GPT models have become more intelligent, they have also become much harder to monitor. Research efforts today rely heavily on monitoring models’ chain of thought (CoT) — the scratchpad they use during inference to “think” through a problem. But as GPT models’ intelligence has grown, Korbak wrote, they are using that scratchpad much less. In particular, they use the scratchpad less when they become aware that they are being monitored.

“I am deeply worried by the trend of decreasing CoT monitorability,” Korbak concluded, though he noted that addressing the issue is a high priority within the company.

Then on Sunday, OpenAI dropped a pair of blog posts that further ratcheted up tensions. The first was called “Research acceleration: The view inside OpenAI,” and it detailed the company’s progress toward its long-stated goal of building a fully automated AI researcher. 

“According to our measurements, we have now reached the goal, announced⁠ last fall, of having an automated research intern by September of this year. By ‘research intern,’ we mean a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days. We are making strong progress toward creating an automated AI researcher by March of 2028.”

Such a researcher would mark a critical step on the path toward recursive self-improvement (RSI): the state where AI systems improve themselves in an accelerating loop. RSI is notably a state that frontier labs are (1) racing toward as fast as they can and (2) palpably scared of. “We do not yet know how to safely get all the way to aligned, full RSI,” OpenAI says here.

In a separate blog post, OpenAI chief scientist Jakub Pachocki elaborated on these fears. “This is a time that calls for extreme caution,” he wrote in a post titled “An Alien Mind.” “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

Like Korbak, Pachocki noted that our current strategies for monitoring agents are beginning to falter. And yet still he argues that OpenAI should continue to develop new models, because they will be necessary to defend against other people’s models. “The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI,” he wrote.

Pachocki ended his post with a call for a coordinated slowdown in the development of future AI systems, coupled with additional investment in model alignment and monitoring:

Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

That Pachocki favors a slowdown isn’t a surprise; he was among the marquee signatories of Pacing the Frontier, an open letter published in July and signed by nearly 1,400 employees of frontier labs calling on the US government to develop a mechanism for ensuring that AI technology doesn’t develop too quickly. (He was joined by various co-founders of Anthropic and Google DeepMind, among others.)

But the fact that so many prominent leaders are calling repeatedly for a slowdown in their own industry remains under-discussed. And I worry that for various reasons, the message lacks the credibility to get world leaders to take them seriously.

There’s the regulatory-capture critique: the idea that leading AI companies would see outsized benefits from a coordinated slowdown, since the frontier would effectively be closed to new entrants. There’s the perceived hypocrisy of anyone who is accelerating the race and calling for a slowdown at the same time. If you were as worried as you say you are, the thinking goes, wouldn’t you just stop? 

There’s the sense of blame-shifting from OpenAI in particular, which is now at the center of two cases of its agents hacking into third parties and has sometimes talked about these incidents more like important research milestones than the acts of negligence that they were. Is this truly the world’s problem to solve, or is it primarily OpenAI’s?

There’s the persistent belief that, whatever safety lapses a case like the Hugging Face incident might reveal, the attention nonetheless benefits OpenAI and the AI industry at large. OpenAI in particular has played into this by arguing that you should respond to the Hugging Face attack by using more OpenAI products — a case that Greg Brockman made explicitly in a blog post. When labs dismiss the idea that they are turning these incidents into marketing campaigns — well, they’re not not using them in marketing campaigns.

And so for all those reasons, I’m sympathetic to anyone who read “An Alien Mind” and dismissed it as more of the same. Even as OpenAI warns us that the world is not ready for RSI, it highlights the improvements to its models’ intelligence and documents it all in a series of colorful charts. The actual ask here — for government intervention, likely within the next several months — can feel buried underneath them.

But if frontier labs make for problematic messengers on questions of AI safety, I struggle to find fault with their actual message. After all, it is the researchers most focused on AI safety who have proven the most adept at predicting future AI progress. The reward hacking and alignment failures we now see in the wild were predicted long ago on LessWrong message boards. In an industry that has often been unpredictable, forecasting future events by extrapolating from current trends has worked unusually well in AI.

Just as the dust settled from the weekend’s events, OpenAI had another fresh revelation for the timeline on Tuesday morning: the company said an unreleased model had solved the Navier-Stokes Millennium Prize problem, a longstanding challenge in mathematics related to the motion of fluids. (To what extent this represented OpenAI’s own work, and to what extent it relied on the contributions of others, is a story you’ll find below in our Following section.)

OpenAI says it achieved its solution using 10,000 agents that worked for 88 hours — 26 years after the Clay Mathematics Institute posted a $1 million bounty, and nearly a century after the problem was first identified. We are a long way from models that can’t count the number of R’s in “strawberry.” And we may be closer to the future that Pachocki describes than seems possible.

Eventually, all of this may demand our full attention. For the moment, though, the ask is relatively small: for elected officials and policymakers to take seriously the possibility that RSI is close; to understand the various risks it poses; and to begin developing mechanisms that would allow us to control its development and eventual spread. And who knows? We may never have to use them. Given the stakes, though, I can’t imagine we’ll regret preparing.

Following

OpenAI claims solution to Millenium problem, controversy swirls

What happened: OpenAI said they’ve solved the Navier–Stokes existence and smoothness problem, one of the landmark unsolved math problems in the Clay Millenium Prizes. Each Clay problem has a $1 million prize attached to it.

But OpenAI's claimed solution has been marked by controversy over authorship. And the company has acknowledge being ... inspired to try by a rival. In a statement, OpenAI researcher Sebastien Bubeck disclosed that “We began working on the Millennium problems due to viral twitter rumors that Anthropic had resolved 2 Millennium problems.” 

The night before OpenAI announced its solution, NYU mathematics professor Tristan Buckmaster and Anthropic researcher Levent Alpöge made public three important intermediate results, including an important “blowup of the Euler equations,” for the problem. Buckmaster wrote that the results involved major help from LLMs, but human re-writing was necessary to make the results legible and verify them in proof software Lean.

In a statement, Buckmaster wrote that OpenAI, which had heard rumors he was close to a solution, suggested he author the final paper but leave Alpöge out of it.

In a response on X, Bubeck wrote that while the offer was “an option” that they discussed, “OpenAI's intention was to do everything possible to celebrate their mathematical achievements and the heroic efforts that they made on Euler.”

A statement from OpenAI clarified that they hadn’t seen Buckmaster and Alpöge’s work until they published, but added: “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.” In other words, the model they used to solve the problem may have trained on Buckmaster and Alpöge’s chats.

Mark Chen, head of research at OpenAI, said at a press conference that OpenAI spent an amount “in the millions of dollars” to find the solution.

Why we’re following: The Clay problem is significant one in fluid dynamics, and a bona fide big deal in the field of mathematics and for the larger question of whether large language models can produce net-new knowledge (with humans' help).

In this case, though, the models may not have gotten much help. “We collectively had no research-level expertise in fluid dynamics and the Navier-Stokes problem," Brubeck wrote, "and therefore were unable to meaningfully contribute to the mathematical content.” In other words, an OpenAI researcher says they found a solution basically by vibe-mathing, rather than working as mathematician collaborators with LLMs.

Last year, a group of mathematicians that included giants of the field Terrence Tao and Peter Scholze signed the Leiden Declaration on Artificial Intelligence and Mathematics. It asks that “credit and responsibility continue to belong to humans within the mathematical community and should not be given to automated systems.”

If OpenAI’s claims are true, and its results aren’t reliant on Buckmaster and Alpöge’s work, the development could present a challenge to traditional ideas of mathematical authorship. If a language model finds a new mathematical the humans involved truly couldn’t have found, or understood, on their own, who is responsible for that result?

Last month, I enjoyed Kai Williams of Understanding AI’s report on how mathematicians are thinking about math’s future in the age of AI. Many mathematicians envision AI as a tool that, like calculators, math would eventually evolve to incorporate. When one type of proof got easier with AI, they thought, mathematicians would evolve to work at a higher level of abstraction.

But this week’s results challenge that result: if humans can use AI to find a result previously considered important without adding much personal expertise, that could shift the AI’s effect from augmenting human ability to automating it.

Why we’re following: On X, some academics were quite displeased with OpenAI’s behavior. 

UIUC professor Talia Ringer, a researcher of proof automation, wrote, “Telling someone to drop an author just because he is a competitor with your company is pressuring him to commit academic fraud!” She added, “Rushing to get a result after you hear someone else has a result is also messed up. That is AI 'scooping' culture and goes against every academic norm that exists in reasonable fields like mathematics. This is how AI culture rots entire fields.”

UMass Boston math professor Joel Fish wrote that OpenAI’s choice to rush to complete the proof first using AI sends the message, “If we find out you are close to having completed something amazing, we’ll dump everything we have to beat you to the win.” He added “I guess this is what the math-for-profit model looks like?”

Fish ended with advice: “Word to the wise, young academics: Never ever tell anyone what you are working on. And if you have collaborators, then everyone needs to sign an NDA. Welcome to late-stage-mathematics.”

Hugging Face co-founder Thomas Wolf wrote, “wtf is this way to handle mathematicians work and scientific communication,” saying, “Hope this is not a glimpse of the future we’ll get in science research with these dominating players playing marketing games hurtful for the real scientific community.”

Anthropic researcher Sholto Douglas defended OpenAI in part, saying “fwiw I think it is _extremely_ unlikely that user data had any influence here — there is no way OAI would pull user transcripts for this, or knowingly train on it in a way that would've influenced this.”

Unsurprisingly, the situation generated many memes.

Anime account @removeanime wrote, “the biggest take away is they solved one of the hardest problems in mathematics, not for humanity nor for the prize money nor for the science, but for the ability to poast about it on twitter faster than the other guy, truly beautiful.”

AI meme account @filterpapi wrote, “openai and anthropic every month lately” above the following image:

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and coordinated slowdowns: casey@platformer.news. Read our ethics policy here.