The ongoing tug of war between open and closed AI

Should AI be open or closed? This is one of the most confusing debates raging in AI discourse at the moment. Those pursuing ‘open’ models are positioned as a counterweight to Big Tech, as if openness can democratize AI for the li’l guy. At the same time, Meta launched Llama, an open-source model — and Meta isn’t exactly the li’l guy. Many call open models “dangerous.” To top it off, the EU’s AI Act offered broad exemptions for “open-models” and a group of thoughtful academics recently signed a joint statement in support of openness in AI. Confused yet? Let’s dig in.

First things first — what is meant by ‘open?’ You can read the full definition by the Open Source Initiative, but to summarize, typically a technology is considered open if it is:

• Transparent: Can I access the code, documentation, and data that compose the technology?

• Reusable: Does the licensing of the technology allow third parties to reuse the code?

• Extensible: Can I build on top of the technology?

You can think of openness as a spectrum but basically, if the answer is ‘yes,’ to all three, the technology is said to be open source (or just ‘open’ for short). Now, anyone using the internet right now is benefiting from open-source software — Linux is open source, the Apache Web Server is open source, etc. This is why openness is often synonymous with democratizing access to technology and innovation: it allows for start-ups and developers to draw upon and build on top of the code of others. Moreover, academic researchers tend to value openness because if systems are transparent, they can be more easily audited. As researcher Deb Raji rightly wrote, “We can only critique and fix what we can see.”

How this plays out in practice, of course, isn’t so simple. In a great new paper, “Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI,” researchers David Gray Widder, Meredith Whittaker, and Sarah Myers West document the history of open-source vis-a-vis corporate power. For example, in 2019, Amazon took a popular open-source database hosted by MongoDB, made it proprietary, and sold it to clients as part of its AWS cloud offering. The history of open-source is littered with examples like this, wherein, as Duncan McCann wrote, an “open regime can lead to increased innovation and economic activity […] in the end the major benefits are being harvested by a small group of companies.”

As I’ve written before, decentralized technology does not decentralize power in crypto. The same principle applies here: open-source technology does not democratize power in AI. This is even more of a challenge with AI than it is with regular ol’ software for one very important reason: the compute power and data required to run AI systems at scale are massive. Widder et al. sum it up nicely:

“In short, the computational resources needed to build new AI models and use existing ones at scale, outside of privatized enterprise contexts and individual tinkering, are scarce, extremely expensive, and concentrated in the hands of a handful of corporations, who themselves benefit from economies of scale, the capacity to control the software that optimizes compute, and the ability sell costly access to computational resources. This significant resource asymmetry undermines any claims to democratization that the availability of ‘open’ AI models might be used to support.”

So, a generative AI model might be theoretically open in that it can be reused or built upon, but you still have to have gobs of resources and compute power to implement these models at scale. This isn’t just about compute power — since data is not in infinite supply, high-quality, proprietary data will become increasingly important. Check out the contract OpenAI penned with Springer in December — these companies have already trained their models on all the free data they can get their hands on, so from here on out the data training costs will be expensive. The next time you read an article suggesting that openness might democratize the playing field or diffuse Big Tech’s power, a red flag should go off in your brain.

Okay, so if AI models require all these resources, why would Meta, which has lots of resources, ostensibly share them with other builders by making Llama 2 open? Because it gives them even more power. Note that this model isn’t fully open; Meta has offered access to the model’s weights, evaluation code, and documentation, but its license isn’t recognized by the Open Source Initiative and it doesn’t pass the transparency test. For example, we have no idea what data Meta used to train the model for Llama. Nevertheless, Meta is making its development framework open to ensure AI developers and researchers build tools and frameworks that “Lego-like snap into place with their own company systems,” as Widder et al. write. In short, Meta wants to make just enough of its model open to leverage the labor of others, and to extend the power of its platform.

Okay, so companies like Meta wield openness in the name of democracy and innovation, masking their own interests. But then there are companies like OpenAI who argue it’s too dangerous to make their models open. Here’s co-founder Ilya Sutskever on the decision to close GPT-4:

“If you believe, as we do, that at some point, AI — AGI — is going to be extremely, unbelievably potent, then it just does not make sense to open-source. It is a bad idea... I fully expect that in a few years it’s going to be completely obvious to everyone that open-sourcing AI is just not wise.”

So what are the dangers posed by ChatGPT that we must not disclose? A new paper by Bommasani et al., “Considerations for Governing Open Foundation Models” outlines the marginal risks — relative to closed models and existing technologies — to open models:

• Disinformation: ChatGPT drives the cost of generating persuasive, targeted disinformation to zero. Yes, that contributes to an information ecosystem that advantages the liar — they can more easily cast doubt on lies or misbehavior that is actually true. But as the paper correctly states, “the key bottleneck for effective influence operations is not disinformation generation but disinformation dissemination.” Right, reducing the marginal cost to zero doesn’t do much harm if millions of people don’t see it. We’re right back to my second-ever essay on Untangled and the problem of amplification and scale, and regulating the reach of platforms.

• Biorisk: The authors argue that the risk that an open version of ChatGPT will enable anyone to make a bioweapon is misguided. Sure, if you ask a large language model to help you make a bioweapon, it will often spit out that information. But it’s able to do that in part because that information is available on the open web — Wikipedia, National Academies of Sciences, Engineering, and Medicine. Not unlike disinformation, the risk of a bioweapon isn’t posed by information alone — it’s the practical use of that information at scale. As the authors point out, one would need to develop pathogens, which requires access to a lab, equipment, expertise, etc. And the production of bioweapons is already closely monitored and regulated.

• Child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII): By contrast, the authors note that open models might indeed present greater risks when it comes to CSAM and NCII because the resources required for open text-to-image models are much lower.

The paper goes on to list other marginal risks, but the broader point is this: companies such as OpenAI spin a narrative about the ‘dangers’ posed by their chatbot, and use this to justify closing off their models to additional transparency. With this in mind, we should ask ourselves: would open-sourcing these models increase the marginal risk of the problem(s)? This requires a much deeper understanding of the problem space than those warning about the existential dangers seem to have, and often the answer is “not really.”

While I agree with ‘team open’ on most points, we shouldn’t let the open vs. closed debate confuse our understanding of what’s going on. Often, these terms are being flung around by companies to advance their interests and power. As AI ethicist and researcher Margaret Mitchell argues, what matters is instead about the context in which the system is being used, and how data is stored and accessed. Ultimately, Mitchell writes, “The answer isn't in declaring open or closed technology bad — that's a distraction that centers the technology. The answer is in figuring out what mechanisms must be in place — in open AND closed systems — to ensure the rights of the people.

I couldn’t agree more! This debate shouldn’t center the technology — it should center the rights of people, and the societal objectives we think AI might reasonably help us achieve. That would open a new conversation about how we protect people’s rights, how we hold companies accountable for decisions that lead to real harm, what problems might actually benefit from AI, and how to make the governance of these systems much more democratic.