It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.
OpenAI turning out to be a total bait and switch. Especially true when your co-founder is actively calling you out on it[1]
Remember kids: if it's not a non-profit organization it is a _for_ profit one! It was silly to expect anything else:
> In 2019, OpenAI transitioned from non-profit to for-profit. The company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment package of US$1 billion into the company. OpenAI then announced its intention to commercially license its technologies, with Microsoft as its preferred partner [2]
I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything.
Of course they can - just because you contribute to open source, and do that because you also benefit from open source projects, doesn't mean you have to do absolutely everything under open source.
Especially considering OpenAI isn't even Microsoft's IP or codebase.
> I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything.
I'm not claiming that. Of course there is place for closed and open elements of their offerings. Let me clarify.
In the past, Microsoft was very aggressive about open source. When they realized this strategy of FUD brings little result, they changed their attitude 180 and decided to embrace it putting literal hearts everywhere.
Personally, I find it hypocritical. There is no love/hate, just business. They will use whatever strategy works to get their advantage. What I find strange is that people fell for it.
But why on this thread then, about GPT-3? It's not even their own company, IP or source to give away.
But even when Microsoft can't open source it because it's not theirs, we still have people posting in this thread that this is further evidence that Microsoft is hypocritical. It sounds a lot like a form of Confirmation Bias to me where any evidence is used as proof that Microsoft is 'anti-open-source'.
In my experience, I work at a completely different company than the one that Ballmer ran. Nearly everyone I talk to speaks of the "Ballmer era" in a negative light, and confirms that Satya literally turned the entire company on its head.
Many things happen every day that would never have happened under Ballmer.
It’s hostile to the GPL licence which means anything licensed under GPL can’t be used in Microsoft’s proprietary products.
I would personally say Microsoft wasn’t necessarily driven by anti open-source hate necessarily, they were just very anti-competitor. Microsoft tried to compete with their biggest competitor? Colour me shocked.
I don't think this should be seen in the light of "open source everything" but more that many see Microsoft doing open source not as part of "being good" but part of their age old "embrace extend extinguish" policy.
I don't know where people got the idea that companies can be "sincere." Sincerity is faithfully communicating your mental state to others. A company's mental state can change on a dime based on the decisionmaking of people who rely on the company for the degree of profit it generates. Any analog to sincerity that you think you see can probably be eliminated by firing one person after an exceptionally bad quarter (or an exceptionally good one.)
Like many other companies, Microsoft loves unpaid labor.
Free Software is about giving freedom and security all the way to the end users - rather than SaaS providers.
If you remove this goal and only focus on open source as a development methodology you end up with something very similar to volunteering for free for some large corporation.
It means restricted stock unit, and it's a kind of company stock unit that may be distributed to some "valued" employees. There is usually a vesting schedule, and you can't do whatever you want with it.
But I was told GPT-3 was too powerful for mere mortal hands (unless you have an account!) and that it would be used for hate speech and to bring about skynet.
How will this project avoid those terrible outcomes?
By putting the cat back in the bag. Oh, it's too late ... useless to think about it - we can't de-invent an invention or stop people from replicating. Its like that time when NSA wanted to restrict crypto.
It's a recurring theme in OpenAI research, they become more and more closed.
For instance their latest model called DALL·E hit the headlines before the release of the paper. Needless to say, the model is not available and no online demo has been published so far.
Because its winner-take-all in this research, not "winner-take-some".
Andrew Yang talked about this and why breaking up Big Tech won't work. No one wants to use the second best search engine. The second best search engine is Bing and I almost never go there.
Tech isn't like automobiles, where you might prefer a Honda over a Toyota, but ultimately they're interchangeable. A Camry isn't dramatically different and doesn't perform dramatically better than an Accord. Whoever builds the best AI "wins" and wins totally.
But they still released the CLIP model which is the complement of DALL-E and used in the DALL-E pipeline as a final filter. There are collabs with CLIP floating around and even a web demo.
Thank you for this info, as you mentioned CLIP is used for re-ranking DALL-E outputs, by itself it is just an (image, text) pairs classification network.
Really. OpenAI assembled some of the best minds from the deep learning community. The problem isn't that they are a for-profit SaaS, the problem is they lied.
I don't really care if OpenAI offers commercial licenses as long as the underlying research is truly open. This way alternative options will become available eventually.
Arguably openAI is one of the most closed industry AI labs (among those that are still participating in the research community), on par only with deep mind (though deepmind at least publishes way more). Funnily enough, FAIR and Google Brain have a vastly better track record wrt. publishing not only papers but also code and models.
It was probably bait and switch to hire top researchers and get initial funding. Now that OpenAI is a household name, they don't have to pretend anymore.
I buy the former, researchers might be happier knowing their work potentially benefits all of humanity, not just a bunch of investors. But wouldn’t it be more difficult to get funding as a non-profit?
It's just never going to be difficult to get funding when you have Elon Musk and Sam Altman as founders (and even more so when founders put one billion of their own money into it).
Sure, but that's OpenAI's particular set of circumstances. Generally speaking I struggle to see investors preferring a nebulous non-profit over a for-profit with a clear path to market.
Of course. It's just that the comment I've been responding to suggested OpenAI going the "open"/non-profit route was to 1) get top researchers and 2) get investment. I was arguing that this doesn't seem to (generally) be a good way to get investment, but I agree with you in that in their case investment just wasn't a consideration at all.
Also gpt2 models and code at least were publicly released and so has a lot of their work.
And yes, they realized they can achieve more by turning for profit and partnering with Microsoft. So true, they are not fully 'open' but pretending they don't release things to the public and making the constant 'more like closedai aimirite' comments is getting old.
OpenAI was built to influence the eventual value chain of AI in directions that would give the funding parties more confidence that their AI bets would pay off.
This value chain basically being one revolving around AI as substituting predictions and human judgement in a business process, much like cloud can be (oversimply) modeled as moving Capex to Opex in IT procurement.
They saw that, like any primarily B2B sector, the value chain was necessarily going to be vertically stratified. The output of the AI value chain is as an input to another value chain, it's not a standalone consumer-facing proposition.
The point of OpenAI is to invest/incubate a Microsoft or Intel, not a Compaq or Sun.
They wanted to spend a comparatively small amount of money to get a feel for a likely vision of the long-term AI value chain, and weaponize selective openness to: 1) establish moats, 2) Encourage commodification of complementary layers which add value to, or create an ecosystem around, 'their' layer(s), and 3) Get insider insight into who their true substitutes are by subsidizing companies to use their APIs
As AI is a technology that largely provides benefit by modifying business processes, rather than by improving existing technology behind the scenes, your blue ocean strategy will largely involve replacing substitutes instead of displacing direct competitors, so points 2 and 3 are most important when deciding where to funnel the largest slice of the funding pie.
_Side Note: Becoming an Apple (end-to-end vertical integration) is much harder to predict ahead of time, relies on the 'taste' and curation of key individuals giving them much of the economic leverage, and is more likely to derail along the way._
They went non-profit to for-profit after they confirmed the hypothesis that they can create generalizeable base models that others can add business logic and constraints to and generate "magic" without having to share the underlying model.
In turn, a future AI SaaS provider can specialize in tuning the "base+1" model, then selling that value-add service to the companies who are actually incorporating AI into their business processes.
It turned out, a key advantage at the base layer is just brute force and money, and further outcomes have shown there doesn't seem to be an inherent ceiling to this; you can just spend more money to get a model which is unilaterally better than the last one.
There is likely so much more pricing power here than cloud.
In cloud, your substitute (for the category) is buying and managing commodity hardware. This introduces a large-ish baseline cost, but then can give you more favorable unit costs if your compute load is somewhat predictable in the long term.
More importantly, projects like OpenStack and Kubernetes have been desperately doing everything to commodotize the base layer of cloud, largely to minimize switching costs and/or move the competition over profits up to a higher layer. You also have category buyers like Facebook, BackBlaze, and Netflix investing heavily into areas aimed at minimizing the economic power of cloud as a category, so they have leverage to protect their own margins.
It's possible the key "layer battle" will be between the hardware (Nvidia/TPUs) and base model (OpenAI) layers.
It's very likely hardware will win this for as long as they're the bottleneck. If value creation is a direct function of how much hardware is being utilized for how long, and the value creation is linear-ish as the amount of total hardware scales, the hardware layer just needs to let a bidding war happen, and they'll be capturing much of the economic profit for as long as that continues to be the case.
However, the hardware appears (I'm no expert though) to be something that is easier to design and manufacture, it's mostly a capacity problem at this point, so over time this likely gets commoditized (still highly profitable, but with less pricing power) to a level where the economic leverage goes to the Base model layer, and then the base layer becomes the oligopsony buyer, and the high fixed investment the hardware layer made then becomes a problem.
The 'Base+1' layer will have a large boom of startups and incumbent entrants, and much of the attention and excitement in the press will be equal parts gushing and mining schaudenfreude about that layer, but they'll be wholly dependent on their access to base models, who will slowly (and deliberately) look more and more boring apart from the occasional handwringing over their monopoly power over our economy and society.
There will be exceptions to this who are able to leverage proprietary data and who are large enough to build their own base models in-house based on that data, and those are likely to be valuable for their internal AI services preventing an 'OpenAI' from having as much leverage over them and being much better matched to their process needs, but they will not be as generalized as the models coming from the arms race of companies who see that as their primary competitive advantage. Facebook and Twitter are two obvious ones in this category, and they will primarily consume their own models, rather than expose them as model-as-a-service directly.
The biggest question to me is whether there's a feedback loop here which leads to one clear winning base layer company (probably the world's most well-funded startup to date due to the inherent upfront costs and potential long-term income), or if multiple large, incumbent tech companies see this as an existential enough question that they more or less keep pace with each other, and we have a long-term stable oligopoly of mostly interchangeable base layers, like we do in cloud at the moment.
Things get more complex when you look to other large investment efforts such as in China, but this feels like a plausible scenario for the SV-focused upcoming AI wars.
Apparently you don't need to be a large company to train GPT-3. EleutherAI is using free GPU from CoreWeave, the largest North American GPU miner, who agreed to this deal to get the final model open sourced and have their name on it. They are also looking at offering it as an API.
I think it's great they're doing this, but GPT-3 is the bellwether not the end state.
Open models will function a lot like Open Source does today, where there are hobby projects, charitable projects, and companies making bad strategic decisions (Sun open sourcing Java), but the bulk of Open AI (open research and models, not the company) will be funded and released strategically by large companies trying to maintain market power.
I'm thinking of models that will take $100 million to $1 billion to create, or even more.
We spend billions on chip fabs because we can project out long term profitability of a huge upfront investment that gives you ongoing high-margin capacity. The current (admittedly early and noisy) data we have about AI models looks very similar IMO.
The other parallel is that the initial computing revolution allowed a large scale shift of business activities from requiring teams of people doing manual activities, coordinated by a supervisor towards having those functions live inside a spreadsheet, word processor, or email.
This replaces a team of people with (outdated) specializations with fewer people accomplishing the same admin/clerical work by letting the computer do what it's good at doing.
I think a similar shift will happen with AI (and other technologies) where work done by humans in cost centers is retooled to allow fewer people to do a better job at less cost. Think compliance, customer support, business intelligence, HR, etc.
If that ends up being the case, donating a few million dollars worth of GPU time doesn't change the larger trends, and likely ends up being useful cover as to why we shouldn't be worried about what the large companies are up to in AI because we have access to crowdsourced and donated models.
I think calling this a "wild-ass guess" undersells it a bit (either that or we have very different definitions of a WAG).Very well though-through and compelling case.
My biggest question is whether composable models are indeed the general case, which you say they confirmed as evidenced by the shift away from non-profit. It's certainly true for some domains, but I wonder if it's universal enough to enable the ecosystem you describe.
This is neat, but almost no startups of any kind, even mid size corps, have such complicated and intricate plans.
More likely: OpenAI was a legit premise, they started to run out of money, MS wanted to license and it wasn't going to work otherwise, so they just took the temperature with their initial sponsors and staff and went commercial.
