比特币今日价格最新报价：4月8日比特币交易价格逼

Thought the open source AI references to camelids were finished? Think again: Yesterday, Together, a Menlo Park, California-based company focused on building a decentralized cloud and open source models, announced RedPajama (yes, like Llama Llama Red Pajama) yesterday.

“In many ways, AI is having its Linux moment,” the company said in a blog post, linking to a January post written by Chris Re, co-founder of Together, Stanford associate professor and co-founder of SambaNova, Snorkel.ai and Factory.

RedPajama is a collaborative project between Together, Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research, and MILA Québec AI Institute to create leading, fully open-source large language models (LLMs). Its effort began with yesterday’s release of a 1.2 trillion token dataset that follows the LLaMA recipe. The data enables any organization to pre-train models that can be permissively licensed. The full dataset is available on Hugging Face and users can reproduce results with Apache 2.0 scripts available on Github.

LLaMA is a state-of-the-art foundation?LLM released in February by Meta with gated access to researchers. Several other models based on LLaMA have come out in recent weeks, including Alpaca, Vicuna and Koala — but those models have not been available for commercial use. There was also some LLaMA-drama when the LLaMA model was leaked on 4chan.

bebe是什么牌子	今年是什么年庚	胃胀吃什么药效果最好	人中龙凤是什么意思	胃炎伴糜烂是什么意思
大学校长是什么级别	性功能下降吃什么药	舌头苦是什么原因	中指戴戒指什么意思	早上喝一杯温开水有什么好处
什么药降尿酸最好	疼痛科主要看什么病	c8是什么意思	人脱水了会有什么表现	什么是海市蜃楼
世界上最贵的车是什么车	卵巢保养最好的方法是什么	幼小衔接都学什么知识	胎动突然频繁是什么原因	菊花和金银花一起泡水有什么效果

验孕棒两条杠什么意思hcv8jop9ns2r.cn	情人节送什么花hcv9jop1ns5r.cn	悬钟为什么叫绝骨hcv8jop2ns6r.cn	小孩脱发是什么原因引起的bysq.com	美国的国鸟是什么cl108k.com
glu是什么氨基酸hcv8jop7ns5r.cn	如意丹的作用是什么hcv8jop8ns6r.cn	狗的鼻子为什么是湿的hcv9jop1ns7r.cn	囊肿里面是什么东西hcv9jop2ns3r.cn	罚的部首是什么mmeoe.com
精华液是什么hcv8jop4ns7r.cn	农历六月十七是什么星座hcv9jop4ns9r.cn	晨对什么hcv8jop2ns0r.cn	紫得什么hcv9jop7ns0r.cn	手掌中间那条线是什么线cl108k.com
yl是什么牌子hcv9jop6ns9r.cn	pvs是什么意思hcv9jop0ns3r.cn	咳嗽喝什么饮料hcv9jop0ns8r.cn	斯什么意思hcv7jop6ns8r.cn	梦见自己结婚了是什么征兆hcv9jop5ns2r.cn

In the coming weeks, Together will release a full suite of LLMs and instruction tuned versions based on the RedPajama dataset. The company emphasized that the forthcoming models will be fully open-source and commercially viable. In a tweet, the company said, “We hope this can be a clean-room, drama-free version. The RedPajama models we release, starting in the coming weeks, will be released under the Apache 2.0 license.”

RedPajama part of a wave of open source AI

As VentureBeat reported last week, open source AI has been having a moment over the past few weeks, following the wave of LLM releases and an effort by startups, collectives and academics to push back on the shift in AI to closed, proprietary LLMs.?

And a camelid-adjacent model, Dolly 2.0 (as in Dolly the Sheep), also made headlines last week when its developer, Databricks, called it the first open, instruction-following LLM for commercial use.

But the largest, state-of-the-art open source LLMs like LLaMA have been limited to the research community. “They are limited in that you can’t build real applications and ship them,” said Vipul Ved Prakash, founder and CEO of Together and previously cofounder of Cloudmark and Topsy. “We think having permissively licensed models is a critical aspect of open source AI.”

Replicating the LLaMA dataset was no small task

The company started with LLaMa, which it called the “leading suite of open base models,” because it was trained on a “very large dataset that was carefully filtered for quality.” Also, the 7 billion parameter LLaMA model is “trained for much longer, well beyond the Chinchilla-optimal point, to ensure the best quality at that model size.”

While neither the dataset nor the model will be identical, the developers aim to create a fully open source reproduction of LLaMA which would be available for commercial applications, and provide a “more transparent pipeline for research.”

The developers did not have access to the LLaMA dataset but had enough of a recipe to go on. “We followed the recipe very carefully to essentially recreate [the LLaMA dataset] from scratch,” said Prakash. The dataset consists of seven data slices, including data from Common Crawl, arxiv, Github, Wikipedia and a corpus of open books.

“For each data slice, we conduct careful data pre-processing and filtering, and tune our quality filters to roughly match the number of tokens as reported by Meta AI in the LLaMA paper,” read the blog post.

“All of the data LLaMA was trained on is openly available data, but the challenge was that they they didn’t provide the actual data set — there’s a lot of work to go from the overview to the actual data set,” said Prakash. For example, he explained, the paper might describe how they picked the best 10,000 from a million documents, but they didn’t give you the 10,000. “So we followed the recipe to repeat all that work to create an equivalent dataset,” he said.

The debate over building transparent systems

Prakash said that the RedPajama project collaborators believe it’s important that systems are transparent. “You know exactly how this model was built, what went into it,” he said. “If you’re trying to improve it, you can start from the dataset.”

The project also brings together a larger community to these models, he added. “I would say academia has really been cut out of foundation model research because of the level of resources required, starting from data to the compute,” he said. He added that there is a small number of people in the world working on these large models today, and if there was broader access, “a lot of brilliant people” around the world would be able to explore different directions of neural architectures, training algorithms and safety research.

“Also, this is one of the first really general AI which can be adapted to different tasks, and we think the applicability is very broad,” he said. “But many different applications are possible only if you have access to the model, the model weights, and adapt them to different computing environments. We see a lot of this happen because of open source AI.”

There is another side to the open source AI debate, however. For example, Ilya Sutskever, OpenAI’s chief scientist and co-founder, recently said it was “wrong” to share research so openly, saying fear of competition and fears over safety — were “self-evident.” He added that “at some point it will be quite easy, if one wanted, to cause a great deal of harm with those models.”

And in a recent interview with VentureBeat, Joelle Pineau, VP of AI research at Meta, said that while accountability and transparency in AI models is essential, the key for Meta is to balance the level of access, which can vary depending on the potential harm of the model.

“My hope, and it’s reflected in our strategy for data access, is to figure out how to allow transparency for verifiability audits of these models,” she said, adding that access could be decided based on the level of potential harm of the model.

On the other hand, she said that some levels of openness go too far. “That’s why the LLaMA model had a gated release,” she explained. “Many people would have been very happy to go totally open. I don’t think that’s the responsible thing to do today.”

Debates around ethical datasets as well

There have also been debates about the ethics of the datasets themselves, whether the models are open or closed. An article last week in The Guardian said that the “enormous datasets used to train the latest generation of these AI systems, like those behind?ChatGPT?and Stable Diffusion, are likely to contain billions of images scraped from the internet, millions of pirated ebooks, the entire proceedings of 16 years of the European parliament and the whole of English-language Wikipedia.”

But Prakash says that he thinks “these models capture in some ways the output of human society and there is a sort of obligation to make them open and usable by everyone.” He added that “most of the magic” of these models comes from the fact that they are trained on “really broad and vast” data.

He also pointed out that the original data is compressed significantly in the actual model. The RedPajama dataset is 5 terabytes, and the models can be as small as 14 GB, ~500x smaller than the original data they are modeling.

“This means that knowledge from the data is abstracted, transformed and modeled in a very different representation of weights and biases of parameters in the neural network model, and not stored and used in its original form,” said Prakash. So, it is “not reproducing the training data — it is derivative work on top of that. From our understanding, it is considered fair use as long as the model is not reproducing the data — it’s learning from it.”

There is no doubt that the open source AI debates are highly-complex. But when asked why the company called the new project RedPajama, the answer was far more simple. “A lot of us have small children,” said Prakash. “It just seemed fun.”