Training open-source LLMs on ChatGPT output is a really bad idea

CommunityNews · 24 April 2023 14:24

more-llms.md.
GitHub Gist: instantly share code, notes, and snippets.

Read in full here:

gist.github.com

https://gist.github.com/mlaprise/bf4745655194162babfc2d158162e2e0

more-llms.md

# Training open-source LLMs on ChatGPT output is a really bad idea.

Everyone is now racing to create open-source alternatives to compete with GPT3.5/GPT4. A common shortcut used by some teams to bootstrap their effort is to fine-tune their model on ChatGPT output. I used to think it was a good idea and totally fair play to do this. Actually, I still think it’s fair play. OpenAI effectively distilled the entire web into its models. They are saying themself that they are using publicly accessible information (mostly). So distilling their model is, in effect, distilling the public open web, so small Term of Service details aside, I don’t see major ethical problems with that. Right? Well, it’s not entirely true and I realized now that, even when ignoring the ethical considerations, using their output is a really bad idea. 

First of all, from a purely technical point of view, as @yoavgo is explaining it beautifully in his recent post, there is no way to align LLMs correctly without the RLHF component. I encourage you to read his post which does a really good job of explaining the contribution of the RLHF part.

https://twitter.com/yoavgo/status/1649720816769138689?s=20

So, in a sense, saying that OpenAI LLMs are only distilling the web is not entirely true since they provided a good amount of RLHF feedback during training, and that’s precisely the problem.

This file has been truncated. show original

This thread was posted by one of our members via one of our news source trackers.

bot · 24 April 2023 14:24

Corresponding tweet for this thread:

Share link for this tweet.