OpenAI’s Newest Tool Explains Why Language Models Behave In Certain Ways

When talking to a chatbot, you may sometimes come across either a genius response from it, or it might say something completely unrelated that will have you wondering where along the lines it connected those dots. OpenAI’s newest tool is pursuing a more transparent relationship with users and its language learning models (LLM) by creating a tool that can identify why these machines behave in specific ways.
 
LLMs have “neurons” that help them pick up on certain patterns, just like the human brain. When a prompt is typed into them, they can highlight keywords, and trace them back to different neurons that hold information on the subject. In the example used by OpenAI, if you ask it about the Marvel universe, it will pick up on “Marvel neurons” and present relevant (and sometimes not so relevant) information as it answers.
 
This tool can examine the tiny parts of a chatbot’s brain, and explain why it responded a certain way. For example, it looks at the words the chatbot processed, and waits for a specific part of the neurons to “light up.” Then, it sends that information to GPT-4, which generates an explanation.
 
GPT-4 checks if the explanation is correct by giving it more words to process, and sees if it can predict the same behavior. If it matches, then the explanation is accurate.
 
“Using this methodology, we can basically, for every single neuron, come up with some kind of preliminary natural language explanation for what it’s doing and also have a score for how well that explanation matches the actual behavior,” Jeff Wu, scalable alignment team leader at OpenAI, said.
 
In total, all 307,200 neurons could be explained via the tool. Such a system could one day open up a better understanding of why biases still run rampant in artifical intelligence, and how to crack down on them. However, OpenAI admits it’s still far off in this regard. That said, it is now available for others to use and refine on GitHub.
 
 
 
[via TechCrunch and SiliconANGLE, Photo 246538875 © Rafael Henrique | Dreamstime.com]