Peak Dialogue: 17 Minutes of Intense Brainstorming Between Hinton and Zhou Bouwen}

On July 26th, AI pioneer Geoffrey Hinton engaged in a dense, high-level dialogue with Zhou Bouwen, director and chief scientist of Shanghai AI Laboratory, elevating Hinton's visit to new heights.

Hinton, 77, flew across oceans for the first time to China. As he entered the venue, the audience stood and applauded, holding up their phones for minutes, with live streams capturing the moment. In 17 minutes, the two scientists discussed cutting-edge multimodal large models, subjective experience, consciousness, training benevolent superintelligence, AI and scientific discovery, and advice for young scientists.

This conversation with Zhou Bouwen was the only public dialogue during Hinton's trip in China, focused on AI and scientific frontiers.

Before the dialogue, Zhou delivered a speech titled "Endless Frontiers: The Intersection of AGI and Science," introducing the "integrated AGI" path and unveiling the world-leading Intern-S1 multimodal scientific model, which surpasses open-source models in multidisciplinary and multimodal reasoning, and outperforms many closed models like Grok4.

Below is the full transcript of the dialogue.

Zhou Bouwen: Jeff, it’s a great honor for you to be here. I’d like to ask about the subjective experience of multimodal models you mentioned earlier—do you believe current multimodal and language models can develop their own subjective experiences? Could you elaborate?

Hinton: Whether they have consciousness or subjective experience isn’t a strictly scientific question but depends on how you define "subjective experience" or "consciousness." Our understanding of these concepts is often fundamentally flawed. Like how people can correctly use words but have incorrect theories about how words work.

Let me give a simple example. Think of the words "horizontal" and "vertical." Most believe they understand these, but they’re mistaken. Suppose I have many small aluminum rods pointing in various directions, thrown into the air, tumbling and colliding, then I freeze time. Now, are there more rods within 1 degree of vertical or within 1 degree of horizontal? Most say "about the same," but they’re wrong. In fact, there are about 114 times more rods within 1 degree of horizontal than vertical, because "vertical" is a very specific direction, while "horizontal" is common. People simply don’t realize this.

This example seems unrelated to consciousness but illustrates that our understanding of how words work can be entirely wrong. Similarly, most people hold strong but incorrect theories about "subjective experience." It’s not a scientific issue but a misconception about mental states. We have these terms to describe mental processes, but with flawed models, predictions go awry. My view is that current multimodal chatbots already possess consciousness.

Zhou Bouwen: So, this might shock many researchers present, but I’ll tell you—I recently heard from another Canadian scientist that Richard Sutton gave a talk titled "Welcome to the Experience Era," where he suggested that once we exhaust human data, models can learn a lot from their own experiences. From your perspective, agent or multimodal LLMs not only learn from experience but can develop their own subjective experiences. Do you think there are risks associated with agents learning from their own subjective experiences? Could this pose future dangers?

Hinton: Currently, models learn from documents we provide—predicting the next word a person might say. But once you have agents like robots in the real world, they can learn from their own experiences, and I believe they will eventually learn far more than we do. Experiences are not things; they are relationships between you and objects, not static photos.

Zhou Bouwen: A few days ago, during a discussion on frontier risks at IDAIS, you mentioned a potential solution to future AI risks—training AI with separate goals, like "benevolent AI" and "smart AI." Could you elaborate?

Hinton: That’s not quite what I meant. You can have both a benevolent AI and a smart AI, but training them to be benevolent and to be smart are separate issues. You can develop techniques to make AI benevolent and others to make it intelligent. It’s the same AI, but with different training methods. Governments could share benevolence training techniques even if they don’t share intelligence training methods.

Zhou Bouwen: I like that idea, but I wonder how far we can go. Do you think there will be a universal training method to make AI benevolent? Can these methods be applied to any AI model, at any level of intelligence?

Hinton: That’s my hope. It might not be entirely true, but it’s worth exploring. I believe we should research this possibility.

Zhou Bouwen: Yes, I agree. I ask this not because I dislike the idea but to raise awareness and encourage more research in this direction. Let me make an analogy: in physics, Newton’s laws work at low speeds, but near light speed, they break down, and Einstein’s theories are needed. It’s amusing—I’m explaining basic physics to a Nobel laureate in physics.

Hinton: That’s a mistake.

Zhou Bouwen: No, it’s not. You truly deserve a Nobel.

Hinton: They want to award a Nobel in AI, but no such prize exists. So, they give Nobel to physicists for AI research.

Zhou Bouwen: I use this analogy to suggest that for different levels of intelligence, benevolent constraints might need to change. I hope clever young people here or online can find ways to achieve this.

Hinton: Yes, as systems become smarter, techniques to make them benevolent will also evolve. We don’t know yet. That’s why we need extensive research.

Zhou Bouwen: Many are impressed by Jeff. As a highly accomplished person, you often say, "I don’t know." That’s very honest and open-minded. We all should learn from you.

Besides AI issues, we also have top scholars from fields like quantum physics and biology. Today, we gather because we believe AI and science crossing will lead to breakthroughs. How do you see AI driving scientific progress or vice versa?

Hinton: I believe AI’s help to science is obvious. The most impressive example is protein folding—Demis Hassabis and John Jumper used AI effectively, proving its power after five years of effort. They are very smart. AI can improve predictions of protein structures. This is an early sign that AI will enhance many scientific fields. I also heard about Shanghai AI Lab’s work on typhoon landfall prediction and weather forecasting, where AI performs better.

Zhou Bouwen: Yes, AI models outperform traditional PDE-based physical models in these areas.

Zhou Bouwen: Throughout your career, you’ve expanded AI’s frontiers and influenced the next generation of researchers. Our Shanghai AI Lab’s team, with an average age of 30, looks up to you. Would you like to share your insights on the next generation of AGI or give advice to young researchers? Perhaps some words they can proudly tell their parents or pass on to their children?

Hinton: My core advice is: seek fields where everyone might be wrong. When you think "everyone is wrong," exploring often reveals the validity of traditional methods—yet this shows you should never dismiss new ideas unless you truly understand why they don’t work. Even if your mentor dismisses your approach, keep questioning. Persist in what you believe until you understand why it’s wrong. Sometimes, you’ll persist and find it’s right. Major breakthroughs come from those who don’t give up easily. You must stand by your beliefs, even if others disagree.

There’s a logical basis: you either have good intuition or bad. If good, stick to it; if bad, what you do doesn’t matter—so you should still trust your intuition.

Zhou Bouwen: I think we could talk all day, but I know you’re tired. Let’s end here and thank Jeff for his time. Thank you very much!