In this thought-provoking conversation between Lex Fridman and Mark Zuckerberg, the two delve into the profound implications of the metaverse for the future of humanity. They discuss how the metaverse is reshaping our perception of reality, allowing for intimate and immersive digital interactions that blur the boundaries between physical and virtual worlds. With a focus on the psychological and social benefits of this evolving technology, they explore the challenges and exciting possibilities that lie ahead as the metaverse continues to develop. Their engaging dialogue offers insights into the transformative power of the metaverse and its potential to redefine how we connect and communicate in the digital age.
Mark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast
Realism in the Metaverse: Both Lex and Mark express their amazement at the level of realism in the Metaverse, where they can interact as if they are in the same room, even though they are physically apart.
Photorealistic Avatars: They discuss the technology behind their photorealistic avatars, which are part of a research project at Meta (formerly Facebook). These avatars are created by scanning their faces and bodies and then encoding the expressions into a codec.
Bandwidth Efficiency: Mark explains that this technology is more bandwidth-efficient than transmitting full video or 3D immersive scenes. It captures the subtleties of the human face, including imperfections, which contribute to immersion.
Importance of Eye Expressions: They highlight the importance of capturing eye expressions and subtle facial movements, as these are crucial for conveying emotions and communication.
Goal of Accessibility: Mark discusses the goal of making this technology accessible to more people by streamlining the scanning process. He envisions a future where people can create photorealistic avatars with just a smartphone in a matter of minutes.
Use Cases: They talk about the potential use cases for the Metaverse, such as remote meetings, gaming, and social interactions. Mark emphasizes that the Metaverse will enable experiences that go beyond traditional video calls, where physical presence can be felt even when people are apart.
Challenges and Progress: Mark acknowledges that there are still challenges to overcome, such as fine-tuning avatar expressions on a per-person basis. He also mentions the upcoming release of Quest 3, a mixed reality product that will enhance the Metaverse experience.
Rollout Timeline: Mark mentions that the rollout of this technology will be gradual, with more people getting access to it over the next few years. The goal is to integrate it into various Meta apps and make it available to a broader audience.
Emotional Impact: Lex expresses his emotional reaction to the Metaverse, describing it as an incredible transition in communication that breaks geographical and physical barriers.
Fidelity and Optimization: They discuss the need to find the optimal level of fidelity for avatars and the challenges related to rendering full-body avatars.
Quest 3 Introduction: Mark introduces Quest 3 as the first mainstream mixed reality device, emphasizing its affordability compared to the previous Quest Pro.
Mixed Reality Technology: They discuss the achievement of mixed reality technology in Quest 3, which blends the physical and digital worlds seamlessly. The technology utilizes cameras on the outside of the headset for pass-through and mapping to the user's perspective.
Resolution and Chipset Improvement: Mark highlights the multiple times improvement in resolution and the use of a powerful chipset in Quest 3 compared to Quest Pro, which enhances graphics and overall performance.
Comfort and Hand Tracking: They mention that Quest 3 is 40% thinner, making it more comfortable to wear. It also includes hand tracking capabilities, eliminating the need for controllers in certain applications.
Use Cases: Lex shares his experience with Quest 3, emphasizing the unique experience of being able to see physical objects and people while interacting with digital content. They discuss the potential applications, such as gaming, virtual learning (like playing a virtual piano), and mixing physical and digital experiences.
Integration of Physical and Digital Realities: Mark predicts a future where physical and digital realities are seamlessly integrated. He envisions a world where digital holograms are as prevalent as physical objects.
Psychological Implications: They explore the psychological implications of photorealistic avatars and discuss how people may prefer different modes of expression in the long term.
Embodiment of AI: Mark anticipates that AI could also be embodied as realistic avatars in mixed reality meetings, offering unique collaborative possibilities.
Acceptance of Mixed Reality: They observe that people seem to easily accept and adapt to mixed reality environments where photorealistic avatars coexist with other forms of avatars or digital objects.
High-Quality Remote Experiences: Lex expresses his preference for high-quality, photorealistic remote experiences, such as sitting on a virtual beach with a friend and enjoying a beer, as it enhances the quality of remote social interactions.
Technology Feat: Both acknowledge the technological achievement of creating mixed reality experiences and recognize the novelty and impressiveness of the technology.
Avatar vs. Reality: Lex expresses how he's starting to forget that Mark's avatar isn't real, prompting a philosophical question about the nature of identity and reality.
Temporal Accuracy of Avatars: Mark discusses the potential for avatars to be realistic in terms of appearance but not necessarily temporally accurate. He mentions subtle questions arising from this, such as whether avatars should reflect the person's current state or a previous one.
Identity and Self-Perception: They delve into the nature of identity and self-perception, considering whether people would present their best versions in their avatars or if the choice would be based on fluctuating physical states.
Psychological Implications: The discussion touches on how people might perceive themselves and others in digital spaces and how this perception could impact psychology and self-image.
Interacting with Deceased Loved Ones: Lex raises the idea of using avatars to interact with deceased loved ones in the metaverse. Mark acknowledges the potential power of such experiences but also highlights the need to establish norms and balance, as it could have both positive and negative effects on grieving individuals.
Handling Death and Digital Content: Mark mentions that Facebook has experience dealing with death and digital content through its social media platforms, including protocols for memorializing accounts while respecting privacy and security concerns. He suggests that some best practices from the current digital world may apply to the metaverse.
Empowering People with AI: Mark emphasizes the goal of empowering individuals with AI tools in the Metaverse to enhance their lives and productivity. The aim is not to create a single AI superintelligence but to distribute AI capabilities widely.
AI Avatars for Creators: Mark discusses the potential for creators, like podcast hosts, to build AI avatars of themselves that can interact with their communities. This could allow creators to engage with their audiences more effectively and efficiently.
Technical Challenges: They acknowledge the technical challenges of creating AI personalities that accurately represent creators while respecting their preferences and boundaries. Training AI models to be predictable and stay within set parameters is important.
Variety of AI Personalities: Mark envisions various AI personalities serving different purposes, including assistant-oriented AIs, business-oriented AIs for customer interactions, and fun personalities for entertainment.
Enriching Social Interaction: They explore how AI personalities can facilitate connections between people, enhance group chats, and even improve gaming experiences by making non-player characters (NPCs) more interesting.
Physical and Digital Integration: Mark and Lex discuss the blend of physical and digital experiences in the Metaverse. They highlight the importance of making digital experiences feel physical and how it can positively impact people's lives.
Ethical and Legal Considerations: The discussion touches on the ethical and legal questions surrounding the Metaverse, such as determining what is allowed in digital environments and how to handle issues like anonymity and toxic behavior. They emphasize the need to strike a balance between freedom and responsible behavior.
Novel Experiences and Opportunities: Mark highlights how the Metaverse can enable novel experiences, such as teleportation-like interactions, and create opportunities for remote work that feels physically present, potentially changing how people work and connect.
Evolution of Technology: They recognize that the full implications of Metaverse technology may not be clear from the outset and that its development will be an ongoing process, shaped by the actions of developers and users.
Open Sourcing LLM-2: Mark discusses the decision to open source LLM-2 (Large Language Model 2) and the positive reception it has received. He highlights the value of sharing foundational AI models with the community.
Llama 3: While Mark doesn't provide specific details about LLM-3 (Large Language Model 3), he mentions that they are working on future foundation models. However, there is no concrete information about its release date or features.
Personalized AI Systems: They talk about the integration of AI models like LLM-2 into personalized AI systems with AI personas. These AI personas can engage in conversations with users and will allow society to learn about the boundaries and applications of such technology.
Learning by Using AI: Mark and Lex emphasize the importance of learning by using AI systems. They expect people to experiment and figure out what works, what's fun, and what's too far in terms of personalization and customization.
Diversity of Use Cases: Mark mentions the diversity of use cases they are exploring with initial AI persona launches, such as text-based adventure games, comedians, and historical figures. They are curious to see how users will interact with these AI personas and generate memes and content.
AI Studio: Mark mentions AI Studio, a platform they are working on that will allow anyone to create AI personas. This platform will enable users to contribute to the development of AI personalities and content.
Community Creativity: Mark looks forward to the full potential of AI development when the entire community can contribute creatively. He believes that the stages of development will lead to a better understanding of how AI can be integrated into various aspects of life.
Unique Approach: Mark notes that the approach they are taking, combining avatars, expressiveness, and interactions across different apps, is distinct from what others are doing in the AI and character-based AI space.
Shifting Notions of Reality: Lex expresses his amazement at how, despite being physically distant, the conversation with Mark Zuckerberg feels like an intimate, in-person interaction. They discuss the idea that the nature of reality is changing, with the emphasis shifting from physical presence to subjective experiences and consciousness.
Metaverse's Role: Mark emphasizes that the metaverse's primary purpose is to enable people to feel present with one another, even when not physically co-located. He highlights the importance of creating a sense of intimacy, trust, and non-verbal communication within the digital environment.
Psychological and Social Benefits: They discuss how the metaverse can offer psychological and social benefits that were previously only possible through physical presence, including enhanced communication, non-verbal cues, and shared experiences.
Technological Challenges: Mark acknowledges that building the metaverse is technologically challenging, involving the development of new computing platforms and software stacks. However, he believes that it's the next frontier for technology and communication.
Evolution of Communication: Mark notes that communication has evolved from text-based social networks to photo-sharing, then video content. He sees photo-realism and immersive experiences as the next step in this evolution, enabling richer and more vivid interactions.
Excitement for the Future: Lex expresses his excitement about the future of the metaverse and thanks Mark for his role in building it. He acknowledges the talented teams at Meta (formerly Facebook) working on this vision.
Humor and Connection: The conversation concludes with a humorous exchange about Lex imagining Mark sitting on a beach without pants, emphasizing the strong connection they've experienced through the metaverse.