UE5 as the platform for Real-World Robotic Sound Design and Interaction
The field of robot audio lacks standardization and a unified approach for integrating sound design and speech into physical robots. Roboticists often resort to default methods, relying on built-in sounds or common text-to-speech APIs without exploring alternative options. When composers or sound designers are involved, they typically create audio in a workstation without the ability to rapidly test it on a robot or even in a robot simulation. This lack of a common language for audio implementations hampers iterative and collaborative work, hindering the ability to build upon past innovations. Consequently, the audio aspect of robotics has remained stagnant for the past 50 years, with most approaches being derived from film robots like R2-D2.
While the absence of standardization poses challenges for creators, it is important to note that no easy pipeline currently exists for any creator working on robot audio. The process of developing sounds for a robot typically involves using a separate platform for sound creation and then requiring additional skills to synchronize the audio with the robot. This dependence on multiple individuals during sound development inhibits iterative feedback. User testing often relies on pre-recorded videos or limited groups of users, with minimal variation in the tested sounds.
Recent studies have demonstrated that sound plays a crucial role in how humans perceive robots and can significantly impact the future of human-robot interactions. Manipulating sound can enhance key attributes such as likeability, perceived intelligence, and trust]. In fact, sound can even alter the perception of a robot’s core performance, as evidenced by a study in which participants believed a robot had improved object-picking capabilities due to sound cues. The lack of a comprehensive approach to robot audio represents a significant missing ingredient in shaping the way robots will interact with humans in the future, affecting aspects such as safety, performance, and integration.
To address these challenges and pave the way for the future of sound and robotics, our project aims to develop an open-source plugin within UE5. This plugin will revolutionize the creation and integration of sound in robotics. UE5 offers a wealth of sound possibilities, and our plugin will leverage these capabilities to establish a standardized and versatile platform for robot audio development. By seamlessly integrating Metasounds, our plugin will provide creators with a unified framework for designing, testing, and implementing audio on robots. The open-source nature of the plugin ensures that it will be accessible to a wide community of developers, fostering collaboration and iterative improvements. We envision a future where sound in robotics becomes an integral and evolving component, enabling safer and more engaging human-robot interactions while empowering creators to explore innovative audio experiences.
Unreal Engine 5 (UE5) is the ideal platform for our project due to its robust and feature-rich environment. By developing our plugin within UE5, we can seamlessly tap into the existing capabilities of Metasounds and Text-to-Speech (TTS) for sound generation. These built-in tools provide powerful audio creation and manipulation features, allowing for dynamic and interactive sound design. Additionally, UE5’s 3D capabilities enable us to spatially position and render audio, creating immersive and realistic auditory experiences. Furthermore, UE5’s ability to dynamically change audio based on various variables and mappings opens up a whole new realm of possibilities. Combining these capabilities within our plugin will result in a system that surpasses current approaches to robot audio.
Our plugin will seamlessly integrate with both physical robot systems and simulations. This ensures that creators can develop and test audio designs in realistic environments without the need for constant access to physical robots. By bridging the gap between simulations and real-world implementations, we can accelerate the iterative feedback process and facilitate comprehensive sound development for a wide range of robots.
Our plugin will have the ability to integrate with an extensive array of robots. Through collaborations with Macquarie University, Monash University, and Georgia Tech, we will have access to over 30 robot platforms. This extensive robot collection allows us to test and validate our platform across a diverse range of robotic systems, ensuring compatibility and broad applicability.
Our project aims to achieve the following goals:
Our project will involve four key developers:
We envision our project extending beyond the initial three-year funding period, with the goal of establishing a thriving and sustainable community. To achieve this, we plan to organize workshops and create online resources that foster collaboration, knowledge-sharing, and continuous development of the platform. Macquarie University has expressed its commitment to supporting this initiative, ensuring ongoing institutional support.
Keirzo, a rapping, drumming robot
Emotional Musical Prosody an approach to non-linguistic sound for robots
Forest Sound for robotic arms