An Innovative Real-Time Mobile Augmented Reality Application in Arts

Due to the popularity of music, motion tracking and augmented reality in recent years, the research topic in these fields is extremely popular. In contrast to every previous work, in this paper, we present an innovative real-time mobile application in arts for helping musicians by integrating motion tracking into augmented reality technology. A kinematic filtering algorithm is utilized for calculating the parameters. According to the computed parameters, each hand of musicians is then tracked by using the Microsoft Kinect. After that, an augmented reality application with an integrated multimedia feature is built based on the PixLive Maker synchronized with music being played. This hybrid application allows musicians to interact with the virtual piano in a new way that is similar to the way in which they are playing a real piano. By pressing any selected piano key on the air, the sound of each note is generated continuously into a song and incorporated with an interface in the smartphone. The new application achieves a suitable computed rate for real-time use. Representative experimental results have shown that the application is beneficial for piano players in arts by allowing them to practice and touch the virtual piano with lower cost and an interactive experience.