Connectivity Between Leap Motion and Android Smartphone for Augmented Reality (AR)-Based Gamelan

. The smartphone development today makes the gadget not only used as a communication tool but also as an entertainment tool such as to play games and play music. The development of the smartphone also supports many technologies that can be run on the smartphone itself, such as Augmented Reality (AR), for example. There are some studies evaluated the AR application combined with Leap Motion, but those studies were using the Software Development Kit (SDK) alpha of the Leap Motion Corp. that is now no longer accessible for the developers to use. This research will provide an option of technique to connect the augmented reality technology to leap motion device as the controller without using the official SDK from Leap Motion Corp. The method proposed in this study is a technique to connect the Leap Motion with Android for Augmented Reality application. This paper also evaluates the technique used to connect the AR technology to Leap Motion so it can be a visual instrument simulation, which applied to the Gamelan traditional music instrument. The experiments resulted in the accuracy rate of the application of 96.43% for right-hand movement and 97.86% for the left-hand motion. These results are the average of the experimental data obtained for each hand. The high accuracy result obtained in the research can be a promising result for the future research.


Introduction
Smartphones that are generally used for communication can also be used as entertainment devices, such as for playing music and games.The availability of music on some services such as Google Play Music or others sometimes confuse the users to choose what kind of music they want to play.In order to provide the music recommendations or streaming services for users, there is an individual Personal Music for Android [1].
Global Positioning System (GPS) and other sensors such as accelerometer and gyroscope that have been embedded in the smartphone so it can present spatially relevant information that may increase the user knowledge about their surroundings so that the smartphone can support better technology for Augmented Reality [2].For some Frihandhika Permana et al. , The Connectivity between Leap Motion.... 147 p-ISSN: 2540-9433; e-ISSN: 2540-9824 companies, AR technology has been applied in their business areas such as for warehouse management, health, retail, insurance, or travel business [3].
Gamelan is the traditional musical instrument from Indonesia which consists of several instruments such as Bonang, Kendang, and other instruments.Along with the development of technology, musical instruments are also developed, and Gamelan was pretty much abandoned.Not many people understand how to play the Gamelan instrument, and it is now even quite difficult to find [4], [5].
Several studies focus on replicating Gamelan into digital music instruments, such as a digital Gamelan [6] and also pocket Gamelan [7].In 2013, a study related to the AR implementation in the Gamelan application was conducted, the application was played by making virtual buttons as markers.By covering specific markers, the system will give the instructions to the application to play the correspond Gamelan notes.Unfortunately, the technique used was not the way to play the actual gamelan [8].
A study in 2016 focused on applying Leap Motion as a control to play a traditional musical instrument.However, the application can only run on the desktop platform so it is not flexible to play and the user must see the monitor screen all the time when the user wishes to coordinate his hand with a musical instrument he wants to play [9].
The studies evaluated the AR applications with Leap Motion are also conducted by [10] and [11].However, both studies used the alpha SDK from Leap Motion Corp. which is no longer accessible for developers.
This paper proposes a technique on how to connect Leap Motion with a smartphone as an Augmented Reality (AR)-based simulation tool.This technique is applied to gamelan musical instruments to prove that the proposed technique accuracy can be measured and tested.The research expects that the methods used in AR instruments with Leap Motion control can be the basis for the applications that require connectivity between AR and Leap Motion.

Related Work
The method used by [8] was a prototyping model with two-stages of problem-solving; the first is the elicitation requirement.The prototype of applications was developed by making input and output processes based on the extracted requirements.Users were able to choose one of the five Gamelan instruments provided in the application.The smartphone then activated the camera to capture the marker, and the covered marker meant that the related Gamelan note would be played.
There was also a study that evaluated the Gamelan as a game applied on multiplatforms (Android and iOS), that use the method of Analyze, Design, Development, Implementation & Evaluation (ADDIE) [12].
Research that conducted by [13] was focusing on the AR technology to support the museum collection recognition.The usability testing resulted in 86% of museum staff satisfaction toward application.The particular value concluded that the application could help the introduction of museum objects to museum visitors.Compatibility testing indicates that these features work validity can be executed very well.
The research conducted by [9] was focusing on the Leap Motion used as a sensor to detect the movements (hand movements) and send the data to the computer as the sign to play the chosen musical instrument.

Methodology
The methodology used in this study consists of six stages, which are: Literature Study, Requirement Analysis, Implementation, Testing, Result Analysis, and Conclusion.It is  The first stage is a literature study which includes observing previous research to enrich the knowledge of the related study.Then, it is followed by the requirement analysis stage that consists of some actions (data retrieval, actor defining, functional requirement, and storyboard).The third stage is implementation which carries out the processes of device specification, implementation boundary, and application results.The testing stage is done in the fourth sequence.This stage includes two kinds of testings (right-hand testing and left-hand testing).The testing result analysis then is done afterward, which belong to the fifth stage (result analysis).The last stage is taking the conclusion based on the analysis result.

System Architecture
Fig 2 describes that in the system Leap Motion sends the hand gesture data to the server, then the server sends the data to the Head Mounted Display (HMD) that has been connected to the smartphone through the WiFi using the User Datagram Protocol (UDP).
User Datagram Protocol (UDP) is a TCP/IP protocol that supports unreliable and connectionless communication.Unreliable means that UDP messages which are sent as datagrams do not have any sequential number.Connectionless means that the UDP messages sent do not require negotiation between two hosts.UDP sets two services not allowed by the IP layer.It allocates port numbers to help recognize distinctive user request and optionally a checksum capability to verify that the information arrived flawless [14]- [17].

2 Leap Motion
Leap Motion Controller is a USB-based device with a small size that allows the computer user to control or play the computer by using the movements [18].Leap Motion Controller capture the hand motions, the finger movements, and also objects such as pen or pencil independently.Leap Motion is 200 times more sensitive compared to the current "free touch" technology.Leap Motion Controller detects and tracks the direction of the hands, fingers, and other objects similar to fingers that report the discrete values of the position and gestures.
Leap Motion produces information of the pose or hand shape by using the frame detection method.Leap Motion generates that information from the palm and fingers consists of hands, fingers, and orientations.The palm center position can be extracted in 3D space from the hand, and the fingertip and hinged bones positions can be derived from the fingers, relatively from the palm center position, and the orientation information that shows which direction in roll, pitch and yaw format [19].There are two cameras in the Leap Motions which can capture the photos from various directions to obtain the hand motion information in 3D space.Therefore Leap Motion can sense the hand and fingers orientations and also the curvature of the palm.The Leap Motion detection range is between 25 mm to 60 mm on the sensor [20].The Leap Motion coordinate system can be seen in Fig 3.
The interaction area reached by the Leap Motion is 2 feet above the controller, with the width of 2 feet on each side (angle 150°), with the depth of 2 feet on each side (angle 120°) [21].The Leap Motion field of view is illustrated in Fig 4.

3 Server
In this study, the server is used to receive data taken by Leap Motion and then process it.The server will resend the data to the smartphone through the same WiFi network.The process data in the server is illustrated in the system diagram in Fig 5.
The function of the server in the system is to equalize the network used by both devices, which are the computer running the server application and the smartphone running the Gamelan simulation application.Table 1 is

3 Head Mounted Display (HMD)
Internal Head Mounted Display (HMD) is a type of computer display device or monitor used on the head of the user that looks like a helmet or glasses.Most HMD is used to support virtual reality technology, but sometimes they are also used for Augmented Reality (AR) technology.Several studies evaluated the HMD implementation in specific areas, such as HMD that is used on Android Application Controller using Gyroscope sensor [22].
There are various types of HMD available on the market developed by multiple companies according to their respective brands.Such as the Oculus Rift, Playstation with the Playstation VR, HTC Vive, Samsung with Gear VR, and Google with the Google Cardboard.
Unlike the Oculus Rift, Playstation VR and HTV Vive, the Gear VR from Samsung and Google Cardboard from Google are commonly referred as the dummy HMD [23], because both of them still require a smartphone as the primary computing device and also as the primary display used.Fig 6 is the image of a Google Cardboard.

Implementation
The system implementation uses the Android Operating System platform and Unity as the developing tool.Table 2 interprets the smartphone minimum requirement that can be used to run the AR-based Gamelan simulation with the Leap Motion control.
Table 3 illustrates the developing device specification used in the system implementation.

Display of the Application
After the Leap Motion is connected to the computer as the server, and the Smartphone is also connected to the same network, we need to open the server application on the computer.Fig 7 shows the appearance of the desktop server application for receiving the data from the Leap Motion.The desktop server interface shows the simulation of hand motion data that sent by the Leap Motion.Several buttons on the server interface are used to adjust the position of the Gamelan hammer in the Augmented Reality.The position arrangement is for X, Y and Z direction and applied to the left and right hand.The server will send the hand recognition data to the Smartphone via the same WiFi network, and it will add the user's hand 3D object as the Gamelan hammer for the simulation as shown in Fig 8.
When the application detects the marker, it will display the object of 3D Gamelan as shown in Fig 9 .When the Gamelan Hammer reaches the object of 3D Gamelan, the system will instruct the application to play the sound of the note according to the related note touched by the user.This process is described in Fig 10.

Functional Requirement
This section describes the functional requirement for the development of the AR-based Gamelan simulation with Leap Motion Controller, as described in Table 4.

Testing
The testing was carried out to test the accuracy of the system.In this experiment, the user is given the completion task to hit the Note 1 to Note 7 in the Augmented Reality gamelan.Each testing involves the movement of the right hand and the left hand to be scanned as the Gamelan hammer.There are seven notes in Gamelan, and each hand will do the task 20 times for each note.
The formulas that are used to derive the accuracy rate and accuracy average are described in the following formula 1 and formula 2.
= ∑   7 (2)    The results from the experiment obtained in table 5 show that for Note 4-7, the accuracy is 100%, while the results for Note 1-3 are below 100%.This is because, on the right-hand experiment, the user had to hit the Note 1 to Note 3 which are located on the left side of the Gamelan.For Note 1 and Note 2, the user only managed to sound 18 out of 20 attempts.Moreover, for Note 3, user managed to sound 19 out of 20 attempts.This is because the range of the of the user's right hand is difficult to reach Note 1 to Note 3 which are located on the leftmost side of the Gamelan.
On the contrary, in table 6 which is the result of a left-hand experiment, Note 1-5 obtained the accuracy of 100%.Whereas for Note 6, the user managed to sound 19 out of 20 attempts.Moreover, for Note 7, the user managed to sound 18 out of 20 attempts.This is because the range of the of the user's left hand is difficult to reach Note 6 and Note 7 which are located on the far right side of the Gamelan.
From these two experiments, the average accuracy of the right-hand testing is 96.43%.The average accuracy of the left-hand testing is 97.86%.The average accuracy for the left hand is slightly better than the right-hand average accuracy because the position of the 3D Gamelan was placed on the left of the user's perspective.

Conclusion
This study has evaluated a methodology to connect the Leap Motion to Android by using UDP server for AR-based Gamelan, so this paper gives the alternative solution about the Leap Motion SDK that has already been blocked.The process is started by recognizing hand gesture by the Leap Motion; then the data is sent to a server that uses the same Wi-Fi network as the Smartphone.The information is processed by the server and is sent to the Smartphone that has installed on the HMD.In user perspective, the 3D object of gamelan is generated as well as the 3D object of the Gamelan mallet.When the mallet is reached to a particular 3D object of gamelan note, then the application will play the specific tone.

Fig 1 .
Fig 1. Hands Model and Leap Motion Coordinate System; (a) Left Hand; (b) Right Hand; c) Leap Motion Source: LeapMotion.com a part of the description of the source code for the local network IP setting used.The most important code part in line 2 and line 3. Line 2 settles the IP address used in the system; line 3 assigns the port used in the system.Frihandhika Permana et al. , The Connectivity between Leap Motion.... 151 p-ISSN: 2540-9433; e-ISSN: 2540-9824 Fig 4. Google Cardboard Source: google.com The user's position playing the AR-based Gamelan simulation with Leap Motion Controller is illustrated in Fig 11.

Fig 5 .
Fig 5.The Display of the Server

Fig 9 .
Fig 9.The User Playing the Gamelan Simulation

Fig 10 .
Fig 10.The Diagram of the Right Hand and Left-Hand Testing

Table 2 .
The Smartphone Minimum Specification

Table 5 .
The Result of the Right-Hand Testing

Table 5 ,
Table 6 and Fig 12 define the result of the accuracy testing for right hand & left hand.On the right-hand testing, the accuracy of 100% is obtained in the measurement of Note 4 to Note 7. The accuracy of 90% is achieved on the measurement of Note 1 and Note 2, and the accuracy of 95% is available on the measurement of Note 3.

Table 6 .
The Result of the Left-Hand Testing