What is embodiment in virtual reality?
Technology
The increase in demand for VR products in recent years has not only made people pay more attention to virtual reality but also expanded the corresponding market. Nowadays VR equipment is portable and affordable, also the developed software environment(mainly games) and user-friendly interface make it more common and households.
Users wearing VR headsets can build an association between a virtual avatar and their own bodies. Since lots of consumer VR hand controllers contain tracking units, the placement and position of a real hand can be directly mapped to the virtual hand in headset[15]. With these features, the variation of classic proprioceptive experiments like the rubber hand illusion can be performed in VR world. In this kind of experiment, the researcher will deliver stimuli to a part of the participant’s body and a visible fake counterpart model. The participants will start to consider the fake model as part of their own body. The ’body ownership’ will be transferred[3].
Previous studies and research have shown that the feelings of body ownership will be most strong when the virtual or fake model looks realistic.[7] In VR, when the passive stimuli are replaced by an active interaction within the virtual environment, the ownership illusion will be stronger when the virtual body follows the users’ movement, compared to when the user‘s avatar is more abstractly presented[19].
However, tomato presence is a little bit different from the rubber hand illusion, it builds on the hand ownership presence illusion. When a user uses VR hand controllers to interact with the object in a virtual environment, for example, picking up a tomato, the mapped virtual hand will be invisible and the control target will be switched to the tomato simultaneously. The hand presence can be maintained using a stand-in object in VR and the brain will intuitively accept it[17].
Figure 1 shows the process of hand presence transferred by tomato in VR game.
(a)
(b) (c)
Figure 1: (a) The user tried to pick up tomato .(b) The user‘s hand attached to the tomato. (c) The hand presence was replaced by tomato presence.
Research on this topic can improve the immersion of VR gameplay, by learning how can tomato presence affects subjective embodiment and body presence. We may learn more about how people feel and understand their bodies in relation to their surroundings. By comparing job simulator with several types of research and games, this report analyzes what does tomato presence suggest about embodiment and presence from different angles. Including the relation and the difference between tomato presence and the traditional VR games that display virtual hands when interaction happens; in which situation can tomato presence enhance immersion and the human brain can naturally accept it; how do users feel when they witness tomato presence happened on the other users.
2 Background
2.a Body ownership
The sense of body ownership has been proposed to emerge from a combination of bottom-up and top-down influences. Here the bottom-up information refers to the afferent sensory information that arrives to our brain from our sensory organs(for example, visual, tactile, and proprioceptive input) and top-down information consists of the cognitive processes that may modulate the processing of sensory stimuli(for example, having sufficient human similarity to presume that a virtual body can the actual body)[6].
Sense of ownership is the feeling of identifying sensations (both internal and external) as affecting, establish ing, and belonging to one’s identified-self[11]. In VR, the body ownership can be mapped to the self-avatar in the virtual environment, the amount by which the participants use their virtual body can affect their responses to body presence[15]. In UCL VR Lab, the author experienced the VR game “Half-Life Alyx” by valve using oculus, in the game the gamer has to use a hand controller to grab the items such as guns, bottles, and boxes. Instead of using a laser indicator to point at the item and the item will teleport into the gamer‘s hand, using the controller to touch and grab up the item in-game will create a stronger sense of hand presence and ownership.
The term “presence” is related to wide-field research, in the explication of immersion, presence means the extent to which the senses are engaged by the mediated environment[12]. Previous research found that it is controversial whether presence can contribute to better task performance. However, in VE, it is true that applications and games that have stronger body presence will produce a better VR experience.
2.b rubber hand illusion
The rubber hand illusion is a psychological experiment, originally reported by Botvinick and Cohen in 1998. Subjects who have normal brain function are positioned with their left hand hidden out of sight[3], what they can see is a fake rubber hand. Both hands will be touched and tapped at the same time, the sensation of touch and the witness of fake hand being touch may make the subjects associate the fake hand with their own body.
By measuring skin conductance(galvanic skin response GSR) through a sensor on the palm or fingers, physiological arousal could be measured. The skin conductance will increase when more sweat is produced from hand, which means the hand experienced sufficient stressful stimuli. Measuring heartbeat is also a valid indicator.
In VR, most equipment will use a motion track sensor to keep track of the user‘s hand motion, by using the controller, can elicit a similar illusion in the virtual environment. Previous work[15] has proved that the player will have similar reaction to the threats in the virtual world, which proved that rubber hand illusion can be applied to the VR field.
2.c tomato presence
“tomato presence” was first proposed by Owlchemy Labs in the game Job simulator, in this game, when player interact with the item in the game, in order to not obscure objects that have been picked up from view, and to ensure that the visible hand does not interpenetrate or collide with objects in an unnatural way, the virtual hand will be hidden. The target item will replace the original hand until the player puts the item down, the movement and the rotation of hand controller are synchronized perfectly to the item[17]. According to developers at Owlchemy Labs, players can adapt to this change naturally and will not cause any loss of hand presence within the game[15].
Except job simulator, the VR games that also used tomato presence are: Rick And Morty VR, the Under Presents, Eleven Table Tennis, Accounting+(partially used).
3 Discussion
3.a Motivation
Most of the VR game will not hide user‘s hand intentionally when user tries to interact with the objects in the environment. Both Sony[14] and valve[18] have developed their controllers which support full-fingers tracking, these controllers can keep track of the movement of each finger and palm in real-time, this allows the player to perform actions that are much more complex and subtle than before. Previous work has proved that realistic virtual hand will create a stronger sense of body ownership[7], smooth and accurate hand-object interacting animation will not only enhance the hand presence and immersion of player but also shows the technical storage of the game company. Figure 2 shows that the process of picking up bottle in game is identical to the real world.
(a) (b)
Figure 2: The player picked up the beer bottle and used the flashlight to check it[9]
Nonetheless, in order to fit the game play, sometimes the game company will try to find another way to perform this kind of action. In-game job simulator, the players are asked to perform various manipulations to the items in a fairly small region, such as grilling sausages on a sausage machine, filling up a bottle using a soda machine, and then scanning the product to sell it to the customer. Various kinds of actions meaning the game company will have to create corresponding hand animations for every unique action.
Regardless of the effort that the programmer will make and the time that the animator is going to spend on the motion capturing since in-game each item has its own size and volume, a universal grab algorithm will cause the virtual hand penetrates through the item or collides with it. For example, the bottle has a narrow shape at the top and wide at the bottom, if the accuracy of the mesh collider is not high enough, grabbing the middle part of the bottle will inevitably overlap the finger or the part of the palm with the bottle. Otherwise, the hand will stick to the item in a certain way, there might be a small distance between hand and item, which reduces the immersion.
Tomato presence more or less solves this problem, it reduces the workload of programmers and the problems like mesh colliding and model penetration will not happen. It can also improve the game experience. In job simulator, when player pulls down the handle of the soda machine, because the hand controller and game do not have a precise touch sense and force simulation, the player might not be able to estimate the amount of drink by sensing how much resistance the handle applies to hand. However, since the hand is replaced by the handle and player‘s view is no longer blocked by the virtual hand, he/she can estimate the amount of soda easier, thus reducing errors. Linear motor and haptic modules may also help with this problem, but this is outside the scope of the paper.
3.b Embodiment and presence
The visual embodiment requires the virtual and real bodies are perceptually spatially coincident. In this case if the Virtual world is the only visual information they can achieve, it will appear to the user as if the virtual hand/body had substituted their real one[2]. This suggests that, usually the users will feel embodiment in virtual avatars with realistic hands. Moreover, when the virtual hand is replaced by the other items such as a 3D arrow(the arrow is positioned centrally at the current controller position and the tip of the arrow is used to select objects)[19], the participants responsiveness to danger is lesser than before, this suggests that they feel less embodiment.
Figure 3 shows the participant feels less embodiment when their virtual hand is replaced by arrow. According to the table, the average degree of arm rises when participant has a virtual hand is approximately 0.25m higher than that when he/she has a arrow “hand”.
Figure 3: (1)The right hand is replaced by arrow (2)the hand is threatened by the falling lamp (3)The result of the experiment, less mean(SEM) means the participant rises up their hand less when the threat happens[19]
3.b.1 Visual perspectives
The difference between the tomato presence and the experiment above is, the virtual hand will not turn to arrow, it becomes invisible once the user attaches the item and the target item substitutes hand temporarily. This can elicit a question: Will seamlessly switching the hand ownership to tomato ownership affect embodiment and presence?
In 2014, A Maselli, M Slater did a research on the mutual relation between body ownership and the experience of the body occupying a given portion of space in the environment(self-location) using Crossmodal Congruency Test (CCT) and The “Tube Task” (TT)[8]. They found that when there is no spatial overlap of the virtual and real bodies, the out-of-body experience will be very obvious and the embodiment will be less strong.
However, surprisingly, when only a partial spatial overlap of the virtual and real bodies is preserved, the participant can still experience a strong full-body ownership illusion. In tomato presence, the position and the rotation of the tomato will not change when the attached hand disappears. In other words, if the user throws a tomato into the air and catches it, there is a short moment that the tomato will look like it immediately stopped moving and then floating in the air, but in fact, at this moment the tomato has already become the virtual hand. There will become a small but noticeable degree of misalignment between the position of the virtual hand. And the position of the tomato, since a considerable overlap between two items become preserved, the sense of embodiment and presence can still be strong.
In another research in 2010, M Slater, B Spanlang and MV Sanchez-Vives used a life-sized virtual human female body to substitute the male participants‘ own bodies to generate body transfer illusion[13]. They changed the perspective position of VR view point and found that bottom-up perceptual mechanisms can temporarily override top-down knowledge. For most of the time, the participants looked up towards the virtual woman(third person perspective), and would see her arm move up and down synchronously with feeling of the strokes, or see the same in the mirror reflection[13].
This can be a revelation, because the rotation angle and position of the tomato will become perfectly synchronized with the actual hand. There is a change that the bottom-up knowledge ( the tomato becomes my virtual hand temporarily) will replace the previous knowledge(I have a virtual hand and it is synchronized with my actual hand) and override the top-down knowledge. Considering that the experiments above became conducted in a third person perspective. In job simulator. Witnessing this bottom-up knowledge change in a first-person perspective could have stronger influence to hand embodiment. The brain may accept this change rapidly and more smoothly than we expected. However, this hypothesis may not hold under several conditions.
3.b.2 Subjective feeling and game play
Most VR games we can find on the app market are first-person. As mentioned in section 3.a, the realistic virtual hand can produce a strong hand presence in half-life alyx. Figure 4 shows that in game the player can control their virtual hand to move, grab and shoot just like in the real world.
What will happen if the tomato presence becomes used in half life? Imaging as soon as the player picks up the pistol, the hand disappears and becomes the “pistol” presence. The player can no longer affirm that he is actually holding the pistol. The lack of the realism in holding will reduce the immersion. Compare to job simulator, one of the main differences of half life is that its graphics is more realistic. Being able to see one’s hands and perform the full operation is essential when VR operations involve such complex motions as shooting a gun.
(a) (b)
Figure 4: The player can see their hands while shooting and moving the board[5]
Another difference is that, in half life, players can see their body avatars, but the body avatar is invisible in job simulator. The core game play of the job simulator is to interact with surrounding objects with hands. And the virtual hand is the only body presence that the player can see in game. Besides, the virtual hand becomes cartoon styled. This game play design will make the players pay more attention to their hands and don‘t need to think about the other movements like moving, dodging, and solving puzzles, which are the main game play elements in half life.
Tomato presence seems to be a suitable system for game like job simulator. In another video about tomato presence(Figure 5), the owlchemy lab shows us the example of different objects becoming the tomato presence. The interesting part is sub figure a) b) and c): in these three images, the process of opening gift knot is very smooth and satisfying. From personal perspective, seeing the hand disappeared in this video does not make me feel unnatural. One of the most important reasons could be the game makes the gift knot disappeared as soon as the player opens it.
This makes the gift knot stop representing the virtual hand automatically so that the player does not need extra steps like throwing the knot away to disable the tomato presence. Since the physical properties of soft objects like gift knots are different from other large solids(for example, the top cover of the gift box in figures d and e ), presenting the process of opening gifts as real as the actual world does not apply to tomato presence. Simplifying the operation steps can reduce the unnatural feeling and make the brain accept it easier, this requires programmers to be very familiar with the interaction of items in the game and optimize them accordingly.
(e) (f) (g)
The research that D Roth, JL Lugrin did in 2016 could also give us some inspiration, in their experiment they let two participants cooperate on a small football field, and perform several basic movements like passing the virtual ball under each others‘ feet. The virtual avatar they chose to use is a body-size wooden manikin(Figure 6). The conclusion is that compared to playing football in the real world, the physical interaction showed significant differences in performance. They think the reason could be due to the disturbed visuomotor synchronization caused by latency, as well as the approximation of physicality in the VR condition[10].
(a)
Figure 6: In the physical task, the number of errors participants made was significantly lower (real world M = 0.91, SD = 1.25 and VR M = 3.281, SD = 1.99) and they need longer time to pass the ball in VR(M = 85.88s) than in real world(M = 24.63s)[10]
Considering that the physic simulation in job simulator is much more realistic than this experiment and they used feet instead of hands. We can not put their conclusions about latency and lack of attention into tomato presence directly. But think reversely, if we try to implement the tomato presence into a foot ball game, the foot will disappear and reappear when the user‘s attaches and detaches the ball while moving it, this vision loss will greatly stagnate the operations. The tomato presence may not be suitable when the game involves full body coordination. It may be helpful when the novice tries to do football juggling or balancing the ball.
Because this kind of movement that requires consistent practice is more mechanical and repetitive, tomato presence can help them to be more focused on the ball itself and get through the novice stage smoothly. This assumption coincides with the explanation of the game play of the job simulator in the last paragraph(“This game play design will make the players pay more attention to their hands”), however, we do not have the experiments and data from previous research to support this directly.
3.b.3 tomato presence in other people‘s view
Previous sections only discussed the subjective experience of tomato presence for participants and players. If a VR application supports multiplayer mode and tomato presence at the same time, what does it look like and how do users feel when they see each other interact with the items in the scene? Should tomato presence only exist as a personal experience that aims to convenient item interaction in VR?
Figure 7 shows a game called Eleven table tennis, in game the player can only see a ping pong paddle and his/her opponent‘s virtual Oculus headset, which helps the player to perceive the direction and angle that the ball is moving and the sight of the opponent.
(a)
Figure 7: Table tennis, from you and your opponent‘s view[4]
In order to have a better understanding of it, I downloaded the game. And tried it on the oculus quest headset, which becomes borrowed from the UCL VR Lab. The physic effect in this game is realistic and the latency is low. Since the player can perceive most of the information needed to play ping pong from the opponent‘s ball paddle and oculus headset orientation, tomato presence does not affect the game experience.
My brain quickly accepted the game setting. Unless the game can perfectly synchronize the movements of people playing table tennis in the actual world(which requires an accurate real-time full body scan). And use high quality player graph texture, the game experience will not become better than the scenario when the tomato presence becomes used.
This personal experience may suggest that the users may not feel uncomfortable when they witness the tomato presence happening to others, as long as it fits the game play. The previous researches discuss the possibility of perceiving rubber hand-like illusion in third person perspective[13], but the author did not find relevant research about the acceptance of tomato presence in the third person to validate the viewpoint directly.
Nonetheless, whether the tomato presence affects the experience largely depends on what we want to achieve. In the professional sports field, VR aims to help sportsmen to gain strategies, physical and mental preparation, and furthermore higher likelihood of getting accomplishment[1]. The professional player requires a much more realistic VR experience, they perceive the information from every part of their opponent‘s body, it is not acceptable no matter how little does tomato presence conceals the hand.
4 Conclusion
This report introduces tomato presence and describes what does tomato presence suggest about embodiment and presence from different angles. Tomato presence could reduce the workload of game programmers. It can become widely used in the scenario when the core mechanics of the game are only related to the hands and the user has limited moving area. By optimizing the animation of tomato presence for different objects, users can accept tomato presence effortlessly. Partial hand and object misalignment after tomato presence happened may not significantly affect the game experience.
When the game becomes not limited to simple item interactions. However, aims to generate a more realistic, immersive world, tomato presence may harm the embodiment. Moreover, when the game involves whole body coordination like playing football and the user can see their virtual body avatar in VR, the existence of tomato presence may reduce the embodiment, which is similar to replacing the virtual hand with the arrow.
The author did not find enough research samples that discuss the different peoples‘ perspectives on tomato presence in the same scene. However, the possible situations that can happen become discussed according to personal experience and theoretically.
In conclusion, the future work will be more in this direction, especially the users‘ feelings when the tomato presence only happened from their subjective view but the other players are still holding items normally in multiplayer games. More experiments should become done to evaluate the impact of game play on tomato presence‘s influence to embodiment and presence.
References
[1] Kunjal Ahir et al. “Application on virtual reality for enhanced education learning, military training and sports”. In: Augmented Human Research 5.1 (2020), pp. 1–9.
[2] D Borland B Spanlang JM Normand. “How to build an embodiment lab: achieving body representation illusions in virtual reality”. In: (2014). url: https://doi.org/10.3389/frobt.2014.00009.
[3] Body transfer illusion. Feb. 2022. url: https://en.wikipedia.org/wiki/Body_transfer_illusion. [4] Eleven Table Tennis Oculus Quest LAN Play. url: https://www.youtube.com/watch?v=ycFp_87dCZw. [5] Half-Life: Alyx Gameplay Video 1. url: https://www.youtube.com/watch?v=LTLotwKpLgk.
[6] Konstantina Kilteni, Raphaela Groten, and Mel Slater. “The sense of embodiment in virtual reality”. In: Presence: Teleoperators and Virtual Environments 21.4 (2012), pp. 373–387.
[7] Lorraine Lin and Sophie Jörg. “Need a Hand? How Appearance Affects the Virtual Hand Illusion”. In: Proceedings of the ACM Symposium on Applied Perception. SAP ’16. Anaheim, California: Association for Computing Machinery, 2016, pp. 69–76. isbn: 9781450343831. doi: 10.1145/2931002.2931006. url: https://doi.org/10.1145/2931002.2931006.
[8] Antonella Maselli and Mel Slater. “Sliding perspectives: dissociating ownership from self-location during full body illusions in virtual reality”. In: Frontiers in human neuroscience 8 (2014), p. 693.
[9] New Bottle Liquid Physics – Half-Life: Alyx. url: https://www.youtube.com/watch?v=mVQNMmnIqMo.
[10] Daniel Roth et al. “Avatar realism and social interaction quality in virtual reality”. In: 2016 IEEE Virtual Reality (VR). IEEE. 2016, pp. 277–278.
[11] Sense of ownership. Feb. 2022. url: https://en.wikipedia.org/wiki/Sense_of_ownership.
[12] Mel Slater. “Immersion and the illusion of presence in virtual reality”. In: British Journal of Psychology 109.3 (2018), pp. 431–433.
[13] Mel Slater et al. “First person experience of body transfer in virtual reality”. In: PloS one 5.5 (2010), e10564.
[14] Sony Prototype VR Controller with Full Hand Tracking. url: https://www.youtube.com/watch ?v= ZWMWzmBbhjw&t=22s.
[15] Anthony Steed. “Effects of Hand Appearance and Behaviour on the Rubber Hand Illusion in Virtual Reality”. In: 2017.
[16] Tomato presence is a gift. url: https://twitter.com/owlchemylabs/status/1169685633696174082. [17] Tomato presence! url: https://owlchemylabs.com/tomatopresence.
[18] Valve Index controllers handling slow finger movements. url: https://www.youtube .com/watch ?v= cjXSXmHZP3Q. [19] Ye Yuan and Anthony Steed. “Is the rubber hand illusion induced by immersive virtual reality?” In: 2010 IEEE Virtual Reality Conference (VR) (2010), pp. 95–102.
Written by Yicheng Zhan
Technology
What is embodiment in virtual reality?
