SHB:Type II (INT): Synthesizing Self-Model and Mirror Feedback Imageries with Applications to Behavior Modeling for Children with Autism

This proposal presents an interdisciplinary, integrated research and education program with three goals. The first goal of the research program is to develop new technologies to (a) facilitate the creation of synthetic visual contents for self-modeling, and (b) combine the synthetic contents with 3D sensing to render a mirror-like feedback on novel display devices called “virtual mirror”. If successful, our proposed software and hardware systems can provide unprecedented flexibility in creating novel behaviors of self in traditional visual medium and novel immersive devices. Beyond the obvious applications in fashion and cosmetics, we hypothesize that our proposed systems would be particularly beneficial to children with autism spectrum disorder (ASD). The second goal of the research program is to better understand the divide between physical and psychological self among children with ASD with the explicit objective of bridging them through systematic manipulation of self-images in the virtual mirror and related technologies. In contrast to their typical lack of interest in social interactions, many children with ASD appear to be highly interested in their own image in mirrors and to others imitating their actions. The virtual mirror can take advantage of both of these factors to enhance the understanding of self/other in children with ASD, resulting in enhanced social functioning. The third goal of the research program is to investigate the therapeutic values of the virtual mirror, as compared with more traditional approaches such as video modeling (VM) or video self-modeling (VSM), in modifying certain behaviors among autistic children. VSM is an evidence-based approach in using oneself in video as a model for behavioral intervention. It has an inherent difficulty when applied in autism due to its lack of interactivity and the infrequent demonstration of the skill of interest. The virtual mirror and other technologies developed under the first goal can provide more effective approach in preparing VSM content and improving interactivity. The Intellectual Merit of the proposed research program is as follows: (1) A novel virtual-mirror device is proposed that combines a network of calibrated depth and visual sensors to render a viewpoint-dependent dynamic view of an arbitrary-shaped virtual mirror on a room-size see-through display; (2) Through multimodal and spatially-diverse sensors, the proposed system can provide high-fidelity, non-intrusive capturing of eye gaze, facial expression, body pose, body movement, and other human behavioral patterns; (3) Novel multimedia processing algorithms will be developed for transferring speech, 2D and 3D physical appearances, as well as behaviors from a source individual and a target individual with limited target training data; (4) Improved understanding of behavioral modeling and the self/other system in autism through systematic manipulation of virtual mirror images will be obtained. The Broader Impact of the proposed program includes the followings: (1) Highly effective behavioral modeling in terms of generalization and duration for children with ASD is expected with the virtual mirror system, which provides a novel combination of instant visual feedback and real-time rendering of new behaviors; (2) Greater flexibility to therapists, teachers, and caretakers in creating material for video-modeling and video self-modeling by using the proposed system in transferring audiovisual behaviors from one video to another. The deployment of such software over the web lowers the technology barrier and promotes consistent practice of behavioral modeling in home and other places beyond clinics; (3) The proposed research program is a collaborative effort of PIs from electrical engineering, psychology, medicine, education, and health sciences. Through a series of specially designed activities, the educational objective is to develop a research program in which problem identification, problem sharing, and problem solving occur collaboratively from the very beginning; (4) Outreach programs including device demonstrations at pediatric clinics, television documentaries of research results, and visits to high schools in rural districts are planned to promote the use of technologies in solving important health and societal problems and to broaden the participation of under-represented groups in STEM activities. Keywords:Virtual Mirror, Visualization, Autism Spectrum Disorders, Self-Recognition, Video Self Modeling