Synthesizing weld pool dynamics via VAE-GAN to enhance human control performance

A process is not truly robotized unless the robot can adaptively respond to it—yet such adaptability varies across processes and often demands application-specific engineering. Learning from Human Operator (HO) who views and controls the welding process via a Human-Robot Interface (HRI) provides a more generalizable approach to acquire the needed adaptation. A fundamental challenge is how the HO we wish to learn from can perform beyond his/her capability limitations. To address this challenge, we propose predicting and rendering the future outcomes to the HO, resulting in predictive human control which is expected to perform better than conventional reactive human control. However, the performance enhancement critically depends on the prediction's accuracy and interpretability by the HO. In welding processes, such prediction entails generating complex, dynamic weld pool images from welding parameters. This task requires not only high-fidelity synthesis to capture key process features but also controlled generation to preserve temporal dependencies with parameters. The predictive model must replicate the unknown dynamics mapping welding parameters to the visual evolution of the weld pool. We propose a deep generative framework for Gas Tungsten Arc Welding (GTAW) consisting of three interconnected stages. First, a Beta Total Correlation Variational Autoencoder (β-TC VAE) compresses weld pool images into a low-dimensional latent space capturing dominant features. Next, a Long Short-Term Memory (LSTM) network models the temporal evolution of latent codes based on welding parameter variations. Finally, a PatchGAN refines the VAE reconstructions, restoring fine-grained stochastic details to enhance visual realism. By combining latent representation learning, temporal modeling, and adversarial refinement, our framework enables realistic, controlled synthesis of dynamic weld pools. This approach is extendable to other manufacturing processes with strong parameter dependencies.