In this interview, Diego Prado, a doctoral candidate at the Chair of Media Technology and researcher at the KI.FABRIK project, discusses their paper and work which was nominated for the IEEE Best Paper Award, looking at how integrating reinforcement learning with human demonstrations accelerates robot learning and bridges the simulation-to-real-world gap, even with imperfect human demonstrations.
1. What did you find out?
We found out that human demonstrations can greatly accelerate the automatic estimation of Virtual Fixtures (VFs) for robot teleoperation using Reinforcement Learning.
Virtual Fixtures are usually defined as an augmented sensory information, displayed on a real environment with the goal of improving the user performance on a task. In robotic manipulation, they are usually constraints in space that coerce the user to move in the right direction.
In our studies, we showed that even when using “bad” demonstrations (when the demonstration did not manage to successfully complete the task that we wanted to solve using teleoperation) the Reinforcement Learning algorithm managed to learn the virtual fixtures considerably faster than when no demonstrations were used.
These demonstrations can not only speed up the learning process, but they also allow the design of VFs for more complex tasks than simple Reinforcement Learning would.
2. What challenges did you face during your research?
Even the best physics engines struggle to simulate accurate forces, friction and contact between objects. This is why we faced several challenges when transferring the simulated results to the real robots (Sim2Real gap). For example, while we were validating the learned skills of a peg-in-hole insertion task, we broke two 3D printed pegs despite having a perfect result in the simulation.
This is especially true for the so called “contact-rich” tasks, where there is a close interaction between the robot and its environment. This way the dynamics of the contact become very complex and the simulation tends to be inaccurate. This means that the skills learned in a physics simulation will not be the same as the skills needed to solve the same task in the real world.
3. Where do you see practical applications?
This method could be applied in industrial scenarios, especially when there is already a Digital Twin or a simulation of the workspace. For example, in industrial assembly tasks that require high precision (e.g. gearbox assembly), using a virtual fixture should speed up the task completion time, as well as reduce the strain on the worker
We are currently working on a follow-up user study where we test whether the impact of these automatically estimated VFs in teleoperation benefit the workers.
The paper from Prado et al. was one of the 3 articles nominated to the IEEE Best paper award - congratulations to all of the researchers' great work!
More information: https://kifabrik.mirmi.tum.de/solutions/telepresence/
Publication: Prado, Diego Fernandez; Larintzakis, Konstantinos; Irsperger, Jan; Steinbach, Eckehard: Accelerating Virtual Fixture Estimation for Robot Manipulation using Reinforcement Learning and Human Demonstrations. 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), 2024
