RoboCup@Home: How NimbRo Cuts Robot Training Time with Semi-Automatic Image Labels
- AIhub
- 872 reads
NimbRo’s labeling method automates about 60% of object annotations, freeing RoboCup@Home teams to spend more of their short setup window testing robot tasks.

Team NimbRo’s robot competes in the RoboCup@Home arena. Raphael Memmesheimer leads the team.

At RoboCup@Home, a robot may get only two preparation days to learn an unfamiliar apartment and as many as 50 objects before it has to act on its own. Team NimbRo’s work on semi-supervised image annotation tackles that bottleneck: its method automatically labels about 60% of the data, reducing the time spent building object recognition models at the venue.
The research, “Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation,” received the best paper award at the RoboCup 2026 symposium. Raphael Memmesheimer, a postdoctoral researcher at the University of Bonn and leader of its NimbRo@Home team, says the practical value is time returned to competition preparation. Teams can spend more of the brief setup period testing full tasks instead of manually labeling images.
Learning the objects in an unfamiliar apartment
The competition environment is a model apartment that teams have not seen in advance. Organizers source many of the groceries locally, so teams may know to expect drinks, snacks and cutlery without knowing which particular products will be waiting. Some objects come from the publicly available YCB Dataset, which teams can prepare for ahead of time. General object sets also help with pre-training, but they cannot cover every local item.
That distinction matters when a robot must identify a specific drink, rather than merely recognize that it is looking at a drink. Memmesheimer says open-vocabulary methods have had difficulty separating similar objects within categories such as beverages. Promptable detection and segmentation tools have worked for five to 10 classes in the team’s experience, but have not scaled reliably to all 50 competition objects.
NimbRo’s method builds on a model trained with objects seen in earlier competitions. It identifies regions that are likely to contain an object, then uses reference images of the current objects to help label them. Combining object segmentation with those references makes annotation less dependent on people labeling every image by hand. The paper reports that roughly 60% of the data can be labeled automatically through this process.

Example labeled images from different competition venues. Figure from “Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation.”
The team used the method at RoboCup 2025 in Brazil and at the German Open. For the 2026 World Cup in Korea, it extended the approach to generate additional data from video. Memmesheimer says the improved vision model made the two preparation days more useful for testing the robot’s performance across tasks.
One vision system, many kinds of work
At the 2026 competition, NimbRo finished third overall. Its strongest performance came in the new laundry task, where the robot had to move a basket from a washing machine to a folding table, open the machine, remove T-shirts and fold them. The robot managed to open the door and begin folding.
The laundry challenge tests perception and manipulation in close succession. The robot must locate the washer door, identify where to grasp its handle, and pull the door open. It then has to reach into a machine it cannot see inside, determine by touch whether it has grasped a T-shirt, and extract the item before moving to the table. NimbRo attributed its performance in this task in part to the semi-automatic labeling work that improved object vision.
Team NimbRo’s robot removes laundry from a washing machine.
Laundry is one of five main task areas. In human-robot interaction, a robot greets arriving guests: it goes to the door, finds and grasps the handle, lets a person enter, speaks with them, guides them to a seat and introduces them to other guests. The sequence repeats as more people arrive, combining navigation, door handling, speech and social interaction.
Team NimbRo’s robot opens a door for a guest.
A breakfast cleanup task puts the emphasis on manipulation. The robot clears cutlery from a table, takes it to a dishwasher, loads a dishwasher tablet, closes the machine and sets the table for the next meal. Another task gives the robot a general-purpose service request in natural language. It might need to retrieve a particular drink or snack and hand it to someone, including a person from the audience who is unfamiliar to the robot.
The restaurant task changes the setting. Teams know the menu, but the robot is placed in an unknown restaurant and has to scan the space, find customers who are signalling for service and take their orders by voice. The room is often noisy, adding difficulty to speech recognition. Together, these tasks demand a system that can combine perception, navigation, language, planning and physical action in changing conditions.
Competition as a test outside the lab
Before competing, robots must pass an initial inspection for safety and interactive capabilities. After that, they face a small number of attempts in settings where failures can come from several parts of the system at once. A robot has to recognize objects, move through an apartment, interact with people, manipulate items and plan a sequence of actions. Memmesheimer’s view is that progress in any one area helps, but reliable performance depends on the parts working together.
He has competed in the league since his student years and first attended a RoboCup@Home World Cup in 2013. That experience helped shape his decision to pursue a PhD and build an academic career around service robots. He values competitions because they test autonomous robots in real environments: after entering the arena, a robot has to operate without a researcher adjusting each step, unlike a demonstration tuned around a prepared dataset or a single video.
Memmesheimer has seen major changes since he began competing, before deep learning became common. Early systems could map and navigate, but their vision could mistake objects with human-like shapes, such as coat hangers or potted plants, for people. Human perception and speech recognition have improved considerably. Task planning remains difficult, while newer approaches such as large language models, open-vocabulary systems and agent-based architectures continue to shift what teams can attempt.
Progress comes in uneven bursts. Each team faces a broad research workload, and competition robots have to perform reliably despite limited trials. New methods can open up capabilities, but putting them together into a system that completes an entire household task is still demanding.
From three divisions to one league
Memmesheimer recalls winning the competition at RoboCup in Eindhoven, when @Home was divided into three leagues: two based on standard platforms and one open-platform division. NimbRo won the open-platform league. The divisions have since merged into a single competition. In 2026, 24 teams from around the world took part, creating a larger field in which teams could compare their approaches directly.
NimbRo’s next technical focus is manipulation. The team wants its robot to pick up a wider range of objects more quickly and improve two-arm handling, while continuing to test how naturally it works alongside people. Better object annotation can ease one part of preparation; the next measure will be whether those perception gains translate into faster, more dependable actions across the full range of household tasks.
Raphael Memmesheimer
![]() |
Memmesheimer is a postdoctoral researcher in the Autonomous Intelligent Systems group at the University of Bonn. He re-established and leads the domestic service robot team NimbRo@Home and is a founding member of the university’s Center for Robotics. His long-term research goal is to develop autonomous service robots that assist people in everyday environments by perceiving, understanding and learning from human behavior through multiple types of sensory information. He has led teams at two universities in international robotics competitions and has won the RoboCup@Home World Championship five times, including three consecutive titles. His team has also earned multiple RoboCup German Open wins and top placements in the European Robotics League and World Robot Summit. |
