The project is carried out within the framework of the National Recovery and Resilience Plan Greece 2.0, funded by the European Union - NextGenerationEU (Implementation Body: HFRI)
HFRI Project Number:16523
The CARPOS project aims atproviding a novel, coachable and human-friendly mobile robotic system, equipped with a robotic manipulator and multiple sensors, able of capturing and exploiting the lifelong experience of farmers through LbD, in order to successfully perform autonomous fruit picking operations, i.e. harvesting without a cutting tool. To achieve this, the system will be capable of accumulating knowledge through both kinesthetic and/or visual coaching by the human farmers intuitively, during its operation; thereby, providing flexibility and adaptability to different types of crops and easy deployment. Through the use of an intuitive human-machine interface, the farmer will be able to monitor the process performed by the robot, and correct it whenever deemed necessary, by coaching the robot, utilizing the above means.
Host institution: Hellenic Mediterranean University
Department: Electrical and Computer Engineering
Laboratory: Control Systems and Robotics Lab.
Location: Heraklion, Crete, Greece
Duration: 2 years
Starting – ending: November 2023 – November 2025
Funding: H.F.R.I – National Recovery and Resilience Plan “Greece 2.0” funded by the European Union –NextGenerationEU
Scientific area: Engineering Sciences & Technology
Fruit picking, i.e. harvesting without a cutting tool, of two different crops:
The key research objectives of CARPOS to successfully realize its vision are summarized in the following points:
To develop novel Learning by Demonstration (LbD) methods that combine visual data (from the camera) and kinesthetic data (position and forces) for the incremental learning of fruit picking skills.
To develop new compliant control schemes that take into account the complete kinematic chain, i.e. the gripper, the robotic manipulator and the robotic platform will be seen as a single kinematic chain. The purpose of this compliant control scheme will be twofold: on the one hand it will enable the kinesthetic teaching of the picking skills (intentional pHRI) and on the other hand it will guarantee the safety of the surrounding humans (human-teacher, or workers in general) in case of possible unintentional collision.
To utilize and apply currently available computer vision solution to the problem of identifying: a) the fruit’s geometry and pose, b) the configuration of the human hands and c) the contact points between
the human fingers and the fruit.
To develop control schemes for the active perception (tracking) of the human hand, avoiding possible occlusions, in order for the robot to observe the motion pattern during the visual demonstration performed by the farmer.
To combine a mobile robotic platform, a robotic manipulator, a robotic gripper, an in-hand RGB- D camera (attached to the manipulator’s end-effector), static RGB-D cameras (attached to the platform), force/torque sensors and 3D-printed robotic fingers, appropriately designed for fruit picking operations, for the implementation and testing of the developed methodologies.
The project’s work is structured into 6 work packages (WPs). The main purpose of each WP foreseen in the project is outlined below:
WP1 – Project management & definition of the requirements, use-case and KPIs:
The project management will ensure the achievement of the project objectives within time and budget constraints, by planning and organizing the work of the research team. Furthermore, the system requirements and Key Performance Indicators (KPIs) will be extracted and then the use-cases will be defined. This will be a continuous task throughout the project.
WP2 – Active perception:
In this WP, methods for actively inspecting the actions of the farmers will be conducted. In particular, methods for the identification of contact points between the human fingers and the fruit (points-of-interest) will be developed, as wells as control schemes for the visual and active occlusion- free tracking of those points-of-interest by the in-hand camera.
WP3 – Physical human robot interaction and HMI:
In this WP, controllers for tackling the problem of intentional (during kinesthetic teaching) and unintentional physical interaction of the robot with the humans will be researched and developed. The developed control schemes will exploit available knowledge of the geometry of the fruit to assist the human-teacher in her/his teaching through kinesthetic demonstration.
Furthermore, a graphical interface will be developed, providing the user with an appropriate control panel for controlling the robot’s teaching procedure and monitoring its autonomous operation.
WP4 – Skill encoding:
In this WP, novel methods for encoding motions and force/impedance profiles will be investigated. In particular, the developed methods will fuse data both from visual observation and kinesthetic demonstration, in order to encode the fruit picking skill and in turn generalize it to the autonomous robotic harvesting of any other fruit of the same type.
WP5 – System integration and testing:
In this WP, the solutions researched and developed in WP2,3 and
4 will be combined and integrated into a single autonomous robotic solution. Furthermore, this WP will be responsible for the testing of the complete solution both in lab and real setups, based on the pre-defined Key Performance Indicators (KPIs).
WP6 – Dissemination of results and Communication Management:
This WP will address the horizontal activities for the public awareness by communicating the project and disseminating the results.
NEWS
🪴Project CARPOS had a strong presence at last week’s 34th IEEE International Conference on Robot and Human Interactive Communication at TU/e, Eindhoven, with two full paper presentations (delivered by the PI of the project and Nikos Efstathopoulos) and an LBR poster. Our “coachable” harvesting robot, able to learn how to pick totally “new” (for the system) fruits from human demonstrations in just hours, attracted significant scientific interest.
At this year’s European Control Conference (ECC 2025), held in Thessaloniki, Greece, the CARPOS research team had the opportunity to present their results and engage in the exchange of scientific perspectives with researchers from across Europe – and some from around the world.
I was truly honored to participate and open the banquet at this exceptional event. The organizing committee, chaired by Professors G. Rovithakis, Z. Doulgeri, and M. Papageorgiou, along with all the volunteers involved, did an outstanding job!
CARPOS project was presented as a part of the research activities of the Hellenic Mediterranean University at Beyond EXPO 2025, which took place in Athens, Greece, on 6th of April 2025.
Professor Sfakiotakis, a member of the CARPOS research team, represented the project, disseminating its vision, objectives and results to the public.
Agricultural scientist and researcher Ioannis Louloudakis presented our poster and discussed with the scientific community about the ambition and progress of CARPOS project, in Agrivoltaics2024 conference, Athens, 4-6 Nov 2024.
We are very happy to participate by presenting our work at the 32nd Mediterranean Conference on Control and Automation (MED 2024), Chania, Greece. Nikolaos Efstathopoulos, PhD Candidate @ HMU and a member of the CARPOS team, presented our work “ActIVO: An Active perception framework for skill transfer through Iterative Visual Observations”, Dimitrios Papageorgiou, Nikolaos Kounalakis, Nikolaos Efstathopoulos, John Fasoulas, Michael Sfakiotakis at the 32nd Mediterranean Conference on Control and Automation (MED 2024), Chania, Greece, which was developed by our team.
During his visit to the Hellenic Mediterranean University, the Minister of Education, Religious Affairs and Sports, Mr. Kyriakos Pierrakakis was informed about the research activities and scope of the CARPOS project, as a part of the achievements of the Control Systems and Robotics Laboratory.
Assistant professor Dimitrios Papageorgiou (PI), presented the basic idea, targeted problems and concept solution of the CARPOS project to two groups of high-school students that visited the Control Systems and Robotics Laboratory of HMU.
Control Systems and Robotics Laboratory participated in the Innodays 2023 Event, in which the CARPOS members had the opportunity to communicate and discuss the project’s vision and exchange ideas with the rest of the scietific and research community.
We are happy to announce the beginning of our HFRI project CARPOS – Coachable Robot for Fruit Picking Operations!
The kickoff meeting took place @ the CSRL facilities of the Hellenic Mediterranean University.
