CARPOS

Coachable Robot for Fruit Picking Operations

The project is carried out within the framework of the National Recovery and Resilience Plan Greece 2.0, funded by the European Union - NextGenerationEU (Implementation Body: HFRI)

HFRI Project Number:16523

CARPOS ambition:

The CARPOS project aims atproviding a novel, coachable and human-friendly mobile robotic system, equipped with a robotic manipulator and multiple sensors, able of capturing and exploiting the lifelong experience of farmers through LbD, in order to successfully perform autonomous fruit picking operations, i.e. harvesting without a cutting tool. To achieve this, the system will be capable of accumulating knowledge through both kinesthetic and/or visual coaching by the human farmers intuitively, during its operation; thereby, providing flexibility and adaptability to different types of crops and easy deployment. Through the use of an intuitive human-machine interface, the farmer will be able to monitor the process performed by the robot, and correct it whenever deemed necessary, by coaching the robot, utilizing the above means.

About

Host institution: Hellenic Mediterranean University

Department: Electrical and Computer Engineering

Laboratory: Control Systems and Robotics Lab.

Location: Heraklion, Crete, Greece

Duration: 2 years

Starting – ending: November 2023 – November 2025

Funding: H.F.R.I – National Recovery and Resilience Plan “Greece 2.0” funded by the European Union –NextGenerationEU

Scientific area: Engineering Sciences & Technology

CARPOS Challenges

  • To apply Learning by Demonstration (LbD) methods for skill transferring in robotically assisted agriculture.
  • To research and develop novel multi-modal LbD methods for fruit picking skills that take into account the geometry and pose of the fruit and combine both visual observation and kinesthetic data for the training.
  • To tackle the problem of cognitively and physically assisting the human-teacher during the kinesthetic teaching of the fruit picking operation.
  • To tackle the problem of human safety, during the human-robot coexistence and collaboration, by developing novel physical human-robot interaction controllers
  • To research and develop novel active perception controllers for observing the human hand via its visual tracking, that avoid possible occlusions

Use cases:

Fruit picking, i.e. harvesting without a cutting tool, of two different crops:

  • Tomatoes (Elpida variety)
  • Prickly pears

CARPOS Objectives:

The key research objectives of CARPOS to successfully realize its vision are summarized in the following points:

Objective 1:

To develop novel Learning by Demonstration (LbD) methods that combine visual data (from the camera) and kinesthetic data (position and forces) for the incremental learning of fruit picking skills.

Objective 2:

To develop new compliant control schemes that take into account the complete kinematic chain, i.e. the gripper, the robotic manipulator and the robotic platform will be seen as a single kinematic chain. The purpose of this compliant control scheme will be twofold: on the one hand it will enable the kinesthetic teaching of the picking skills (intentional pHRI) and on the other hand it will guarantee the safety of the surrounding humans (human-teacher, or workers in general) in case of possible unintentional collision.

Objective 3:

To utilize and apply currently available computer vision solution to the problem of identifying: a) the fruit’s geometry and pose, b) the configuration of the human hands and c) the contact points between
the human fingers and the fruit.

Objective 4:

To develop control schemes for the active perception (tracking) of the human hand, avoiding possible occlusions, in order for the robot to observe the motion pattern during the visual demonstration performed by the farmer.

Objective 5:

To combine a mobile robotic platform, a robotic manipulator, a robotic gripper, an in-hand RGB- D camera (attached to the manipulator’s end-effector), static RGB-D cameras (attached to the platform), force/torque sensors and 3D-printed robotic fingers, appropriately designed for fruit picking operations, for the implementation and testing of the developed methodologies.

The project’s work is structured into 6 work packages (WPs). The main purpose of each WP foreseen in the project is outlined below:

WP1 – Project management & definition of the requirements, use-case and KPIs:

The project management will ensure the achievement of the project objectives within time and budget constraints, by planning and organizing the work of the research team. Furthermore, the system requirements and Key Performance Indicators (KPIs) will be extracted and then the use-cases will be defined. This will be a continuous task throughout the project.

WP2 – Active perception:

In this WP, methods for actively inspecting the actions of the farmers will be conducted. In particular, methods for the identification of contact points between the human fingers and the fruit (points-of-interest) will be developed, as wells as control schemes for the visual and active occlusion- free tracking of those points-of-interest by the in-hand camera.

WP3 – Physical human robot interaction and HMI:

In this WP, controllers for tackling the problem of intentional (during kinesthetic teaching) and unintentional physical interaction of the robot with the humans will be researched and developed. The developed control schemes will exploit available knowledge of the geometry of the fruit to assist the human-teacher in her/his teaching through kinesthetic demonstration.
Furthermore, a graphical interface will be developed, providing the user with an appropriate control panel for controlling the robot’s teaching procedure and monitoring its autonomous operation.

WP4 – Skill encoding:

In this WP, novel methods for encoding motions and force/impedance profiles will be investigated. In particular, the developed methods will fuse data both from visual observation and kinesthetic demonstration, in order to encode the fruit picking skill and in turn generalize it to the autonomous robotic harvesting of any other fruit of the same type.

WP5 – System integration and testing:

In this WP, the solutions researched and developed in WP2,3 and
4 will be combined and integrated into a single autonomous robotic solution. Furthermore, this WP will be responsible for the testing of the complete solution both in lab and real setups, based on the pre-defined Key Performance Indicators (KPIs).

WP6 – Dissemination of results and Communication Management:

This WP will address the horizontal activities for the public awareness by communicating the project and disseminating the results.

NEWS

  • 29/3/2024 Presentation of CSRL to high-school students

    Assistant professor Dimitrios Papageorgiou (PI), presented the basic idea, targeted problems and concept solution of the CARPOS project to two groups of high-school students that visited the Control Systems and Robotics Laboratory of HMU.

  • 22/3/2024 – Assistant professor Ioannis Fasoulas presented the vision, perspectives and results of the CARPOS project to the representatives of visited German Universities at the facilities of HMU. (22nd of March, DAAD visit to Hellenic Mediterranean University Campus)

  • CSRL participation in Innodays Exhibition. 24-26 Nov. 2023

    Control Systems and Robotics Laboratory participated in the Innodays 2023 Event, in which the CARPOS members had the opportunity to communicate and discuss the project’s vision and exchange ideas with the rest of the scietific and research community.

  • 15 Nov. 2023 – Kickoff meeting

    We are happy to announce the beginning of our HFRI project CARPOS – Coachable Robot for Fruit Picking Operations!

    The kickoff meeting took place @ the CSRL facilities of the Hellenic Mediterranean University.