Development of an Augmented Reality based Interactive Assistance System to support people with cognitive impairments to work on Applications in production and / or distribution environment

Supervisors: Christian Wurll (HKA), Björn Hein (HKA)

Research Entity: “Robotics and Autonomous Systems" Group at the Institute of Applied Research

Many tasks in a production or distribution environment require certain skills and especially cognitive capabilities, which are sometimes difficult to fulfill by people with cognitive impairments or with a low level of qualification. Typical examples are e.g. assembly tasks in production or order picking and packing tasks in distribution centers. In order to improve the productivity some companies have experimented with the usage of augmented and / or mixed reality features. But these approaches have not considered the restrictions imposed by workers with cognitive impairments.

In this project we will develop an augmented reality system for selected work stations which can be operated either manually or in a human robot collaborative approach. The AR system will be extended by a multi-modal sensor and actor system (e.g. sound, vibration, etc.) to support the worker with cognitive impairments. Thus the system can adapt its support level depending on the given constraints. Complex written instructions must be converted into simple instructions and can be presented either on a monitor or audio system. Finally a sensor system shall monitor the stress level of the worker to eventually slow down the operation in case he or she is overwhelmed with the given task.

In this PhD project, the research fields include:

  • To develop an augmented reality system for typical production and distribution tasks with a set of features to support workers with cognitive impairments

  • To develop and integrate multi-modal sensor and actor systems to measure the stress level and to guide the worker

  • To develop a method how the work can be split between a collaborative robot and a worked depending on the worker’s capabilities

Desired qualifications of the PhD student:

  • University degree (M.Sc.) with excellent grades in Computer Science or related fields

  • Strong programming skills in at least one programming language (preferably Python)

  • Knowledge in design and programming of robotics with experience in ROS2

  • Experience in the usage of virtual and augmented reality systems (Unity 3D, Microsoft HoloLens2

  • Good English language skills (your responsibilities include writing publications and giving international presentations)

  • Interest in working with people with visual impairment