Pedagogical speech assistant to supplement lectures for students with visual impairments

Supervisors: Matthias Wölfel (HKA), Rainer Stiefelhagen (KIT) or Alex Waibel (KIT)

Faculty: Informatics and Business Information Systems

The digitalization of university teaching has hardly progressed beyond simple non-interactive formats. Common educational practice is focused on visual content in classic lecture formats, such as slides, scripts, click tests, and lecture videos. Thus, the provided information cannot easily be accessed by students with visual impairments and support individual needs.

Speech-based assistance systems offer natural access that do not rely on visual content. Text can be entered using automatic speech recognition and read out using speech synthesis.

While today speech assistance is capable to provide general information, e.g. weather forecast, it does not allow access to individualized information. To overcome this gap current digitally supported teaching practice should be supplemented by an interactive component in natural language. Based on supplementary material to lectures, such as audio recordings, lecture slides and scripts, AI-based voice assistance should automatically be generated. The AI-based voice assistance needs to be adapted to the specific needs of people with visual impairments including adaptation in navigation, input, and output modalities, etc.

In this PhD project, the research fields include:

  • To research obstacles and needs faced by people with visual impairments to access lecture material.

  • To improve the existing conversational assistance system by extracting relevant information from supplementary material to lectures.

  • To design and implement the user interface according to the specific needs of people with visual impairments.

  • To investigate and evaluate the technological progress and to conduct scientific evaluations with end users.

Desired qualifications of the PhD student:

  • University degree (M.Sc.) with excellent grades in computer science or computer science in media

  • Programming knowledge in Python including TensorFlow, PyTorch or similar

  • Pre-knowledge in Natural Language Processing

  • Experience or skills in the design and evaluation of socio-technical assistance systems

  • Good English language skills (your responsibilities include writing publications and giving international presentations)