Skip to main content

Pedagogical Agent that uses Speech and Gestures help K-12 students learn science

Primary Investigators:Gautam Biswas
Marcos Quinones

Brief Description of Project:
We have been developing science learning environments for K-12 students using learning by modeling methods to learn science and computing concepts by building computational models of scientific phenomena. We use AI/ML techniques to track students' progress and difficulties as they work in our environment.

This project will develop virtual agents that can interact with students using speech and gestures to help them overcome their difficulties and advance in their learning tasks. The student working on this project will develop an animated agent that can communicate with students using natural language dialog in web-based learning environments.

Desired Qualifications:
Sufficient programming experience, e.g., should have taken the course CS 3251 - Intermediate Software Design. The NLP modules will be built using Python and Python libraries. The overall system is developed using a Javascript-based web interface.

Interest in AI and ML. Students who have completed CS 4260 - Introduction to Artificial Intelligence will be given preference.

Nature of Supervision:
The PI will get the student started on the research project by providing all of the background material and required research papers.

The PI will also meet with the student at least once a week to review progress, answer questions, and help determine the next steps in the project work.

The student will work very closely with the PI's graduate students working on the research project, and the Research Engineer coordinating the development of the system. Some of the framework for speech analysis has already been designed by the graduate students; the student will add to the already developed framework. C2STEM (Collaborative Computational STEM Learning Environment) is a fully-developed system to which we are adding functionality in the form of virtual agents that interact with students.

A Brief Research Plan (period is for 10 weeks):
Weeks 1 and 2: Background Reading; Familiarity with C2STEM system; and an initial understanding of the design of the virtual agent.

Weeks 3 and 4: Develop a deep understanding of the text analysis approach (we will use a speech-to-text converter to convert student speech into text), and develop a detailed design for the speech understanding module.

Weeks 5 and 6: Implement and test the speech understanding module. Start the design for incorporating the speech understanding modules into the virtual agent framework. Design the speech generation module.

Weeks 7 and 8: Focus on integration of the speech understanding and speech generation modules into the animated virtual agent system. The agent will be capable of making gestures with their hands.

Weeks 9 and 10: Testing and refinement; work with the Research Engineer and graduate students to integrate agent into C2STEM environment for Fall classroom studies. Generate Report.

Number of Open Slots: 1
Contact Information:
Name: Gautam Biswas
Department: Computer Science