Linking across Multimodal Sensors with LLM Generative Agents

The expanding usage of various types of sensors in our daily life has led to generation of ever-growing data lakes and also led to significant shift in how individuals consume data to monitor their health, daily routine, and interact with technology. However, binding data across all different sources of data is not a trivial task and requires access to huge quality labeled datasets. On the other hand, there is only a limited amount of labeled data due to various issues such as the expensive process of annotation, privacy, etc. To address this, self-supervised learning (SSL) models are proposed to leverage the abundance of raw unlabeled data to train models which are less dependent on the availability of labeled datasets.

In this student research project, we aim to leverage the ability of self-supervised models in capturing rich representations of concepts recorded by sensor data and integrate it with the capability of large language models (LLMs) in making meaningful and explainable inferences. Based on recent multimodal SSL works, COCOA [1] and ImageBind[2], we target various sources of data and integrate them with open-sourced LLMs such as LLAMA2 or PALM-E to interpret and reason about the underlying sensor data.

In this project, we expect the student to develop an evaluation setup to examine the capability of our proposed LLM-based model in interpreting and extracting insights from physical activity and physiological data. The project requires the student to be familiar with deep learning concepts and have experience in using either TensorFlow or Pytorch.