← Portfolio04 / MOBILE INTELLIGENCE · UBICOMP/ISWC 2024

Personal AI, on device.

We built a research prototype that combines smartphone sensing with a language model running on the same Android phone.

Conference demo · UbiComp/ISWC 2024

Published by ACM SIGCHI.

Watch on YouTube

Giving the model context from the phone

A generic prompt can miss what happened during a person’s day. We explored whether locally collected screen text and short self-report questionnaires could give the model enough context to respond more specifically. Keeping inference on the phone lets this pipeline run without sending the prompt to a cloud LLM.

How the prototype works

  1. Collect and export. We extended AWARE-Light to export sensing data to files on Android. The case study used screen text and periodic experience-sampling questionnaires.
  2. Assemble the prompt. A template brings together the question, user and domain context, sensing data and requested output format. This makes it possible to see what information the model is being asked to use.
  3. Generate locally. We ran Llama 3 8B through llama.cpp in Termux on a Pixel 8 Pro. The Android tool Automate helped trigger the pipeline and present the response.

What the demo shows

The student case study combines a summary of phone activity with questionnaire responses to generate a personalised reflection. The demo illustrates the path from data collection to a context-aware answer on the phone.

The trade-offs of running locally

The paper reports approximately 16.5% RAM usage and 3% battery drain over five minutes of on-device inference in this case study. These measurements come from one prototype and handset. The paper also identifies occasional output errors and calls for broader sensor coverage and larger user studies.

The prototype demonstrates that the workflow can run on a phone. The quality and usefulness of its recommendations across users remain open research questions. Read the demo paper

← Back to portfolioTalk about this project