Skip to main content

Trinetra: Mobile Technologies for Assisting the Blind & Visually Impaired

Researchers: Priya Narasimhan, Moshe Mahler

Research Area: Mobility | Secure Home Computing

Abstract

The initial scope of the Trinetra system was to assist the blind and visually impaired with the task of grocery shopping, using the Carnegie Mellon campus store for our first deployment. We have since developed other assistive technologies, a currency identifier and a commute planner system - also targeted for the visually-impaired. In keeping with our philosophy of cost-effectiveness, we have used off-the-shelf components, such as a smart phone that is commonly owned by blind people and has accessibility features such as text-to-speech software. Our approach involves that Internet-enabled and Bluetooth-enabled smart phone in conjunction with an embedded Bluetooth-based barcode-scanning device (Baracoda's barcode-scanning pen.) The blind shopper would be able to locate and browse items in the store without assistance of a store clerk. The idea is to provide for enough levels of identification within the store to allow a blind shopper to find the right aisle and shelf containing the product of interest (e.g. soup) and then to discriminate between different products of the same type (e.g. chicken soup versus tomato soup) once the overall shelf location of the product of interest has been identified.

This project will undertake three fundamental activities that will be essential to making Trinetra a reality - core infrastructural development, cognitive enhancement and quality-of-service aspects.

Core Infrastructual Development: We have started to explore the use of optical character recognition and Radio Frequency Identification (RFID) technologies within the context of Trinetra.

Cognitive Enhancements: Most users tend to purchase the same common grocery items on a weekly basis, and other items on a monthly basis. We incorporated a cache on the smart phone that stores the barcode-to-text mappings of commonly purchased items. We intend to make this cache cognitively smarter over time by recording the user's shopping purchases. We will then be able to weigh each item's importance and therefore its permanence in the cache. This will allow us to use a a smart cache replacement policy, and more importantly, an anticipatory cache-replacement policy where the cache can anticipate that a user might require a refill of specific item.

Quality of Service:We are exploring security and privacy concerns. With a user accessing so much information from his/her surroundings, there is an equal possibility of the user's information (e.g. preferences for specific items, favorite bus stops) to be accessed by individuals who wish to exploit this information for malicious ends.