An elderly man wakes up at dawn in his small studio apartment. He lights the stove to make a pot of tea, switches on the toaster oven, and takes some bread and jelly from the cupboard. After gulping down his morning medication, a computer-generated voice gently reminds him to turn off the toaster. Later that day, his daughter accesses a secure Web site where she scans a check-list, completed by the sensor network in her father's apartment. He's eating regularly, taking his medicine on schedule, and continuing to manage life on his own. That information puts her mind at ease.
This is just one application of Human Activity Recognition being developed by Intel Research Seattle and the University of Washington. Human Activity Recognition melds sensor networks with new data mining and machine learning techniques to infer a wide range of human activities. Human Activity Recognition developers believe that by enabling ubiquitous computers to observe our behavior, with our permission of course, those computers will be better suited to act on our behalf. The paradigm is known as Proactive Computing.
"If computers can really understand what we're up to, maybe they'll do the right things more often," says laboratory director James Landay, also a professor of computer science and engineering at the University of Washington.
Early in their work, the researchers realized that the easiest way for a computer to deduce what we're doing is by identifying the objects we're interacting with. According to former Human Activity Recognition principal investigator Ken Fishkin, the approach is best expressed in vintage horror films featuring the Invisible Man.
The Invisible Man makes a phone call
"You know that the Invisible Man is in the bathroom and you're watching his toothpaste and toothbrush move around," Fishkin says. "You can't see him, but you're still pretty sure that he's brushing his teeth."
After realizing that many activities can be inferred in that way, the next step was to devise a sensor network capable of making those kinds of observations.
While computer vision technology continues to improve, Landay says that it still isn't ready for prime time. Human Activity Recognition calls for a sensor network that's cheap, easy-to-deploy, and not likely to break under the stresses of daily life. After all, someone's life may depend on it.
The researchers found their solution in Radio Frequency Identification (RFID) tags, tiny microchips that contain a globally unique serial number. When RFID tags are interrogated with a radio signal from a nearby reader device, they answer by transmitting that serial number. Already, RFID tags are used to track some pallets of goods through the supply chain and pharmaceuticals in hospitals. Eventually, RFID tags may replace UPC barcodes on consumer products, enabling things like automated check-out at grocery stores.
"Although an RFID tag doesn't tell you much, it can say something as simple as 'I am on a bottle of shampoo,'" Fishkin says. "No matter what room that bottle is in or who picks it up, an RFID reader can still identify it as a bottle of shampoo."
The iGlove is a wearable RFID reader. The Handheld RFID Reader software toolkit is available as a free download.
Rather than use traditional wall or ceiling-mounted industrial RFID readers in their Human Activity Recognition experiments, the researchers designed a wearable reader called the iGlove. As soon as the user picks up a tagged object, the iGlove interrogates the RFID chip and wirelessly transmits that data to a central computer. While the iGlove is useful for their early experiments, the researchers hope the next generation of
wearable reader will be in the less-cumbersome form of a wristwatch or bracelet.
This summer, Human Activity Recognition was tested in an early pilot deployment at the University of Washington Medical School. As an expert and a novice each performed the same medical procedure--in this case, preparing a breathing tube for insertion--the "traces" of their activities were recorded by the iGlove. The goal is to build an educational aid that infers the behavior of anesthesiology students as they practice various procedures and compares their performance to that of an expert.
"The hope is that students can practice and assess their skills themselves without the doctor present," Fishkin says. "In the long-term, the system could even help guide them through the procedures with warning messages or other useful information."
The Medical School study was not the first time the iGlove was put through its paces though. Late last year, the researchers tested the system's ability to monitor more than a dozen Activities of Daily Living, a set of activities including cooking, taking medication, and performing housework that are taken into consideration by caregivers when measuring cognitive decline in the elderly. In this study, Human Activity Recognition logged numerous ADLs with no human intervention. For example, if the system detected that the wearer has picked up a tea kettle and also a carton of milk, it inferred that he was making tea.
Fortunately, the researchers developed a novel way to avoid manually entering the ingredients and sequence of every ADL that the system should recognize. A massive repository of these "activity models" already exists.
"The Web has step-by-step instructions on how to do everything from cook a meal to fix a leaky faucet," Landay says.
Still, the nuggets of useful information have to be located and parsed into a form that the Human Activity Recognition software can deal with. To that end, Intel Research scientist Matthai Philipose and his colleagues developed the Probabilistic Activity Toolkit (PROACT), software that mines text documents and the Web for activity structures. Given those "sets of instructions," and the sequences of objects detected by the iGlove, the PROACT's machine learning algorithm statistically infers what the user is doing.
The researchers are currently honing Human Activity Recognition in preparation for the next pilot deployment, most likely in the realm of proactive healthcare for the elderly.
"This is a chance to take a real and inexpensive technology outside the lab and actually help people get through their day," Fishkin says. "If we could reduce caregiver stress by just a few percent, I would consider that to be a huge success."
