Hearing problems are remarkably prevalent worldwide, affecting a substantial portion of the global population. According to the World Health Organization, over 466 million people, or approximately 5% of the world’s population, experience a disabling hearing loss. This figure includes individuals of all ages, from children to the elderly. Moreover, as the global population ages, the number of adults with hearing impairments is expected to rise significantly. It is estimated that by 2050, the number of people with hearing impairments could rise to over 900 million.
Hearing loss can have far-reaching consequences, affecting communication, education, employment opportunities, and overall quality of life for those affected. Driving, in particular, can present unique challenges for individuals who are hearing-impaired, as auditory cues play a crucial role in road safety. One of the most significant difficulties faced by the hearing-impaired while driving is the inability to hear important sounds, such as sirens from emergency vehicles. Emergency sirens serve as vital warnings, alerting drivers to quickly respond and make way for ambulances, fire trucks, or police cars rushing to respond to urgent situations. Without the ability to hear these sirens, hearing-impaired drivers may not be aware of nearby emergency vehicles, leading to potentially hazardous situations.
Engineer Jan Říha realized that recent advances in machine learning and edge computing have opened up a tremendous opportunity to assist the hearing-impaired in these situations, and perhaps even save lives. He developed a small device called PionEar that uses an audio classification algorithm to recognize the sounds of emergency vehicles, then provide a visual cue to alert the driver. Not only was the unit designed to be simple to use, but it was also pretty simple to create, thanks to some great off-the-shelf hardware and the Edge Impulse machine learning development platform.
At the heart of the PionEar is a Syntiant TinyML Board. With an Arm Cortex-M0+ processor and 256 KB of flash memory, this is a very capable platform for mobile devices. And with an NDP101 Neural Decision Processor onboard, Syntiant’s TinyML can make short work of running inferences with machine learning models that have been highly optimized for resource-constrained devices by Edge Impulse. The board also has several sensors, including just the sort of high-quality microphone that is needed to build PionEar, making it an all-in-one solution for many use cases.
The TinyML board slowly sips power, so it can operate on a small LiPo battery for several days without a recharge. But Říha also included a solar cell and MPPT charge controller such that, under ideal conditions, the user can install the device, then just forget about it. A custom PCB is wired to the TinyML board to provide the alert system — a series of LEDs in the pattern of the PionEar logo. A light sensor is also included such that the LEDs are not too bright under conditions of low light, potentially distracting the driver.
With the hardware sorted out, Říha turned his attention to building the machine learning classification pipeline with Edge Impulse Studio. The first step in building a classifier is either tracking down an appropriate dataset, or collecting that data manually. Fortunately in this case, there are many publicly available options to choose from. Data was taken from multiple datasets containing the sounds of emergency vehicle sirens, other road sounds, as well as talking and other normal, ambient sounds that may be encountered while driving. A diverse, high-quality dataset is an essential component in creating an accurate, well-generalized machine learning model, so always take care to not breeze through this step.
The dataset was uploaded to Edge Impulse using the Data Acquisition tool, which automatically assigned labels based on the file names, and also split the data into training and test sets. After that, Říha begin work on designing the impulse. Created with a simple GUI, this impulse defines how data flows through the pipeline, from a raw audio sample to a prediction of its class. Preprocessing steps were added to split the audio into small segments, then extract the most relevant features to minimize the amount of hardware resources that would be needed downstream. Finally, the features were forwarded into a neural network classifier that can distinguish between normal, background noises and the sirens of emergency vehicles.
A few model hyperparameters were adjusted, and the data augmentation feature was turned on to help the model to better generalize to unseen conditions, then the training process was initiated. After it completed, metrics were provided to assist in understanding how well the model is performing. A very nice average classification accuracy of 96.7% was observed.
To validate that result, the more stringent Model Testing tool, which uses data that was not included in the training process, was also utilized. This backed up the previous result, showing a 96.3% rate of accuracy. That is excellent for a proof of concept, so Říha used the Deployment tab in Edge Impulse Studio to deploy the full classification pipeline to the physical hardware — you do not want to have a reliance on the cloud while driving, not to mention the privacy concerns associated with sending a constant stream of audio from the inside of your car over the public Internet.
Since the Syntiant TinyML Board is fully supported by Edge Impulse, this is very simple to do. There are a number of options available, ranging from downloading a prebuilt binary containing the full pipeline, to an NDP101 library that can be used when building a custom application.
After deployment, Říha took the PionEar out for a real-world test in the car to see how well it performed. It was demonstrated as being smart enough to not trigger inappropriately, and when an emergency vehicle passed by with sirens on, the device lit up and alerted the driver exactly as would be expected.
Not only is the PionEar easy to build, but it is also inexpensive. Would this device benefit you, or someone you know? If so, check out the project write-up and the public Edge Impulse Studio project for all the details you need to build your own.
Want to see Edge Impulse in action? Schedule a demo today.