It seems like whenever the topic of a discussion turns to potholes, everyone inevitably believes that no one has a worse problem with them than they do in their area. That should not be entirely surprising considering how prevalent this issue is. On average, potholes cost US drivers more than $3 billion in vehicle damage each year. And that does not even factor in time spent making roadside repairs or waiting for a tow truck.But how is it that we find ourselves in this mess? Local governments have teams devoted to maintaining roadways and filling potholes, after all.
A major reason why potholes are not repaired in a timely manner is a lack of information — the local crew responsible for the repair needs to be made aware of the problem. How often have you driven by a pothole without reporting it? If you are like most people, you probably have no idea where to report the pothole even if you had the time to do so. Recognizing this problem, Justin Lutz has developed an automated tool to detect and report potholes using computer vision and machine learning. His solution was surprisingly simple and made use of off-the-shelf hardware and Edge Impulse.
The hardware consists of the tiny but mighty Sony Spresense development kit, along with a Spresense camera board and Spresense LTE extension board. The main development board comes standard with a six-core CPU to make short work of running tinyML inferences, and a GPS receiver that can be used to determine the precise geographical location of the device. The camera board gives the device the ability to capture images of its environment, in this case to seek out potholes. The LTE extension board makes it possible to communicate wirelessly over large areas using cellular networks. Lutz envisions these assembled components being installed in vehicles that regularly traverse large segments of the roadway, like mail and garbage trucks.
But before the pothole detector can be deployed to the field, it needs to be able to, well, detect potholes. To add the required intelligence, Edge Impulse Studio was used to build a machine learning classifier. Lutz found a dataset containing examples of both potholes and normal, undamaged road segments and uploaded them using the data acquisition tool. An impulse was then designed, starting with a step to preprocess the image data to extract the most important features. After that, the features were fed into a MobilenetV1 96x96 0.1 neural network classifier that can predict if it is seeing normal road conditions or a pothole.
The model was trained using the previously acquired data, then evaluated for performance. It was found that correct classifications were made approximately 80% of the time on average. Not a bad start at all. With a bit more data, this accuracy would be expected to improve significantly, but this test certainly proves Lutz’s concept to be sound.
Having completed development of the classification pipeline, it was time to deploy it to the physical device hardware. One of the options available in Edge Impulse makes it possible to deploy the model as an Arduino library. This offers a lot of flexibility in designing your own application logic around the machine learning pipeline, so Lutz chose to go this route. In addition to detecting potholes, his code also determines GPS coordinates and sends an MQTT message to a remote server when a pothole is detected. This information can be used by the authority responsible for road repairs to map out all current potholes.
It is easy to imagine a fleet of vehicles equipped with a device like this keeping a map of road maintenance issues updated. Such a device could save motorists a lot of money, and perhaps even save lives by making roads safer. It is amazing to think how simple it is to build impactful projects like this, quickly and inexpensively, with Edge Impulse and off-the-shelf hardware. To learn more, or to adapt the methods to your own problem of interest, take a look at the project write-up.
Want to see Edge Impulse in action? Schedule a demo today.