Dealing with deteriorating infrastructure is a serious problem that all developed nations face. When bridges, roadways, and buildings are at risk of catastrophic failure from decades of wear and tear, people’s lives and livelihoods can be destroyed in an instant. To minimize the likelihood of infrastructure failures, large teams of trained professionals need to continually monitor vast transportation networks and many thousands of other structures in an attempt to locate the early warning signs of impending problems. Unfortunately, this type of manual monitoring is very costly, often unable to monitor structures as frequently as would be desired, and prone to missing warning signs.
Repetitive work like these infrastructure inspections leads many to seek out automated solutions to replace the labor-intensive manual work presently being done. This is much easier said than done, however, because it takes a trained eye to spot trouble. Applying his background in machine learning to this problem, Naveen Kumar has put forth a possible solution to some of our inspection-related woes. By leveraging a machine learning classifier built with Edge Impulse Studio, Kumar has shown how concrete can be automatically, and continually, inspected for cracks with a high degree of accuracy. Since the first sign of problems that appears in many structures is cracks in concrete, this tool could be very impactful in the future.
To keep the hardware build compact and portable, a Seeed reTerminal was selected by Kumar for his proof of concept. The device comes with a built-in LCD display and some input buttons. A Raspberry Pi Compute Module 4 with 4 GB of RAM powers the reTerminal and provides abundant processing power for the highly-optimized algorithms produced by Edge Impulse. A Raspberry Pi Camera v2 was connected to the reTerminal to allow for the capturing of images. The stock 32-bit Raspberry Pi OS was upgraded to the newer 64-bit flavor to ensure that the system is operating at top speed.
Classifying concrete cracks is a critical part of the device design, but it is not the only important factor. Finding the location of the crack in each image is also important to make use of the device as efficient as possible for end users. That would typically mean that an object detection algorithm would be employed, but Kumar did not want to have to manually add bounding boxes (specifying the crack location) to each image — the dataset he intended to use did not already contain this information. So instead, a specialized type of classifier was used, called a GAP-CNN. This is a convolutional neural network with added global average pooling (GAP) layers that expresses localization of detected classes in the form of a heatmap. GAP-CNN models only require that an image as a whole be assigned a class, for example "crack" or “no crack.”
Kumar downloaded data to train the model from the Mendeley Data dataset, which contains concrete crack images for classification purposes. The dataset is divided up into 20,000 images of cracked concrete, and 20,000 additional images of concrete without cracks. 25,000 further, randomly sampled images were sourced from the COCO-Minitrain dataset consisting of various scenes from the natural world to help prevent the model from inappropriately classifying unexpected images as concrete.
The dataset was uploaded to Edge Impulse using the CLI, where they were automatically assigned labels and split into training and testing sets. Next, an impulse was created to preprocess the image data to optimize downstream analysis steps. A transfer learning MobileNetV2 block was then added to the pipeline, after which Kumar entered into the "expert" mode to tweak the model and add in the GAP layers required for localization. With the model design complete, the training process was kicked off, using the previously uploaded image samples.
The model was quantized to optimize performance on edge computing devices, then was assessed for performance. It was found that the model had achieved an average classification accuracy rate of 99.6%. There is not really much room for improvement with a great result like that, so Kumar moved on to deploying the model to the Seeed reTerminal in preparation for real-world use.
The quantized model was downloaded from Edge Impulse and loaded onto the reTerminal. This model was used with TensorFlow Lite’s inferencing engine with the XNNPACK library of optimized neural network operators enabled for maximal performance. A simple Python script was created to kick off the process by selecting it from the reTerminal’s desktop app. Once started, the device will go to work classifying the images it captures in real-time. By clicking a button, the view can be switched to a heatmap that can be used to localize a detected concrete crack.
Perhaps one day a large number of devices like this will be deployed to continually watch for early signs of problems with concrete structures across an entire city. Until then, take a look at the project documentation to get some great tips on building your own predictive maintenance devices. And when you are ready to start developing your own idea, Kumar’s public Edge Impulse project is the perfect starting point to get you going.
Want to see Edge Impulse in action? Schedule a demo today.