Having some type of metric to reference makes it easier to work toward fitness goals. If someone wants to eat healthier to lose weight, for instance, counting calories is much better than simply "trying to eat better" without setting measurable boundaries. Another popular metric is step counting. Everything from smartwatches to phones can already collect this type of data, and it provides a reasonably good approximation of one’s level of physical activity.
However, metrics are not magic. The number you see on your screen is only useful if it accurately represents what you are trying to measure. And when it comes to steps, they are not all created equal. Steps taken during a lazy stroll through the park are not the same as those taken while climbing a steep mountain trail. But a typical step counter treats them all in the same exact way.
Climbing steep mountain trails clearly involves far greater physical exertion, and machine learning enthusiast Solomon Githu thinks step counters should reflect that fact. But how might that work? Is it possible to build a compact, practical, wearable device that understands not just that you took a step, but also the type of terrain it occurred on? Githu wanted to find out, so he laced up his walking shoes and got to work.
One small step for machine learning
If a device of this sort is cumbersome or fiddly, no one will ever use it. So any thoughts of using cameras or powerful computers for processing quickly went out the window. Instead, Githu decided to build the device into a tiny box that attaches to a shoe. This design makes it completely transparent to the user, but it also places severe constraints on the hardware and software that powers the system. It has to be very small and extremely energy efficient to be practical for real-world use.
To help reach this goal, Githu chose to work with the Seeed Studio XIAO nRF52840 Sense development board. This tiny board comes equipped with an energy-efficient Nordic Semiconductor nRF52840 microcontroller, a 6-axis inertial measurement unit (IMU), and a Bluetooth radio, all of which would be essential for this project. To protect it from the elements, the hardware was installed in a 3D-printed case with hooks that attach it securely to shoelaces.
Githu planned to use the IMU to collect information about the three-dimensional position and motion of the foot. The raw data would be extremely challenging to interpret, so he decided to build a machine learning classifier to do the job. This classifier would be able to tell when a step was taken, and whether the person using the device was walking uphill, downhill, or on level ground.
But while the nRF52840 is a powerful microcontroller, it could easily choke on a machine learning algorithm. For this reason, the Edge Impulse platform was used to develop the model. Using Edge Impulse, a model can be tuned to work on severely resource-constrained hardware without sacrificing accuracy.
Walking the walk (to collect data)
With a sketch of the project fully worked out, Githu got down to business and started collecting training data for the classifier. Custom firmware was loaded onto the XIAO to make it transmit a steady stream of IMU measurements through its serial interface. Githu walked around a parking garage wearing the device to capture data from inclines, declines, and level ground. This data was received by a laptop, which in turn uploaded the data directly to an Edge Impulse project using the edge-impulse-data-forwarder tool, which is a part of the Edge Impulse CLI. Just a few minutes of data was collected for this proof of concept, but a more diverse dataset would be needed for a real-world deployment.
Initially, I collected around 200 seconds of data for the 3 classes:
flat_surfaceuphill_surfaceanddownhill_surface. However, after deploying the model I observed that there was a huge bias because the model was not trained on still motion. To fix this, I added another class (no_motion) of IMU data from still positions such as standing. Eventually the dataset had 250 seconds of IMU data for the 4 classes.
To make use of this data, Githu turned his attention to the creation of an impulse. An impulse defines precisely how incoming sensor data is processed, from the time it is captured until the moment that the machine learning model makes a prediction. In this case, the incoming data was first split into one-second windows, then a spectral analysis was conducted. This is a great way to surface the most informative features in IMU data. Finally, the features were fed into the classifier, which predicts the type of terrain a person is walking on.
Next, the impulse was trained so that it could learn the unique characteristics of each class of data. After the process completed, metrics were displayed that revealed the classifier had achieved an accuracy level of better than 88%. The model testing tool, which uses only data that was not included in the training process, confirmed this result. This level of performance is more than good enough for a proof of concept, but if better performance was needed, a larger training dataset would do the trick.
Crossing the finish line
Since the XIAO development board is compatible with the Arduino ecosystem, Githu deployed the impulse as an Arduino library. The EON compiler and 8-bit model quantization were enabled during the deployment. This reduces the amount of memory and processing power that is required for inferences without having a meaningful impact on accuracy levels. Githu modified the Arduino source code such that the model’s predictions would be periodically transmitted via Bluetooth.
To wrap things up, he also developed a custom phone app. This app acts as a dashboard that shows a summary of one’s activity over time. And rather than just showing step counts, like traditional fitness apps, this one puts them in context. Counts of steps on inclines, declines, and flat surfaces are shown separately.
Did this project get your creative juices flowing? What other capabilities would you give a machine learning-powered step counter? Or maybe you have ideas for improving something other than a step counter? Whatever it is that you have in mind, reading up on Githu’s work and using Edge Impulse will help to take you from zero to prototype in a weekend’s time.