Is it just me, or are video games becoming less innovative with each new console release? There was a time when a new generation of consoles would bring with them a quantum leap in graphics capabilities — but today, nothing much seems to change. And what about the fun and unique control systems, like the Nintendo Power Glove or the Xbox Kinect, that once spiced up our gaming experiences? Now it is tough to find anything other than the same old D-pad and buttons we have had for decades.
As far as graphics are concerned, we might simply be hitting the upper limits of what’s possible. You can’t really do better than photorealistic quality, after all. However, that same argument doesn’t hold true with respect to control systems. It is hard to argue that game pads are the optimal input device for every style of game. So what has happened to all of the innovation we once saw?
All about the Benjamins?
Maker Jallson Suryo thinks it could all come down to economics. Designing an entirely new type of user interface is a very expensive undertaking. If everything doesn’t come together at the end of that process and pay off with adoring fans throwing money at the developer, the bottom line could take a big hit. Maybe what we are seeing is the video game industry’s version of the old saying: “no one ever got fired for buying IBM.” Game pads may not be exciting, but you’re sure to have a job on Monday if your new console uses them.
However, this level of caution may no longer be warranted. As Suryo demonstrated in a recent project, advances in AI have slashed the costs and timelines of these types of development efforts. Inexpensive, off-the-shelf consumer electronics and the user-friendly Edge Impulse platform make it possible to quickly iterate on ideas without breaking the bank.
A game plan for game development
Suryo’s goal was to create a Kinect-like experience and demo it on a few games. If he could build some interesting prototypes by himself on a shoestring budget, then there might be some merit to this idea. Just imagine how much more a well-resourced game studio could accomplish using similar techniques!
The first step in the process would be the development of a gesture recognition system. Once you have a practical system that can accurately recognize different body positions, you can bolt it on to any other software that requires it. To demonstrate the versatility of a generic gesture recognition system, Suryo intended to integrate one into both The Dinosaur Game and Pong.
As far as the hardware for this system is concerned, Suryo selected a Raspberry Pi 5 single-board computer and a basic webcam. A Raspberry Pi is not exactly a beast in terms of raw computing power, but it is inexpensive and extremely user-friendly. Since Edge Impulse was chosen for development of the gesture recognition system, the Raspberry Pi’s modest resources wouldn’t be a problem anyway. Edge Impulse models are tuned for peak performance on far more resource-constrained platforms than this.
The easy part…
With the plan laid out, Suryo jumped into the nuts and bolts of bringing the project to life. He decided on a set of four hand gestures (“neutral” (fist), “five” (open hand), “peace” (V-sign), and “good” (thumbs up)), then collected a dataset consisting of multiple examples of each gesture via webcam. After creating a new project in Edge Impulse, he uploaded the dataset using the Data Acquisition tool. Since the goal was object detection, each gesture was identified by drawing a bounding box around it and assigning a label in the Labeling Queue tool.
Would you believe that Suryo was already halfway to a completed gesture recognition algorithm? To wrap up this part of the project, he pivoted over to the Impulse Design tab. That allowed him to design and configure a pipeline that preprocesses incoming image data and runs a YOLOv5 object detection algorithm that specifies not only which gesture has been recognized, but where in the image it was detected.
Suryo trained the newly-created model by specifying a few parameter values, such as the number of training cycles to complete and the learning rate, then clicking a button to start the process. When all was said and done, a precision score exceeding 96% was reported. That is good enough to ship, so Suryo deployed the model to the Raspberry Pi with the help of the Edge Impulse Linux CLI.
Putting it all together
For the finishing touches on the project, Suryo used the PyGame Python library. This is one of the easiest and fastest ways around to rapidly develop a game prototype. Both The Dinosaur Game and Pong were coded in the usual way, but instead of using key presses to control the action in the game, these versions use gestures.
By continually running the gesture recognition impulse that was previously created in the background, players can simply make gestures in front of their webcam to control the games. Reading the outputs of the model, rather than key presses, allows these actions to easily be integrated into the games.
The prototypes created by Suryo are quite basic, but it is easy to imagine more complex games and interactions being built using the same general approach. Instead of allocating a huge amount of resources that could sink a company, Suryo’s method makes it realistic for a single developer to experiment with wild new ideas in a weekend.
If you are interested in learning more about how the Edge Impulse platform can streamline your development efforts, check out the full project write-up for more details.