Integrating PoseNet with Phaser 3 by writing my own plugin.
For a personal project I wanted to develop an interactive experience for children using PoseNet.
PoseNet is “a machine learning model which allows for real-time human pose estimation in the browser”. More on PoseNet and how it works can be found in this article: “Real-time Human Pose Estimation in the Browser with TensorFlow.js”.
For the first two or three weeks, my project was built with plain javascript, javascript canvas and an integration with WebGL/ Three.js to spawn the targets. This worked and I’ve learned a few things about integrating the relative coordinates of PoseNet in the coordinate system of Three.js but it wasn’t exactly the best way to go.
Since I actually had a game in mind (start-tutorial-game-ending-time out), I got the advice to switch to the game framework, Phaser.
Creating a PoseNet plugin for Phaser
I already had set up my PoseNet model which received the webcam stream of the user and then analysed the possible key points (nose, eyes, ankles, wrists..). So I had to find a way to access these key points and handle them in my all of the scenes in Phaser. At first I created a start-up scene (similar to a preload scene) where I loaded the webcam en the Posenet model and passed that on to the other scenes by using the scene data. Each scene then did a pose estimation and handled the key points.
This worked, but regarding performance and optimisation, this wasn’t the best way either. So I then got the advice to write my own Phaser plugin. “Writing your own plugin” sounds rather fancy and difficult which frightened me at first, but it’s actually not that big of a deal.
I googled “phaser 3 plugin template” and the first thing that came up was this: https://github.com/photonstorm/phaser3-plugin-template. A very handy template to create a scene plugin. Note: from what I understand Phaser uses scene plugins and global plugins, they both work differently and have their own advantages depending on the way you want to use them.
Seeing this was the first good source I found, I decided to implement this template in my project. I wrote my posenet functionalities in there and returned the different poses so all the scenes could access the poses and their key points. The performance dropped and the pose estimation slowly happened frame by frame (in comparison with a relative smooth estimation before). After a few logs I discovered that what had happened was that I was (perhaps wrongly) used the scene plugin. Which was started for all the scenes (11 scenes at that time) and was run 11 times at the same time. Of course my performance was bad!
So I went back to Google and then opted for a global plugin. I made sure this only started once and integrated my pose net functionalities in there as well. The performance was good, the key points where returned and I had access to my global plugin in all the scenes. So end good, all good.
The plugin
Now for the PoseNet-Phaser-plugin, you can fork/ copy/ download it (whatever you want) via my Github and I’ve also added an example. I think it’s an interesting option for alternative game control and proves to be fun addition to your game. :)
https://github.com/gwen-bo/posenet-plugin-phaser
Integrating the plugin in Phaser 3
I imported and initialised the global plugin in my index.js (game config).
And handled the functions in all the different scenes like this (inspired by an example on Phaser Examples). There are probably better ways to handle the plugin and the functionalities, but for my game this worked fine and improved my performance.
Thank you for reading!
