I came across a super cool demo by @monolesan that demonstrated pose detection in the browser using Tensorflow's poseNet model. The blur effect was also super cool and effective. I decided that I wanted to have that effect to follow me everywhere and that's how I got the idea to turn it into a Chrome Extension.

My first obstacle was porting the posture detection methods into a package that was a bit easier to add logic to and so I used an npm package react-posenet to basically rebuild the method introduced by @monolesan.

At that point I had the effect working, but due to the nature of the chrome extension ecosystem I had to request camera permissions for every new site / domain due to the pose detection code being in a content script. A quick Stack Overflow search later and I had my answer.

Do the pose detection in the options script which has a static url and pass the pose data for the blur effect to the content script via messaging.

I got that "working" aka detecting good posture vs bad posture, but then ran into more issues to iron out while using the extension.

The implementation of PoseNet / Tensorflow + continuous message streaming is really intensive in the browser

Turns out that PoseNet was really taxing running continuously, and I struggled with this for a while until I came across a more lightweight pose detection model by the name of MoveNet.

The camera only has a message port connection while the window is active

This is due to browsers throttling javascript execution for background / inactive windows. The result is that the user has to make the extension pop-up visible at least once for the extension to function properly. I'm looking into web workers to get around this issue.

Options to adjust the good /bad posture effect

Right now the posture detection works by setting a baseline position of the right eye and then when you deviate from a certain range on the y-axis away from that position, it's considered "bad posture". This works well enough but I think we can be even more accurate by considering the user's shoulder position as well as their head.

You can try the extension here - Chrome Webstore link

and check out the landing page that includes a simple demo of the detection at https://posture.krose.work/