I came across a super cool demo by @monolesan that demonstrated pose detection in the browser using Tensorflow's poseNet model. The blur effect was also super cool and effective. I decided that I wanted to have that effect to follow me everywhere and that's how I got the idea to turn it into a Chrome Extension.

My first obstacle was porting the posture detection methods into a package that was a bit easier to add logic to and so I used an npm package react-posenet to basically rebuild the method introduced by @monolesan.

At that point I had the effect working, but due to the nature of the chrome extension ecosystem I had to request camera permissions for every new site / domain due to the pose detection code being in a content script. A quick Stack Overflow search later and I had my answer.

Do the pose detection in the options script which has a static url and pass the pose data for the blur effect to the content script via messaging.

I got that "working" aka detecting good posture vs bad posture, but then ran into more issues to iron out while using the extension.

The implementation of PoseNet / Tensorflow + continuous message streaming is really intensive in the browser

Turns out that PoseNet was really taxing running continuously, and I struggled with this for a while until I came across a more lightweight pose detection model by the name of MoveNet.

The camera only has a message port connection while the window is active

This is due to browsers throttling javascript execution for background / inactive windows. The result is that the user has to make the extension pop-up visible at least once for the extension to function properly. I'm looking into web workers to get around this issue.

Options to adjust the good /bad posture effect

Right now the posture detection works by setting a baseline position of the right eye and then when you deviate from a certain range on the y-axis away from that position, it's considered "bad posture". This works well enough but I think we can be even more accurate by considering the user's shoulder position as well as their head.

You can try the extension here

and check out the landing page that includes a simple demo of the detection at https://posture.krose.work/