3D Jupyter Notebooks and WebGL Integration for OpenVINO

Recently:

• Coding a human pose demo. The video of the presentation is below.

And the code is here

openvino_notebooks/222-human-pose-estimation-3d.ipynb at main · spencergotowork/openvino_notebooks

This demo uses a third-party toolkit — pythreejs. Why I use it? Because it was already a natural combination of Jupyter and Python. And this helps us not have to write a lot of javascript code, it was created for python graphing on the Web, it is now used by many popular Github projects.

figure.1. Relationships between tools

figure.2. Data flow

This demo allow the user use the webcam or the video file.

• Try to rewrite a simple 3D engine, now it can draw some points and edges, and can be displayed in the form of animation.You can use your keyboards to move the camera. If the user want to display it in the jupyter, he can hook the screen image and display it in jupyter as frame.

I add the code into the utils directory of notebooks.

openvino_notebooks/notebooks/utils/3dengine at main · spencergotowork/openvino_notebooks

Challenges and plans:

• When I convert some of the C++ code to python code, there are some bugs. Therefore, I think it is very important to solve this problem next, otherwise there is no way to use it on other platforms.

figure.3. pose_extractor module in open_model_zoo

• So far there are two different proposals. Which one will be more popular? When I use PyThreejs, I basically don’t have a lot of problems, I don’t have to worry about matrices, projections and the interaction with the mouse, I just have to worry about drawing lines. When I used the new simple engine, even the simplest drawing of the grid required a lot of care, but it was a lot of fun. I don’t know which of the above two schemes do you think is better, or do you have any other better suggestions, how can I use them to be more interesting and meaningful