This is the result of the research which I undertook for the last 3 weeks(Feb 4th - Feb 22nd 2020).
I first started off with trial of shape_predictor_68_landmark_detection with
opencv. by default I used my webcam to capture the landmarks live. This was
pretty interesting as I understood the basic needs of it. It needs some input
with good brightness.
As my final goal was to get the landmark tracks to 3D rig, I took note of the ctrls positions. Initially I was using Luna character courtesy of http://projects.adiendendra.com. (I will get back on this as to why I changed after some time). Here I was taking 41 custom landmark features based on the ctrl positions of character face. next I tried to build my own data set to train the Dlib shape predictor using train_shape_predictor. I made a collection of 256 photos and added the landmarks on each individual photo using imgLab. I tried with all the mentioned parameters on Dlib Api. This was not successful as the number of data set was not sufficient. The original 68 shape predictor was done with 2000+ photos from the ibug 300 dataset. It was not successful because the landmarks were jittery. Further I tried was slicing the 68 based on my need. I took the face landmark point number from the below image and kept them as my landmark.
To Maya:
I tried to run my opencv code in maya. Unfortunately it didn't go past import cv2 erring out with numpy.core.multiarray.
I installed numpy, scipy, matplotlib to python 2.7. also append the sys.path to python2.7 site-packages.
when I executed python2 in terminal, import opencv didn't have any issues.
my friend and ex-collegue prethish bhasuran(prethishb@gmail.com) suggested me that the compiled numpy version does not match with maya compiler version
subprocess was also an option suggested by him.
In one of maya forums it was noted that importing numppy.core.multiarray will solve the issue.
When all this didn't work, I went back to researching on opencv replacement in my code. I was using opencv to convert the image to grayscale and also to print out the landmark on the image.
for the grayscale I replaced with PIL
printing out is not necessary as I am mainly concerned about the x, y values of landmarks.
After re-writing with PIL, there was one more issue of world space which I solved by grouping and scaling the group. When I constrained with ctrls the ctrls lost connection with the rig.
After much trial and error I figured the issue lies with the rig.
changed the rig to malcolm, solved the issue. Tried with another rig from web
which was also successful.
For malcolm rig I took 32 custom landmark features based on the ctrl positions
of character face. There was pre-existing jitter which I wanted to solve without losing accuracy
researched on different smoothing techniques.
Single Exponential smoothing
Double Exponential smoothing(Holts)
Triple Exponential smoothing
Below is Different Double Exponential smoothing(Holts, LASP) with different smoothing levels. The x represents x values of a single landmark point and Y represents time. orange: Actual Data Blue: Smoothed Data smoothing level 500
​smoothing Parameter: 1000
smoothing Parameter: 2000
Although Double Exponential smoothing was bettter than the other 2, it really didn't help me. Stuck to simplified filter with a 0.9 time tolerance. Below is Some Of the Tests I have done with this.
Conclusion: The research has failed but not miserably as it depends on numerous factors
Webcam vs saved video:
opencv does a better mapping with webcam rather than a loading a video.
It also depends upon the quality of the video. I was able to get better output with 2k resolution rather than video recorded from webcam.
videos should have better brightness.
good training sets can make it better.
face ++ can be used import api keys to imglab which will improve the efficiency of landmark plotting.
I am in the phase of making my code efficient and my repo cleaner. I will be pushing my code to github soon. InCase anyone wants it sooner, ping me at vardhan.adiga@gmail.com