3D密集面部对齐-基于Pytorch实现的快速+准确+稳定的3D面部对齐算法-优质项目实战.zip

# Towards Fast, Accurate and Stable 3D Dense Face Alignment <p align="center"> <img src="docs/images/webcam.gif" alt="demo" width="512px"> </p> ## Getting started ### Requirements See [requirements.txt](./requirements.txt), tested on macOS and Linux platforms. The Windows users may refer to [FQA](#FQA) for building issues. Note that this repo uses Python3. The major dependencies are PyTorch, numpy, opencv-python and onnxruntime, etc. If you run the demos with `--onnx` flag to do acceleration, you may need to install `libomp` first, i.e., `brew install libomp` on macOS. ### Usage 1. Download the project 2. Build the cython version of NMS, Sim3DR, and the faster mesh render <!-- ```shell script cd FaceBoxes sh ./build_cpu_nms.sh cd .. cd Sim3DR sh ./build_sim3dr.sh cd .. # the faster mesh render cd utils/asset gcc -shared -Wall -O3 render.c -o render.so -fPIC cd ../.. ``` or simply build them by --> ```shell script sh ./build.sh ``` 3. Run demos ```shell script # 1. running on still image, the options include: 2d_sparse, 2d_dense, 3d, depth, pncc, pose, uv_tex, ply, obj python3 demo.py -f examples/inputs/emma.jpg --onnx # -o [2d_sparse, 2d_dense, 3d, depth, pncc, pose, uv_tex, ply, obj] # 2. running on videos python3 demo_video.py -f examples/inputs/videos/214.avi --onnx # 3. running on videos smoothly by looking ahead by `n_next` frames python3 demo_video_smooth.py -f examples/inputs/videos/214.avi --onnx # 4. running on webcam python3 demo_webcam_smooth.py --onnx ``` The implementation of tracking is simply by alignment. If the head pose > 90° or the motion is too fast, the alignment may fail. A threshold is used to trickly check the tracking state, but it is unstable. You can refer to [demo.ipynb](./demo.ipynb) or [google colab](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv) for the step-by-step tutorial of running on the still image. For example, running `python3 demo.py -f examples/inputs/emma.jpg -o 3d` will give the result below: <p align="center"> <img src="docs/images/emma_3d.jpg" alt="demo" width="640px"> </p> Another example: <p align="center"> <img src="docs/images/trump_biden_3d.jpg" alt="demo" width="640px"> </p> Running on a video will give: <p align="center"> <img src="docs/images/out.gif" alt="demo" width="512px"> </p> More results or demos to see: [Hathaway](https://guojianzhu.com/assets/videos/hathaway_3ddfa_v2.mp4). <!-- Obviously, the eyes parts are not good. --> ### Features (up to now) <table> <tr> <th>2D sparse</th> <th>2D dense</th> <th>3D</th> </tr> <tr> <td><img src="docs/images/trump_hillary_2d_sparse.jpg" width="360" alt="2d sparse"></td> <td><img src="docs/images/trump_hillary_2d_dense.jpg" width="360" alt="2d dense"></td> <td><img src="docs/images/trump_hillary_3d.jpg" width="360" alt="3d"></td> </tr> <tr> <th>Depth</th> <th>PNCC</th> <th>UV texture</th> </tr> <tr> <td><img src="docs/images/trump_hillary_depth.jpg" width="360" alt="depth"></td> <td><img src="docs/images/trump_hillary_pncc.jpg" width="360" alt="pncc"></td> <td><img src="docs/images/trump_hillary_uv_tex.jpg" width="360" alt="uv_tex"></td> </tr> <tr> <th>Pose</th> <th>Serialization to .ply</th> <th>Serialization to .obj</th> </tr> <tr> <td><img src="docs/images/trump_hillary_pose.jpg" width="360" alt="pose"></td> <td><img src="docs/images/ply.jpg" width="360" alt="ply"></td> <td><img src="docs/images/obj.jpg" width="360" alt="obj"></td> </tr> </table> ### Configs The default backbone is MobileNet_V1 with input size 120x120 and the default pre-trained weight is `weights/mb1_120x120.pth`, shown in [configs/mb1_120x120.yml](configs/mb1_120x120.yml). This repo provides another config in [configs/mb05_120x120.yml](configs/mb05_120x120.yml), with the widen factor 0.5, being smaller and faster. You can specify the config by `-c` or `--config` option. The released models are shown in the below table. Note that the inference time on CPU in the paper is evaluated using TensorFlow. | Model | Input | #Params | #Macs | Inference (TF) | | :-: | :-: | :-: | :-: | :-: | | MobileNet | 120x120 | 3.27M | 183.5M | ~6.2ms | | MobileNet x0.5 | 120x120 | 0.85M | 49.5M | ~2.9ms | **Surprisingly**, the latency of [onnxruntime](https://github.com/microsoft/onnxruntime) is much smaller. The inference time on CPU with different threads is shown below. The results are tested on my MBP (i5-8259U CPU @ 2.30GHz on 13-inch MacBook Pro), with the `1.5.1` version of onnxruntime. The thread number is set by `os.environ["OMP_NUM_THREADS"]`, see [speed_cpu.py](./speed_cpu.py) for more details. | Model | THREAD=1 | THREAD=2 | THREAD=4 | | :-: | :-: | :-: | :-: | | MobileNet | 4.4ms | 2.25ms | 1.35ms | | MobileNet x0.5 | 1.37ms | 0.7ms | 0.5ms | ### Latency The `onnx` option greatly reduces the overall **CPU** latency, but face detection still takes up most of the latency time, e.g., 15ms for a 720p image. 3DMM parameters regression takes about 1~2ms for one face, and the dense reconstruction (more than 30,000 points, i.e. 38,365) is about 1ms for one face. Tracking applications may benefit from the fast 3DMM regression speed, since detection is not needed for every frame. The latency is tested using my 13-inch MacBook Pro (i5-8259U CPU @ 2.30GHz). The default `OMP_NUM_THREADS` is set 4, you can specify it by setting `os.environ['OMP_NUM_THREADS'] = '$NUM'` or inserting `export OMP_NUM_THREADS=$NUM` before running the python script. <p align="center"> <img src="docs/images/latency.gif" alt="demo" width="640px"> </p>