video-to-pose3D:一键将视频转换为3D姿势

<p align="center"> <img src="demo/logo.png" width="200" height="100"> </p> ## LightTrack: A Generic Framework for Online Top-Down Human Pose Tracking ### Update 5/16/2019: Add Camera Demo [[Project Page](http://guanghan.info/projects/LightTrack)] [[Paper](https://arxiv.org/pdf/1905.02822.pdf)] [[Github](http://github.com/Guanghan/lighttrack)] [](https://paperswithcode.com/sota/pose-tracking-on-posetrack2017?p=lighttrack-a-generic-framework-for-online-top) **With the provided code, you can easily:** - **Perform online pose tracking on live webcam.** - **Perform online pose tracking on arbitrary videos.** - **Replicate ablation study experiments on PoseTrack'18 Validation Set.** - **Train models on your own.** - **Replace pose estimators or improve data association modules for future research.** #### Real-life Application Scenarios: - [Surveillance](https://youtu.be/P9Bzs3cSF-w) / [Sport analytics](https://youtu.be/PZVGYmr7Ryk) / Security / Self-driving / Selfie video / Short videos (Douyin, Tiktok, etc.) ## Table of Contents - [LightTrack](#LightTrack) * [Table of Contents](#table-of-contents) * [Overview](#overview) * [Prerequisites](#Prerequisites) * [Getting Started](#getting-started) * [Demo on Live Camera](#Demo-on-Live-Camera) * [Demo on Arbitrary Videos](#Demo-on-Arbitrary-Videos) * [Validate on PoseTrack 2018](#validate-on-posetrack-2018) * [Evaluation on PoseTrack 2018](#evaluation-on-posetrack-2018) * [Qualitative Results](#qualitative-results-on-posetrack) * [Quantitative Results](#quantitative-results-on-posetrack) * [Performance on PoseTrack 2017 Benchmark (Test Set)](#quantitative-results-on-posetrack) * [Training](#training) * [Pose Estimation Module](#1-pose-estimation-module) * [Pose Matching Module: SGCN](#2-pose-matching-module) * [Limitations](#limitations) * [Citation](#citation) * [Reference](#reference) * [Contact](#contact) ## Overview LightTrack is an effective light-weight framework for human pose tracking, truly online and generic for top-down pose tracking. The code for [the paper](https://arxiv.org/pdf/1905.02822.pdf) includes LightTrack framework as well as its replaceable component modules, including detector, pose estimator and matcher, the code of which largely borrows or adapts from [Cascaded Pyramid Networks](https://github.com/chenyilun95/tf-cpn) [[1]], [PyTorch-YOLOv3](https://github.com/eriklindernoren/PyTorch-YOLOv3), [st-gcn](https://github.com/yysijie/st-gcn) and [OpenSVAI](https://doc-opensvai.readthedocs.io/en/latest/) [[3]].  In contrast to **Visual Object Tracking** (VOT) methods, in which the visual features are implicitly represented by kernels or CNN feature maps, we track each human pose by recursively updating the bounding box and its corresponding pose in an explicit manner. The bounding box region of a target is inferred from the explicit features, i.e., the human keypoints. Human keypoints can be considered as a series of special visual features. The advantages of using pose as explicit features include: * (1) The explicit features are human-related and interpretable, and have very strong and stable relationship with the bounding box position. Human pose enforces direct constraint on the bounding box region. * (2) The task of pose estimation and tracking requires human keypoints be predicted in the first place. Taking advantage of the predicted keypoints is efficient in tracking the ROI region, which is almost free. This mechanism makes the online tracking possible. * (3) It naturally keeps the identity of the candidates, which greatly alleviates the burden of data association in the system. Even when data association is necessary, we can re-use the pose features for skeleton-based pose matching. (Here we adopt **Siamese Graph Convolutional Networks (SGCN)** for efficient identity association.) **Single Pose Tracking** (SPT) and **Single Visual Object Tracking** (VOT) are thus incorporated into one unified functioning entity, easily implemented by a replaceable single-person human pose estimation module. Below is a simple step-by-step explanation of how the LightTrack framework works.  (1). Detection only at the 1st Frame. Blue bboxes indicate tracklets inferred from keypoints.  (2). Detection at every other 10 Frames. Red bbox indicates keyframe detection.  (3). Detection at every other 10 Frames for multi-person: * At non-keyframes, IDs are naturally kept for each person; * At keyframes, IDs are associated via spatial consistency. For more technical details, please refer to our arXiv paper. ## Prerequisites - Set up a Python3 environment with provided anaconda environment file. ```Shell # This anaconda environment should contain everything needed, including tensorflow, pytorch, etc. conda env create -f environment.yml ``` ### (Optional: set up the environment on your own) - Install [PyTorch](http://pytorch.org/) 1.0.0 (or higher) and TorchVision. (Siamese Graph Convolutional Network) - Install [Tensorflow](https://www.tensorflow.org/install) 1.12. Tensorflow v2.0 is not tested yet. (Human Pose Estimator) - Install some other packages: ```Shell pip install cython opencv-python pillow matplotlib ``` ## Getting Started - Clone this repository and enter the ~~dragon~~ lighttrack folder: ```Shell git clone https://github.com/Guanghan/lighttrack.git; # build some necessities cd lighttrack/lib; make; cd ../graph/torchlight; python setup.py install # enter lighttrack cd ../../ ``` - If you'd like to train LightTrack, download the [COCO dataset](http://cocodataset.org/#download) and the [PoseTrack dataset](https://posetrack.net/users/download.php) first. Note that this script will take a while and dump 21gb of files into `./data/coco`. For PoseTrack dataset, you can replicate our ablation experiment results on the validation set. You will need to register at the official website and create entries in order to submit your test results to the server. ```Shell sh data/download_coco.sh sh data/download_posetrack17.sh sh data/download_posetrack18.sh ``` ### Demo on Live Camera | PoseTracking Framework | Keyframe Detector | Keyframe ReID Module | Pose Estimator | FPS | |:----------:|:-----------:|:--------------:|:----------------:|:---------:| | LightTrack | YOLOv3 | Siamese GCN | MobileNetv1-Deconv | 220* / 15 | - Download weights. ```Shell cd weights; bash ./download_weights.sh # download weights for backbones (only for training), detectors, pose estimators, pose matcher, etc. cd -; ``` - Perform pose tracking demo on your Webcam. ```Shell # access virtual environment source activate py36; # Perform LightTrack demo (on camera) with light-weight detector and pose estimator python demo_camera_mobile.py ``` ### Demo on Arbitrary Videos | PoseTracking Framework | Keyframe Detector | Keyframe ReID Module | Pose Estimator | FPS | |:----------:|:-----------:|:--------------:|:----------------:|:---------:| | LightTrack | YOLOv3 | Siamese GCN | MobileNetv1-Deconv | 220* / 15 | - Download demo video. ```Shell cd data/demo; bash ./download_demo_video.sh # download the video for demo; you could later replace it with your own video for fun cd -; ``` - Perform online tracking demo. ```Shell # access virtual environment source activate py36; # Perform LightTrack demo (on arbitrary video) with light-weight detector and pose estimator python demo_video_mobile.py ``` - After processing, pose tracking results are stored in standardized OpenSVAI format JSON files, located at [**data/demo/jsons/**]. - Visualized images and videos h