用神经网络确定权重的matlab代码-dannce:丹斯

 Repository Contributors: Timothy Dunn, Jesse Marshall, Diego Aldarondo, William Wang, Kyle Severson DANNCE (3-Dimensional Aligned Neural Network for Computational Ethology) is a convolutional neural network (CNN) that calculates the 3D positions of user-defined anatomical landmarks on behaving animals from videos taken at multiple angles. The key innovation of DANNCE compared to existing approaches for 2D keypoint detection in animals (e.g. [LEAP](https://github.com/talmo/leap), [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut)) is that the network is fully 3D, so that it can learn about 3D image features and how cameras and landmarks relate to one another in 3D space. We also pre-trained DANNCE using a large dataset of rat motion capture and synchronized video, so the standard network has extensive prior knowledge of rodent motions and poses. DANNCE's ability to track landmarks transfers well to mice and other mammals, and works across different camera views, camera types, and illumination conditions.  ## Example Results #### Mouse  #### Rat  ## DANNCE Installation `DANNCE` requires a CUDA-enabled GPU and appropriate drivers. We have tested DANNCE on NVIDIA GPUs including the Titan V, Titan X Pascal, Titan RTX, V100, and Quadro P5000. On an NVIDIA Titan V, DANNCE can make predictions from ~10.5 frames per second from 6 camera views. DANNCE is also embarrassingly parallel over multiple GPUs. The following combinations of operating systems and software have been tested: | OS | Python | TensorFlow | CUDA | cuDNN | PyTorch | |:---------------------:|:------:|:----------:|:----:|:-----:|:-------:| | Ubuntu 16.04 or 18.04 | 3.7.x | 2.2.0 - 2.3.0 | 10.1 | 7.6 | 1.5.0 - 1.7.0 | | Windows 10 | 3.7.x | 2.2.0 - 2.3.0 | 10.1 | 7.6 | 1.5.0 - 1.7.0 | We recommend installing DANNCE using the following steps: 1. Clone the github repository ``` git clone --recursive https://github.com/spoonsso/dannce cd dannce ``` 2. If you do not already have it, install [Anaconda](https://www.anaconda.com/products/individual). 3. Set up a new Anaconda environment with the following configuration: \ `conda create -n dannce python=3.7 cudatoolkit=10.1 cudnn ffmpeg` 4. Activate the new Anaconda environment: \ `conda activate dannce` 5. Install PyTorch: \ `conda install pytorch=1.7 -c pytorch` 6. Update setuptools: \ `pip install -U setuptools` 7. Install DANNCE with the included setup script from within the base repository directory: \ `pip install -e .` Then you should be ready to try the quickstart demo! \ These installation steps were tested with Anaconda releases 4.7.12 and 2020.02, although we expect it to work for most conda installations. After installing Anaconda, and presuming there are no issues with GPU drivers, the installation should take less than 5 minutes. A note on the PyTorch requirment. PyTorch is not required, but 3D volume generation is significantly faster when using PyTorch than with TensorFlow or NumPy. To use TensorFlow only, without having to install the PyTorch package, simply toggle the `predict_mode` field in the DANNCE configuration files to `tf`. To use NumPy volume generation (slowest), change `predict_mode` to `None`. ## Quickstart Demo To test your DANNCE installation and familiarize yourself with DANNCE file and configuration formatting, run DANNCE predictions for `markerless_mouse_1`. Because the videos and network weight files are too large to host on GitHub, use the links in these files to download necessary files and place them in each associated location: ``` demo/markerless_mouse_1/DANNCE/train_results/link_to_weights.txt demo/markerless_mouse_1/DANNCE/train_results/AVG/link_to_weights.txt demo/markerless_mouse_1/videos/link_to_videos.txt demo/markerless_mouse_2/videos/link_to_videos.txt ``` Alternatively, on Linux you can run the following commands from the base `dannce` repository. For markerless_mouse_1: ``` wget -O vids.zip https://tinyurl.com/DANNCEmm1vids; unzip vids.zip -d vids; mv vids/* demo/markerless_mouse_1/videos/; rm -r vids vids.zip; wget -O demo/markerless_mouse_1/DANNCE/train_results/weights.12000-0.00014.hdf5 https://tinyurl.com/DANNCEmm1weightsBASE; wget -O demo/markerless_mouse_1/DANNCE/train_results/AVG/weights.1200-12.77642.hdf5 https://tinyurl.com/DANNCEmm1weightsAVG ``` For markerless_mouse_2: ``` wget -O vids2.zip https://tinyurl.com/DANNCEmm2vids; unzip vids2.zip -d vids2; mv vids2/* demo/markerless_mouse_2/videos/; rm -r vids2 vids2.zip ``` Once the files are downloaded and placed, run: ``` cd demo/markerless_mouse_1/; dannce-predict ../../configs/dannce_mouse_config.yaml ``` This demo will run the `AVG` version of DANNCE over 1000 frames of mouse data and save the results to: \ `demo/markerless_mouse_1/DANNCE/predict_results/save_data_AVG.mat` The demo should take less than 2 minutes to run on an NVIDIA Titan X, Titan V, Titan RTX, or V100. Run times may be slightly longer on less powerful GPUs. The demo has not been tested using a CPU only. Please see the *Wiki* for more details on running DANNCE and customizing configuration files. ## Using DANNCE on your data ### Camera Calibration To use DANNCE, acquisition cameras must calibrated. Ideally, the acquired data will also be compressed. Synchronization is best done with a frametime trigger and a supplementary readout of frame times. Calibration is the process of determining the distortion introduced into an image from the camera lens (camera intrinsics) and the position and orientation of cameras relative to one another in space (camera extrinsics). When acquiring our data, we typically calibrated cameras in a two-step process. We first used a checkerboard to find the camera intrinsics. We then used an 'L-frame' to determine the camera extrinsics. The L-frame is a calibrated grid of four or more points that are labeled in each camera. A checkerboard can also be used for both procedures. We have included two examples of calibration using MATLAB (in `Calibration/`). Some tips: 1. Try to sample the whole volume of the arena with the checkerboard to fully map the distortion of the lenses. 2. If you are using a confined arena (e.g. a plexiglass cylinder) that is hard to wand, it often works to compute the calibration without the cylinder present. 3. More complicated L-Frames can be used, and can help, for computing the extrinsics. Sometimes using only a four point co-planar L-frame can result in a 'flipped' camera, so be sure to check camera poses after calibration. It is often helpful to compress videos as they are acquired to reduce diskspace needed for streaming long recordings from multiple cameras. This can be done using ffmpeg or x264, and we have included two example scripts in `Compression/`. One, `campy.py`, was written by Kyle Severson and runs ffmpeg compression on a GPU for streaming multiple Basler cameras. A second, CameraCapture, was originally written by Raj Poddar and uses x264 on the CPU to stream older Point Grey/FLIR cameras (eg Grasshopper, Flea3). We have included both a compiled version of the program and the original F-Sharp code that can be edited in Visual Studio. *Mirrors.* Mirrors are a handy way to create new views, but there are some important details when using them with DANNCE. The easiest way to get it all to work with the dannce pipeline is to create multiple videos from the video with mirrors, with all but one sub-field of view (FOV) blacked out in each video. This plays well with the center-of-mass finding network, which currently expects to find only one animal in a given frame. When calibrating the mirror setup, we have used one intrinsic parameter calibration over the entire FOV of the camera, typically by moving the experimental setup away from the camera (moving the camera could cause changes in the in