pytorch3d-main.zip

# Introduction Implicitron is a PyTorch3D-based framework for new-view synthesis via modeling the neural-network based representations. # License Implicitron is distributed as part of PyTorch3D under the [BSD license](https://github.com/facebookresearch/pytorch3d/blob/main/LICENSE). It includes code from the [NeRF](https://github.com/bmild/nerf), [SRN](http://github.com/vsitzmann/scene-representation-networks) and [IDR](http://github.com/lioryariv/idr) repos. See [LICENSE-3RD-PARTY](https://github.com/facebookresearch/pytorch3d/blob/main/LICENSE-3RD-PARTY) for their licenses. # Installation There are three ways to set up Implicitron, depending on the flexibility level required. If you only want to train or evaluate models as they are implemented changing only the parameters, you can just install the package. Implicitron also provides a flexible API that supports user-defined plug-ins; if you want to re-implement some of the components without changing the high-level pipeline, you need to create a custom launcher script. The most flexible option, though, is cloning PyTorch3D repo and building it from sources, which allows changing the code in arbitrary ways. Below, we descibe all three options in more details. ## [Option 1] Running an executable from the package This option allows you to use the code as is without changing the implementations. Only configuration can be changed (see [Configuration system](#configuration-system)). For this setup, install the dependencies and PyTorch3D from conda following [the guide](https://github.com/facebookresearch/pytorch3d/blob/master/INSTALL.md#1-install-with-cuda-support-from-anaconda-cloud-on-linux-only). Then, install implicitron-specific dependencies: ```shell pip install "hydra-core>=1.1" visdom lpips matplotlib accelerate ``` Runner executable is available as `pytorch3d_implicitron_runner` shell command. See [Running](#running) section below for examples of training and evaluation commands. ## [Option 2] Supporting custom implementations To plug in custom implementations, for example, of renderer or implicit-function protocols, you need to create your own runner script and import the plug-in implementations there. First, install PyTorch3D and Implicitron dependencies as described in the previous section. Then, implement the custom script; copying `pytorch3d/projects/implicitron_trainer` is a good place to start. See [Custom plugins](#custom-plugins) for more information on how to import implementations and enable them in the configs. ## [Option 3] Cloning PyTorch3D repo This is the most flexible way to set up Implicitron as it allows changing the code directly. It allows modifying the high-level rendering pipeline or implementing yet-unsupported loss functions. Please follow the instructions to [install PyTorch3D from a local clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone). Then, install Implicitron-specific dependencies: ```shell pip install "hydra-core>=1.1" visdom lpips matplotlib accelerate ``` You are still encouraged to implement custom plugins as above where possible as it makes reusing the code easier. The executable is located in `pytorch3d/projects/implicitron_trainer`. > **_NOTE:_** Both `pytorch3d_implicitron_runner` and `pytorch3d_implicitron_visualizer` executables (mentioned below) are not available when using local clone. Instead users should use the python scripts `experiment.py` and `visualize_reconstruction.py` (see the [Running](Running) section below). # Running This section assumes that you use the executable provided by the installed package (Option 1 / Option 2 in [#Installation](Installation) above), i.e. `pytorch3d_implicitron_runner` and `pytorch3d_implicitron_visualizer` are available. > **_NOTE:_** If the executables are not available (e.g. when using a local clone - Option 3 in [#Installation](Installation)), users should directly use the `experiment.py` and `visualize_reconstruction.py` python scripts which correspond to the executables as follows: - `pytorch3d_implicitron_runner` corresponds to `<pytorch3d_root>/projects/implicitron_trainer/experiment.py` - `pytorch3d_implicitron_visualizer` corresponds to `<pytorch3d_root>/projects/implicitron_trainer/visualize_reconstruction.py` For instance, in order to directly execute training with the python script, users can call: ```shell cd <pytorch3d_root>/projects/ python -m implicitron_trainer.experiment <args>` ``` If you have a custom `experiment.py` or `visualize_reconstruction.py` script (as in the Option 2 [above](#Installation)), replace the executable with the path to your script. ## Training To run training, pass a yaml config file, followed by a list of overridden arguments. For example, to train NeRF on the first skateboard sequence from CO3D dataset, you can run: ```shell dataset_args=data_source_ImplicitronDataSource_args.dataset_map_provider_JsonIndexDatasetMapProvider_args pytorch3d_implicitron_runner --config-path ./configs/ --config-name repro_singleseq_nerf \ $dataset_args.dataset_root=<DATASET_ROOT> $dataset_args.category='skateboard' \ $dataset_args.test_restrict_sequence_id=0 test_when_finished=True exp_dir=<CHECKPOINT_DIR> ``` Here, `--config-path` points to the config path relative to `pytorch3d_implicitron_runner` location; `--config-name` picks the config (in this case, `repro_singleseq_nerf.yaml`); `test_when_finished` will launch evaluation script once training is finished. Replace `<DATASET_ROOT>` with the location where the dataset in Implicitron format is stored and `<CHECKPOINT_DIR>` with a directory where checkpoints will be dumped during training. Other configuration parameters can be overridden in the same way. See [Configuration system](#configuration-system) section for more information on this. ### Visdom logging Note that the training script logs its progress to Visdom. Make sure to start a visdom server before the training commences: ``` python -m visdom.server ``` > In case a Visdom server is not started, the console will get flooded with `requests.exceptions.ConnectionError` errors signalling that a Visdom server is not available. Note that these errors <b>will NOT interrupt</b> the program and the training will still continue without issues. ## Evaluation To run evaluation on the latest checkpoint after (or during) training, simply add `eval_only=True` to your training command. E.g. for executing the evaluation on the NeRF skateboard sequence, you can run: ```shell dataset_args=data_source_ImplicitronDataSource_args.dataset_map_provider_JsonIndexDatasetMapProvider_args pytorch3d_implicitron_runner --config-path ./configs/ --config-name repro_singleseq_nerf \ $dataset_args.dataset_root=<CO3D_DATASET_ROOT> $dataset_args.category='skateboard' \ $dataset_args.test_restrict_sequence_id=0 exp_dir=<CHECKPOINT_DIR> eval_only=True ``` Evaluation prints the metrics to `stdout` and dumps them to a json file in `exp_dir`. ## Visualisation The script produces a video of renders by a trained model assuming a pre-defined camera trajectory. In order for it to work, `ffmpeg` needs to be installed: ```shell conda install ffmpeg ``` Here is an example of calling the script: ```shell pytorch3d_implicitron_visualizer exp_dir=<CHECKPOINT_DIR> \ visdom_show_preds=True n_eval_cameras=40 render_size="[64,64]" video_size="[256,256]" ``` The argument `n_eval_cameras` sets the number of renderring viewpoints sampled on a trajectory, which defaults to a circular fly-around; `render_size` sets the size of a render passed to the model, which can be resized to `video_size` before writing. Rendered videos of images, masks, and depth maps will be saved to `<CHECKPOINT_DIR>/video`. # Configuration system We use hydra and OmegaConf to parse the configs. The config schema and default values are defined by the dataclasses implementing the modules. More specifically, if a class derives from `Configurable`, i