# LoFTR: Detector-Free Local Feature Matching with Transformers
### [Project Page](https://zju3dv.github.io/loftr) | [Paper](https://arxiv.org/pdf/2104.00680.pdf)
<br/>
> LoFTR: Detector-Free Local Feature Matching with Transformers
> [Jiaming Sun](https://jiamingsun.ml)<sup>\*</sup>, [Zehong Shen](https://zehongs.github.io/)<sup>\*</sup>, [Yu'ang Wang](https://github.com/angshine)<sup>\*</sup>, [Hujun Bao](http://www.cad.zju.edu.cn/home/bao/), [Xiaowei Zhou](http://www.cad.zju.edu.cn/home/xzhou/)
> CVPR 2021
![demo_vid](assets/loftr-github-demo.gif)
## TODO List and ETA
- [x] Inference code and pretrained models (DS and OT) (2021-4-7)
- [x] Code for reproducing the test-set results (2021-4-7)
- [x] Webcam demo to reproduce the result shown in the GIF above (2021-4-13)
- [x] Training code and training data preparation (expected 2021-6-10)
Discussions about the paper are welcomed in the [discussion panel](https://github.com/zju3dv/LoFTR/discussions).
:triangular_flag_on_post: **Updates**
- Check out [QuadTreeAttention](https://github.com/Tangshitao/QuadTreeAttention), a new attention machanism that improves the efficiency and performance of LoFTR with less demanding GPU requirements for training.
- :white_check_mark: Integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/Kornia-LoFTR)
## Colab demo
Want to run LoFTR with custom image pairs without configuring your own GPU environment? Try the Colab demo:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1BgNIOjFHauFoNB95LGesHBIjioX74USW?usp=sharing)
## Installation
```shell
# For full pytorch-lightning trainer features (recommended)
conda env create -f environment.yaml
conda activate loftr
# For the LoFTR matcher only
pip install torch einops yacs kornia
```
We provide the [download link](https://drive.google.com/drive/folders/1DOcOPZb3-5cWxLqn256AhwUVjBPifhuf?usp=sharing) to
- the scannet-1500-testset (~1GB).
- the megadepth-1500-testset (~600MB).
- 4 pretrained models of indoor-ds, indoor-ot, outdoor-ds and outdoor-ot (each ~45MB).
By now, the environment is all set and the LoFTR-DS model is ready to go!
If you want to run LoFTR-OT, some extra steps are needed:
<details>
<summary>[Requirements for LoFTR-OT]</summary>
We use the code from [SuperGluePretrainedNetwork](https://github.com/magicleap/SuperGluePretrainedNetwork) for optimal transport. However, we can't provide the code directly due its strict LICENSE requirements. We recommend downloading it with the following command instead.
```shell
cd src/loftr/utils
wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/superglue.py
```
</details>
## Run LoFTR demos
### Match image pairs with LoFTR
<details>
<summary>[code snippets]</summary>
```python
from src.loftr import LoFTR, default_cfg
# Initialize LoFTR
matcher = LoFTR(config=default_cfg)
matcher.load_state_dict(torch.load("weights/indoor_ds.ckpt")['state_dict'])
matcher = matcher.eval().cuda()
# Inference
with torch.no_grad():
matcher(batch) # batch = {'image0': img0, 'image1': img1}
mkpts0 = batch['mkpts0_f'].cpu().numpy()
mkpts1 = batch['mkpts1_f'].cpu().numpy()
```
</details>
An example is given in `notebooks/demo_single_pair.ipynb`.
### Online demo
Run the online demo with a webcam or video to reproduce the result shown in the GIF above.
```bash
cd demo
./run_demo.sh
```
<details>
<summary>[run_demo.sh]</summary>
```bash
#!/bin/bash
set -e
# set -x
if [ ! -f utils.py ]; then
echo "Downloading utils.py from the SuperGlue repo."
echo "We cannot provide this file directly due to its strict licence."
wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/utils.py
fi
# Use webcam 0 as input source.
input=0
# or use a pre-recorded video given the path.
# input=/home/sunjiaming/Downloads/scannet_test/$scene_name.mp4
# Toggle indoor/outdoor model here.
model_ckpt=../weights/indoor_ds.ckpt
# model_ckpt=../weights/outdoor_ds.ckpt
# Optionally assign the GPU ID.
# export CUDA_VISIBLE_DEVICES=0
echo "Running LoFTR demo.."
eval "$(conda shell.bash hook)"
conda activate loftr
python demo_loftr.py --weight $model_ckpt --input $input
# To save the input video and output match visualizations.
# python demo_loftr.py --weight $model_ckpt --input $input --save_video --save_input
# Running on remote GPU servers with no GUI.
# Save images first.
# python demo_loftr.py --weight $model_ckpt --input $input --no_display --output_dir="./demo_images/"
# Then convert them to a video.
# ffmpeg -framerate 15 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4
```
</details>
### Reproduce the testing results with pytorch-lightning
You need to setup the testing subsets of ScanNet and MegaDepth first. We create symlinks from the previously downloaded datasets to `data/{{dataset}}/test`.
```shell
# set up symlinks
ln -s /path/to/scannet-1500-testset/* /path/to/LoFTR/data/scannet/test
ln -s /path/to/megadepth-1500-testset/* /path/to/LoFTR/data/megadepth/test
```
```shell
conda activate loftr
# with shell script
bash ./scripts/reproduce_test/indoor_ds.sh
# or
python test.py configs/data/scannet_test_1500.py configs/loftr/loftr_ds.py --ckpt_path weights/indoor_ds.ckpt --profiler_name inference --gpus=1 --accelerator="ddp"
```
For visualizing the results, please refer to `notebooks/visualize_dump_results.ipynb`.
<br/>
<!-- ### Image pair info for training on ScanNet
You can download the data at [here](https://drive.google.com/file/d/1fC2BezUSsSQy7_H65A0ZfrYK0RB3TXXj/view?usp=sharing).
<details>
<summary>[data format]</summary>
```python
In [14]: npz_path = './cfg_1513_-1_0.2_0.8_0.15/scene_data/train/scene0000_01.npz'
In [15]: data = np.load(npz_path)
In [16]: data['name']
Out[16]:
array([[ 0, 1, 276, 567],
[ 0, 1, 1147, 1170],
[ 0, 1, 541, 5757],
...,
[ 0, 1, 5366, 5393],
[ 0, 1, 2607, 5278],
[ 0, 1, 736, 5844]], dtype=uint16)
In [17]: data['score']
Out[17]: array([0.2903, 0.7715, 0.5986, ..., 0.7227, 0.5527, 0.4148], dtype=float16)
In [18]: len(data['name'])
Out[18]: 1684276
In [19]: len(data['score'])
Out[19]: 1684276
```
`data['name']` is the image pair info, organized as [`scene_id`, `seq_id`, `image0_id`, `image1_id`].
`data['score']` is the overlapping score defined in [SuperGlue](https://arxiv.org/pdf/1911.11763) (Page 12).
</details> -->
## Training
See [Training LoFTR](./docs/TRAINING.md) for more details.
## Citation
If you find this code useful for your research, please use the following BibTeX entry.
```bibtex
@article{sun2021loftr,
title={{LoFTR}: Detector-Free Local Feature Matching with Transformers},
author={Sun, Jiaming and Shen, Zehong and Wang, Yuang and Bao, Hujun and Zhou, Xiaowei},
journal={{CVPR}},
year={2021}
}
```
## Copyright
This work is affiliated with ZJU-SenseTime Joint Lab of 3D Vision, and its intellectual property belongs to SenseTime Group Ltd.
```
Copyright SenseTime. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```
没有合适的资源?快使用搜索试试~ 我知道了~
资源推荐
资源详情
资源评论
收起资源包目录
基于神经网络6-Dof的3D重建技术.zip (213个子文件)
format.cc 3KB
FeatureManager.cpp 100KB
Bundler.cpp 39KB
Utils.cpp 18KB
Frame.cpp 13KB
pybind_api.cpp 8KB
SIFTImageManager.cpp 5KB
CUDACache.cpp 4KB
LossGPU.cpp 4KB
bindings.cpp 758B
bindings.cpp 717B
SolverBundling.cu 55KB
CUDAImageUtil.cu 44KB
cuda_ransac.cu 43KB
CUDASolverBundling.cu 25KB
gridencoder.cu 20KB
SIFTImageManager.cu 13KB
common.cu 9KB
SBA.cu 7KB
CUDACache.cu 5KB
dockerfile 5KB
driller.gif 5.18MB
problem_setup_c.gif 2.1MB
milk_jug.gif 2.09MB
preview_results_c.gif 1.05MB
.gitignore 407B
.gitignore 356B
.gitmodules 163B
format.h 107KB
format-inl.h 101KB
format.h 101KB
core.h 96KB
cuda_SimpleMatrixUtil.h 51KB
core.h 46KB
pattern_formatter.h 38KB
cutil_math.h 38KB
format-inl.h 32KB
SolverBundlingDenseUtil.h 27KB
printf.h 25KB
color.h 21KB
SolverBundlingEquationsLie.h 16KB
SIFTImageManager.h 15KB
chrono.h 13KB
logger_impl.h 12KB
Utils.h 11KB
spdlog.h 11KB
os.h 10KB
LieDerivUtil.h 9KB
ranges.h 9KB
posix.h 9KB
FeatureManager.h 9KB
registry.h 8KB
Frame.h 7KB
tweakme.h 6KB
thread_pool.h 6KB
logger.h 6KB
common.h 6KB
ansicolor_sink.h 5KB
rotating_file_sink.h 5KB
wincolor_sink.h 5KB
ostream.h 5KB
ICPUtil.h 5KB
CUDAImageUtil.h 5KB
CUDATimer.h 4KB
bin_to_hex.h 4KB
time.h 4KB
daily_file_sink.h 4KB
file_helper.h 4KB
CUDASolverBundling.h 4KB
Bundler.h 3KB
fmt_helper.h 3KB
mpmc_blocking_q.h 3KB
android_sink.h 3KB
async.h 3KB
SBA.h 3KB
async_logger_impl.h 3KB
syslog_sink.h 3KB
common.h 3KB
SolverBundlingState.h 3KB
stdout_sinks.h 3KB
locale.h 3KB
CUDACache.h 2KB
async_logger.h 2KB
dist_sink.h 2KB
cuda_ransac.h 2KB
periodic_worker.h 2KB
base_sink.h 2KB
basic_file_sink.h 2KB
cudaUtil.h 2KB
stdout_color_sinks.h 2KB
circular_q.h 1KB
EigenDenseBaseAddons.h 1KB
CUDACacheUtil.h 1KB
sink.h 1KB
gridencoder.h 1KB
common.h 1KB
ostream_sink.h 1KB
log_msg.h 1KB
console_globals.h 1KB
null_sink.h 1KB
共 213 条
- 1
- 2
- 3
资源评论
小码蚁.
- 粉丝: 2584
- 资源: 4344
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功