<img src="assets/magicleap.png" width="240">
### Research @ Magic Leap (CVPR 2020, Oral)
# SuperGlue Inference and Evaluation Demo Script
## Introduction
SuperGlue is a CVPR 2020 research project done at Magic Leap. The SuperGlue network is a Graph Neural Network combined with an Optimal Matching layer that is trained to perform matching on two sets of sparse image features. This repo includes PyTorch code and pretrained weights for running the SuperGlue matching network on top of [SuperPoint](https://arxiv.org/abs/1712.07629) keypoints and descriptors. Given a pair of images, you can use this repo to extract matching features across the image pair.
<p align="center">
<img src="assets/teaser.png" width="500">
</p>
SuperGlue operates as a "middle-end," performing context aggregation, matching, and filtering in a single end-to-end architecture. For more details, please see:
* Full paper PDF: [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://arxiv.org/abs/1911.11763).
* Authors: *Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich*
* Website: [psarlin.com/superglue](https://psarlin.com/superglue) for videos, slides, recent updates, and more visualizations.
* `hloc`: a new toolbox for visual localization and SfM with SuperGlue, available at [cvg/Hierarchical-Localization](https://github.com/cvg/Hierarchical-Localization/). Winner of 3 CVPR 2020 competitions on localization and image matching!
We provide two pre-trained weights files: an indoor model trained on ScanNet data, and an outdoor model trained on MegaDepth data. Both models are inside the [weights directory](./models/weights). By default, the demo will run the **indoor** model.
## Dependencies
* Python 3 >= 3.5
* PyTorch >= 1.1
* OpenCV >= 3.4 (4.1.2.30 recommended for best GUI keyboard interaction, see this [note](#additional-notes))
* Matplotlib >= 3.1
* NumPy >= 1.18
Simply run the following command: `pip3 install numpy opencv-python torch matplotlib`
## Contents
There are two main top-level scripts in this repo:
1. `demo_superglue.py` : runs a live demo on a webcam, IP camera, image directory or movie file
2. `match_pairs.py`: reads image pairs from files and dumps matches to disk (also runs evaluation if ground truth relative poses are provided)
## Live Matching Demo Script (`demo_superglue.py`)
This demo runs SuperPoint + SuperGlue feature matching on an anchor image and live image. You can update the anchor image by pressing the `n` key. The demo can read image streams from a USB or IP camera, a directory containing images, or a video file. You can pass all of these inputs using the `--input` flag.
### Run the demo on a live webcam
Run the demo on the default USB webcam (ID #0), running on a CUDA GPU if one is found:
```sh
./demo_superglue.py
```
Keyboard control:
* `n`: select the current frame as the anchor
* `e`/`r`: increase/decrease the keypoint confidence threshold
* `d`/`f`: increase/decrease the match filtering threshold
* `k`: toggle the visualization of keypoints
* `q`: quit
Run the demo on 320x240 images running on the CPU:
```sh
./demo_superglue.py --resize 320 240 --force_cpu
```
The `--resize` flag can be used to resize the input image in three ways:
1. `--resize` `width` `height` : will resize to exact `width` x `height` dimensions
2. `--resize` `max_dimension` : will resize largest input image dimension to `max_dimension`
3. `--resize` `-1` : will not resize (i.e. use original image dimensions)
The default will resize images to `640x480`.
### Run the demo on a directory of images
The `--input` flag also accepts a path to a directory. We provide a directory of sample images from a sequence. To run the demo on the directory of images in `freiburg_sequence/` on a headless server (will not display to the screen) and write the output visualization images to `dump_demo_sequence/`:
```sh
./demo_superglue.py --input assets/freiburg_sequence/ --output_dir dump_demo_sequence --resize 320 240 --no_display
```
You should see this output on the sample Freiburg-TUM RGBD sequence:
<img src="assets/freiburg_matches.gif" width="560">
The matches are colored by their predicted confidence in a jet colormap (Red: more confident, Blue: less confident).
### Additional useful command line parameters
* Use `--image_glob` to change the image file extension (default: `*.png`, `*.jpg`, `*.jpeg`).
* Use `--skip` to skip intermediate frames (default: `1`).
* Use `--max_length` to cap the total number of frames processed (default: `1000000`).
* Use `--show_keypoints` to visualize the detected keypoints (default: `False`).
## Run Matching+Evaluation (`match_pairs.py`)
This repo also contains a script `match_pairs.py` that runs the matching from a list of image pairs. With this script, you can:
* Run the matcher on a set of image pairs (no ground truth needed)
* Visualize the keypoints and matches, based on their confidence
* Evaluate and visualize the match correctness, if the ground truth relative poses and intrinsics are provided
* Save the keypoints, matches, and evaluation results for further processing
* Collate evaluation results over many pairs and generate result tables
### Matches only mode
The simplest usage of this script will process the image pairs listed in a given text file and dump the keypoints and matches to compressed numpy `npz` files. We provide the challenging ScanNet pairs from the main paper in `assets/example_indoor_pairs/`. Running the following will run SuperPoint + SuperGlue on each image pair, and dump the results to `dump_match_pairs/`:
```sh
./match_pairs.py
```
The resulting `.npz` files can be read from Python as follows:
```python
>>> import numpy as np
>>> path = 'dump_match_pairs/scene0711_00_frame-001680_scene0711_00_frame-001995_matches.npz'
>>> npz = np.load(path)
>>> npz.files
['keypoints0', 'keypoints1', 'matches', 'match_confidence']
>>> npz['keypoints0'].shape
(382, 2)
>>> npz['keypoints1'].shape
(391, 2)
>>> npz['matches'].shape
(382,)
>>> np.sum(npz['matches']>-1)
115
>>> npz['match_confidence'].shape
(382,)
```
For each keypoint in `keypoints0`, the `matches` array indicates the index of the matching keypoint in `keypoints1`, or `-1` if the keypoint is unmatched.
### Visualization mode
You can add the flag `--viz` to dump image outputs which visualize the matches:
```sh
./match_pairs.py --viz
```
You should see images like this inside of `dump_match_pairs/` (or something very close to it, see this [note](#a-note-on-reproducibility)):
<img src="assets/indoor_matches.png" width="560">
The matches are colored by their predicted confidence in a jet colormap (Red: more confident, Blue: less confident).
### Evaluation mode
You can also estimate the pose using RANSAC + Essential Matrix decomposition and evaluate it if the ground truth relative poses and intrinsics are provided in the input `.txt` files. Each `.txt` file contains three key ground truth matrices: a 3x3 intrinsics matrix of image0: `K0`, a 3x3 intrinsics matrix of image1: `K1` , and a 4x4 matrix of the relative pose extrinsics `T_0to1`.
To run the evaluation on the sample set of images (by default reading `assets/scannet_sample_pairs_with_gt.txt`), you can run:
```sh
./match_pairs.py --eval
```
Since you enabled `--eval`, you should see collated results printed to the terminal. For the example images provided, you should get the following numbers (or something very close to it, see this [note](#a-note-on-reproducibility)):
```txt
Evaluation Results (mean over 15 pairs):
AUC@5 AUC@10 AUC@20 Prec MScore
26.99 48.40 64.47 73.52 19.60
```
The resulting `.npz` files in `dump_match_pairs/` will now contain scalar values related to the evaluation, computed on the sample images provided. Here is what you should find in one of the generated evaluation files:
```python
>>> import numpy as np
>>> path = 'dump_match_pairs/scene0711_00_frame-001680_scene0711_00_frame-001995_evaluation.npz'
>>>
没有合适的资源?快使用搜索试试~ 我知道了~
SuperGlue
共50个文件
py:14个
txt:12个
pyc:11个
需积分: 0 0 下载量 169 浏览量
2024-09-09
16:44:56
上传
评论
收藏 144.34MB ZIP 举报
温馨提示
SuperGlueSuperGlue
资源推荐
资源详情
资源评论
收起资源包目录
SuperGlue.zip (50个子文件)
SuperGlue
demo_superglue.py 10KB
assets
phototourism_test_pairs.txt 186KB
megadepth_train_scenes.txt 765B
video_imgs_pair.txt 0B
teaser.png 122KB
yfcc_test_pairs_with_gt.txt 1.74MB
yfcc_test_pairs_with_gt_original.txt 1.74MB
phototourism_test_pairs_original.txt 186KB
megadepth_validation_scenes.txt 180B
scannet_sample_pairs.txt 915B
indoor_evaluation.png 498KB
indoor_matches.png 500KB
magicleap.png 39KB
scannet_sample_pairs_with_gt.txt 4KB
phototourism_sample_pairs.txt 577B
scannet_test_pairs_with_gt.txt 458KB
LICENSE 7KB
utils
read_results_npz.py 293B
save_imgs.py 1KB
frame_img
pair_create.py 867B
add
count_frame.py 246B
requirements.txt 69B
models
utils.py 18KB
__init__.py 0B
weights
superglue_outdoor.pth 46MB
superpoint_v1.pth 4.96MB
superglue_indoor.pth 46MB
superglue.py 11KB
matching.py 2KB
superpoint.py 7KB
__pycache__
superpoint.cpython-39.pyc 6KB
matching.cpython-38.pyc 2KB
superglue.cpython-38.pyc 10KB
__init__.cpython-39.pyc 166B
utils.cpython-39.pyc 16KB
utils.cpython-38.pyc 16KB
superpoint.cpython-38.pyc 6KB
superglue.cpython-39.pyc 10KB
__init__.cpython-38.pyc 189B
matching.cpython-39.pyc 2KB
deploy
superglue.pt 4.97MB
superpoint.pt 46.09MB
export.py 2KB
keypoints_pytorch.py 8KB
keypoints.py 11KB
.gitignore 23B
pt
keypoints.pt 4.97MB
__pycache__
match_pairs.cpython-39.pyc 6KB
README.md 19KB
match_pairs.py 11KB
共 50 条
- 1
资源评论
Just_a_Way_Plus
- 粉丝: 79
- 资源: 23
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功