# Interactive Classification for Deep Learning Interpretation
We have designed and developed an interactive system that allows users to experiment with deep learning image classifiers and explore their robustness and sensitivity.
Selected areas of an image can be removed in real time with classical computer vision inpainting algorithms, allowing users to ask a variety of "what if" questions by experimentally modifying images and seeing how the deep learning model reacts.
The system also computes class activation maps for any selected class, which highlight the important semantic regions of an image the model uses for classification.
The system runs fully in browser using Tensorflow.js, React, and SqueezeNet.
An advanced inpainting version is also available using a server running the PatchMatch algorithm from the [GIMP Resynthesizer plugin](https://github.com/bootchk/resynthesizer).
[![YouTube video demo](images/thumbnail.png)](https://youtu.be/llub5GcOF6w)
This is the code repository for the accepted [CVPR 2018][cvpr18] Demo: **Interactive Classification for Deep Learning Interpretation**.
Visit our research group homepage [Polo Club of Data Science][poloclub] at [Georgia Tech](http://www.gatech.edu) for more related research!
## Example Scenario: Interpreting "Failed" Classification
The modified image (left), originally classified as *dock* is misclassified as *ocean liner* when the masts of a couple boats are removed from the original image (right).
The top five classification scores are tabulated underneath each image.
![Failed classification](images/failure.png)
## Installation
Download or clone this repository:
```bash
git clone https://github.com/poloclub/interactive-classification.git
```
Within the cloned repo, install the required packages with yarn:
```bash
yarn
```
## Usage
To run, type:
```bash
yarn start
```
## Advanced Inpainting
The following steps are needed to set up PatchMatch inpainting, which currently only works on Linux:
1. Clone the [Resynthesizer](https://github.com/bootchk/resynthesizer) repository and follow the instructions for building the project (stop after running `make`)
2. Find the `libresynthesizer.a` shared library in the generated `lib` folder and copy it to the `inpaint` folder in this repository
3. Run `gcc resynth.c -L. -lresynthesizer -lm -lglib-2.0 -o prog` (may have to install glib2.0 first) to generate the `prog` executable
4. You can now run `python3 inpaint_server.py` and PatchMatch will be used as the inpainting algorithm when running the React application with `yarn start`.
## Citation
**[Interactive Classification for Deep Learning Interpretation][site]**
[Angel Cabrera][angel], [Fred Hohman][fred], [Jason Lin][jason], [Duen Horng (Polo) Chau][polo]
*Demo, Conference on Computer Vision and Pattern Recognition (CVPR). June 18, 2018. Salt Lake City, USA.*
```
@article{cabrera2018interactive,
title={Interactive Classification for Deep Learning Interpretation},
author={Cabrera, Angel and Hohman, Fred and Lin, Jason and Chau, Duen Horng},
journal={Demo, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
year={2018},
organization={IEEE}
}
```
## Researchers
| Name | Affiliation |
|-----------------------|---------------------------------|
| [Angel Cabrera][angel] | Georgia Tech |
| [Fred Hohman][fred] | Georgia Tech |
| [Jason Lin][jason] | Georgia Tech |
| [Duen Horng (Polo) Chau][polo] | Georgia Tech |
## License
MIT License. See [`LICENSE.md`](LICENSE.md).
## Contact
For questions or support [open an issue][issues].
[site]: http://fredhohman.com/papers/interactive-classification
[paper]: http://fredhohman.com/papers/18-interactive-cvpr.pdf
[video]: https://youtu.be/llub5GcOF6w "Video"
[fred]: http://fredhohman.com "Fred Hohman"
[angel]: http://cabreraalex.com/ "Alex Cabrera"
[jason]: http://jlin.xyz/ "Jason Lin"
[polo]: http://www.cc.gatech.edu/~dchau/ "Polo Chau"
[issues]: https://github.com/poloclub/interactive-classification/issues "Issues"
[cvpr18]: http://cvpr2018.thecvf.com
[poloclub]: https://poloclub.github.io
没有合适的资源?快使用搜索试试~ 我知道了~
interactive-classification:深度学习解释的交互式分类
共59个文件
jpg:18个
js:15个
json:5个
需积分: 12 1 下载量 180 浏览量
2021-07-23
19:30:08
上传
评论
收藏 3.24MB ZIP 举报
温馨提示
深度学习解释的交互式分类 我们设计并开发了一个交互式系统,允许用户试验深度学习图像分类器并探索它们的鲁棒性和敏感性。 可以使用经典的计算机视觉修复算法实时删除图像的选定区域,允许用户通过实验修改图像并查看深度学习模型的React来提出各种“假设”问题。 该系统还为任何选定的类计算类激活图,突出显示模型用于分类的图像的重要语义区域。 该系统使用 Tensorflow.js、React 和 SqueezeNet 在浏览器中完全运行。 使用运行 Resynthesizer的 PatchMatch 算法的服务器也可以使用高级修复版本。 这是已接受的 Demo: Interactive Classification for Deep Learning Interpretation的代码库。 访问我们的研究小组主页了解更多相关研究! 示例场景:解释“失败”分类 当几艘船的桅杆从原始图像(右)中移除时
资源详情
资源评论
资源推荐
收起资源包目录
interactive-classification-master.zip (59个子文件)
interactive-classification-master
.gitignore 305B
README.md 4KB
inpaint
resynth.c 5KB
prog 112KB
inpaint_server.py 1KB
libresynthesizer.a 171KB
inpaint.py 2KB
public
manifest.json 241B
hockey2.jpg 25KB
lighthouse.jpg 23KB
skiing.jpg 23KB
sailboat.jpg 31KB
bank.jpg 21KB
index.html 462B
hockey.jpg 41KB
soccer.jpg 26KB
beach.jpg 26KB
baseball.jpg 34KB
.yarnclean 469B
package.json 687B
src
Original.js 3KB
index.css 62B
util.js 5KB
registerServiceWorker.js 4KB
inpaint.js 5KB
Options.js 4KB
App.css 2KB
index.js 254B
heapqueue.js 3KB
App.js 3KB
.Options.js.swp 20KB
Modified.js 8KB
squeezenet
squeezenet.js 10KB
util.js 642B
index.js 167B
imagenet_classes.js 33KB
build
manifest.json 241B
hockey2.jpg 25KB
static
css
main.d35cc17f.css.map 4KB
main.d35cc17f.css 2KB
js
main.309e3888.js 1.14MB
main.309e3888.js.map 6.91MB
lighthouse.jpg 23KB
skiing.jpg 23KB
sailboat.jpg 31KB
asset-manifest.json 196B
bank.jpg 21KB
service-worker.js 3KB
index.html 600B
hockey.jpg 41KB
soccer.jpg 26KB
beach.jpg 26KB
baseball.jpg 34KB
LICENSE.md 1KB
images
thumbnail.png 101KB
success.png 514KB
failure.png 399KB
yarn.lock 240KB
squeezenet
package-lock.json 817B
共 59 条
- 1
火锅与理想
- 粉丝: 32
- 资源: 4569
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
评论0