# Motion Fused Frames (MFFs)
Pytorch implementation of the article [Motion fused frames: Data level fusion strategy for hand gesture recognition](http://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w41/Kopuklu_Motion_Fused_Frames_CVPR_2018_paper.pdf)
```diff
- Update: Code is updated for Pytorch 1.5.0 and CUDA 10.2
```
<p align="center"><img src="./images/motion_fused_frames.jpg" align="middle" width="500" title="Motion Fused Frames" /></p>
### Installation
* Clone the repo with the following command:
```bash
git clone https://github.com/okankop/MFF-pytorch.git
```
* Setup in virtual environment and install the requirements:
```bash
conda create -n MFF python=3.7.4
pip install -r requirements.txt
```
### Dataset Preparation
Download the [jester dataset](https://www.twentybn.com/datasets/something-something) or [NVIDIA dynamic hand gestures dataset](http://research.nvidia.com/publication/online-detection-and-classification-dynamic-hand-gestures-recurrent-3d-convolutional) or [ChaLearn LAP IsoGD dataset](http://www.cbsr.ia.ac.cn/users/jwan/database/isogd.html).
Decompress them into the same folder and use [process_dataset.py](dataset_process/process_dataset.py) to generate the index files for train, val, and test split. Poperly set up the train, validatin, and category meta files in [datasets_video.py](gesture_recognition/datasets_video.py). Finally, use directory [flow_computation](https://github.com/okankop/flow_computation) to calculate the optical flow images using Brox method.
将数据集下载到一个文件夹后,数据集处理分三步:
1. 用process_dataset.py生成训练、验证和测试的索引文件
2. 用flow_computation计算光流
Assume the structure of data directories is the following (处理好的文件目录):
```misc
~/MFF-pytorch/
datasets/
jester/
rgb/
.../ (directories of video samples)
.../ (jpg color frames)
flow/
u/
.../ (directories of video samples)
.../ (jpg optical-flow-u frames)
v/
.../ (directories of video samples)
.../ (jpg optical-flow-v frames)
model/
.../(saved models for the last checkpoint and best model)
```
### Running the Code
Followings are some examples for training under different scenarios:
* Train 4-segment network with 3 flow, 1 color frames (4-MFFs-3f1c architecture)
(重新训练)
```bash
python main_old.py jester RGBFlow --arch BNInception --num_segments 4 --consensus_type MLP --num_motion 3 --batch-size 32
```
* Train resuming the last checkpoint (4-MFFs-3f1c architecture)
(从检查点【之前训练结果】恢复训练)
```bash
python main_old.py jester RGBFlow --resume=<path-to-last-checkpoint> --arch BNInception --consensus_type MLP --num_segments 4 --num_motion 3 --batch-size 32
```
* The command to test trained models (4-MFFs-3f1c architecture). Pretrained models are under [pretrained_models](pretrained_models).
(测试模型)
```bash
python test_models.py jester RGBFlow pretrained_models/MFF_jester_RGBFlow_BNInception_segment4_3f1c_best.pth.tar --arch BNInception --consensus_type MLP --test_crops 1 --num_motion 3 --test_segments 4
```
All GPUs are used for the training. If you want a part of GPUs, use CUDA_VISIBLE_DEVICES=...
### Citation
If you use this code or pre-trained models, please cite the following:
```bibtex
@InProceedings{Kopuklu_2018_CVPR_Workshops,
author = {Kopuklu, Okan and Kose, Neslihan and Rigoll, Gerhard},
title = {Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2018}
}
```
### Acknowledgement
This project is built on top of the codebase [TSN-pytorch](https://github.com/yjxiong/temporal-segment-networks). We thank Yuanjun Xiong for releasing [TSN-Pytorch codebase](https://github.com/yjxiong/temporal-segment-networks), which we build our work on top. We also thank Bolei Zhou for the insprational work [Temporal Segment Networks](https://arxiv.org/pdf/1711.08496.pdf), from which we imported [process_dataset.py](https://github.com/metalbubble/TRN-pytorch/blob/master/process_dataset.py) to our project.
没有合适的资源?快使用搜索试试~ 我知道了~
基于手势识别的人机交互系统。
共50个文件
py:39个
md:4个
jpg:2个
需积分: 5 0 下载量 141 浏览量
2024-05-09
22:44:00
上传
评论
收藏 427KB ZIP 举报
温馨提示
基于手势识别的人机交互系统。通过MFF运动融合帧深度学习方法与传统视觉手势识别算法相结合,实现较为准确的手势识别效果,进而实现非接触式的体感交互。.zip
资源推荐
资源详情
资源评论
收起资源包目录
基于手势识别的人机交互系统。通过MFF运动融合帧深度学习方法与传统视觉手势识别算法相结合,实现较为准确的手势识别效果,进而实现非接触式的体感交互。.zip (50个子文件)
content
show_prepare
__init__.py 0B
show_prepare.py 7KB
test.py 2KB
communication
__init__.py 81B
send.py 2KB
receive.py 3KB
dataset_process
deal_video_no_cut.py 5KB
__init__.py 0B
process_dataset.py 3KB
check_datasets.py 2KB
get_label.py 416B
cutVideo.py 7KB
deal_video_tem.py 1KB
deal_video_adjust.py 3KB
copyVideo.py 2KB
deal_video.py 3KB
choose_train_validation.py 972B
LICENSE 3KB
gesture_recognition
__init__.py 0B
lib
flow_computation.dll 119KB
old_interface
__init__.py 0B
test_models.py 8KB
main_old.py 13KB
MLPmodule.py 1KB
run_deal_video.py 7KB
main.py 13KB
datasets_video.py 3KB
dataset.py 7KB
gesture_system.py 18KB
models.py 18KB
experiment
test_area_catch_click_swipe.md 691B
test_dis_two_hands.md 551B
test_move_speed.md 933B
transforms.py 14KB
ops
utils.py 829B
__init__.py 48B
basic_ops.py 1KB
opts.py 4KB
gesture_location_system.py 30KB
.gitmodules 107B
show_UI
imageview_control.py 992B
__init__.py 0B
state_show.py 1KB
main_control.py 4KB
ppt_control.py 1KB
requirements.txt 298B
.gitignore 204B
images
motion_fused_frames.jpg 223KB
network_arch.jpg 130KB
README.md 4KB
共 50 条
- 1
资源评论
生瓜蛋子
- 粉丝: 3815
- 资源: 4660
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
- 实验二:IP协议分析.zip
- 驱动代码驱动代码驱动代码驱动代码
- SVID_20240523_141155_1.mp4
- Code for the complete guide to tkinter tutorial
- 关于百货中心供应链管理系统.zip
- SimpleFolderIcon-master 修改Unity的Project下的文件夹图标
- A python Tkinter widget to display tile based maps
- A pure Python library for adding tables to a Tkinter application
- Vector资源文件.zip
- MobaXterm-Installer
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功