Consistent Sparse Deep Learning: Theory and Computation
===============================================================
We propose a frequentist-like method for learning sparse DNNs and justify its consistency under the Bayesian framework. The structure of the sparse DNN can be consistently determined using a Laplace approximation-based marginal posterior inclusion probability approach on a trained Bayesian neural network with mixture of normal prior.
### Related Publication
Yan Sun <sup> * </sup>, Qifan Song <sup> * </sup> and Faming Liang, [Consistent Sparse Deep Learning: Theory and Computation.](https://arxiv.org/pdf/2102.13229.pdf) *JASA, in press*.
### Reproduce Experimental Results in the Paper:
#### Simulation:
Generate Data:
```{python}
python Generate_Data.py
```
Regression:
```{python}
python Simulation_Regression.py --data_index 1 --activation 'tanh'
python Simulation_Regression.py --data_index 1 --activation 'relu'
```
Regression Baseline:
```{python}
python Dropout_Regression.py --data_index 1 --activation 'tanh'
python Dropout_Regression.py --data_index 1 --activation 'relu'
python Spinn_Regression.py --data_index 1 --activation 'tanh'
python Spinn_Regression.py --data_index 1 --activation 'relu'
python DPF_Regression.py --data_index 1 --activation 'tanh'
python DPF_Regression.py --data_index 1 --activation 'relu'
```
Classification
```{python}
python Simulation_Classification.py --data_index 1 --activation 'tanh'
python Simulation_Classification.py --data_index 1 --activation 'relu'
```
Classification Baseline:
```{python}
python Dropout_Classification.py --data_index 1 --activation 'tanh'
python Dropout_Classification.py --data_index 1 --activation 'relu'
python Spinn_Classification.py --data_index 1 --activation 'tanh'
python Spinn_Classification.py --data_index 1 --activation 'relu'
python DPF_Classification.py --data_index 1 --activation 'tanh'
python DPF_Classification.py --data_index 1 --activation 'relu'
```
Structure Selection
```{python}
python Simulation_Structure.py --data_index 1
```
Structure Selection Baseline:
```{python}
python Spinn_structure.py --data_index 1
```
#### Real Data:
CIFAR ResNet Compression
```{python}
python cifar_run.py --model_path 'resnet_32_10percent/' --sigma0 0.00002 --lambdan 0.00001 -depth 32 --seed 1
python cifar_run_vb.py --model_path 'resnet_32_10percent_vb/' --sigma0 0.00002 --lambdan 0.00001 --prune_ratio 0.1 -depth 32 --seed 1
python cifar_run.py --model_path 'resnet_32_5percent/' --sigma0 0.00006 --lambdan 0.0000001 -depth 32 --seed 1
python cifar_run_vb.py --model_path 'resnet_32_5percent_vb/' --sigma0 0.00006 --lambdan 0.0000001 --prune_ratio 0.05 -depth 32 --seed 1
python cifar_run.py --model_path 'resnet_20_10percent/' --sigma0 0.00004 --lambdan 0.000001 -depth 20 --seed 1
python cifar_run_vb.py --model_path 'resnet_20_10percent_vb/' --sigma0 0.00004 --lambdan 0.000001 --prune_ratio 0.1 -depth 20 --seed 1
python cifar_run.py --model_path 'resnet_20_20percent/' --sigma0 0.000006 --lambdan 0.000001 -depth 20 --seed 1
python cifar_run_vb.py --model_path 'resnet_20_20percent_vb/' --sigma0 0.000006 --lambdan 0.000001 --prune_ratio 0.2 -depth 20 --seed 1
```
Consistent-Sparse-Deep-Learning-Theory-and-Computation
需积分: 10 200 浏览量
2021-03-08
07:50:47
上传
评论
收藏 59KB ZIP 举报
Consistent-Sparse-Deep-Learning-Theory-and-Computation-main.zip （22个子文件） 
Consistent-Sparse-Deep-Learning-Theory-and-Computation-main experiments 
DPF_Regression.py 15KB Spinn_Classification.py 20KB Spinn_Regression.py 16KB Simulation_Regression.py 19KB vb_net.py 29KB Spinn_structure.py 13KB run_bnn.py 13KB Simulation_Structure.py 25KB Dropout_Regression.py 6KB Dropout_Classification.py 7KB cifar_run.py 13KB cifar_run_vb.py 13KB resnet_vb.py 9KB DPF_Classification.py 17KB resnet.py 8KB transforms.py 2KB haha 1B Generate_Data.py 14KB Simulation_Classification.py 21KB process_data.py 2KB vb_net.py 29KB README.md 3KB
评论0