deep learning framework in C++14.zip

# MNIST Digit Classification [MNIST](http://yann.lecun.com/exdb/mnist/) is a well-known dataset of handwritten digits. We'll use [LeNet-5](http://yann.lecun.com/exdb/lenet/)-like architecture for MNIST digit recognition task. LeNet-5 is proposed by Y. LeCun<a id="cit_LeCun1998">[[LeCun1998]](#LeCun1998)</a>, which is known to work well on handwritten digit recognition. We replace LeNet-5's RBF layer with normal fully-connected layer. ## Constructing Model Let's define the LeNet network. At first, you have to specify loss-function and learning-algorithm. Then, you can add layers from top to bottom by `operator<<`. ```cpp // specify loss-function and learning strategy network<sequential> nn; adagrad optimizer; // connection table, see Table 1 in [LeCun1998] #define O true #define X false static const bool tbl [] = { O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O, O, X, X, X, O, O, O, X, X, O, X, O, O, O, X, O, O, O, X, X, O, O, O, O, X, X, O, X, O, O, X, X, O, O, O, X, X, O, O, O, O, X, O, O, X, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O }; #undef O #undef X // construct nets // // C : convolution // S : sub-sampling // F : fully connected nn << convolutional_layer(32, 32, 5, 1, 6, // C1, 1@32x32-in, 6@28x28-out padding::valid, true, 1, 1, backend_type) << tanh_layer(28, 28, 6) << average_pooling_layer(28, 28, 6, 2) // S2, 6@28x28-in, 6@14x14-out << tanh_layer(14, 14, 6) << convolutional_layer(14, 14, 5, 6, 16, // C3, 6@14x14-in, 16@10x10-in connection_table(tbl, 6, 16), padding::valid, true, 1, 1, backend_type) << tanh_layer(10, 10, 16) << average_pooling_layer(10, 10, 16, 2) // S4, 16@10x10-in, 16@5x5-out << tanh_layer(5, 5, 16) << convolutional_layer(5, 5, 5, 16, 120, // C5, 16@5x5-in, 120@1x1-out padding::valid, true, 1, 1, backend_type) << tanh_layer(1, 1, 120) << fully_connected_layer(120, 10, true, // F6, 120-in, 10-out backend_type) << tanh_layer(10); ``` What does ```tbl``` mean? LeNet has "sparsity" between S2 and C3 layer. Specifically, each feature map in C3 is connected to a subset of S2's feature maps so that each of the feature maps gets different set of inputs (and hopefully they become complementary feature extractors). Tiny-dnn supports this sparsity by ```connection_table``` structure which parameters of constructor are ```bool``` table and number of in/out feature maps. ## Loading Dataset Tiny-dnn supports MNIST idx format, so all you have to do is calling `parse_mnist_images` and `parse_mnist_labels` functions. ```cpp // load MNIST dataset std::vector<label_t> train_labels, test_labels; std::vector<vec_t> train_images, test_images; parse_mnist_labels(data_dir_path + "/train-labels.idx1-ubyte", &train_labels); parse_mnist_images(data_dir_path + "/train-images.idx3-ubyte", &train_images, -1.0, 1.0, 2, 2); parse_mnist_labels(data_dir_path + "/t10k-labels.idx1-ubyte", &test_labels); parse_mnist_images(data_dir_path + "/t10k-images.idx3-ubyte", &test_images, -1.0, 1.0, 2, 2); ``` >Note: >Original MNIST images are 28x28 centered, [0,255]value. >This code rescale values [0,255] to [-1.0,1.0], and add 2px borders (so each image is 32x32). If you want to use another format for learning nets, see [Data Format](https://github.com/tiny-dnn/tiny-dnn/wiki/Data-Format) page. ## Defining Callback It's convenient if we can check recognition rate on test data, training time, and progress for each epoch while training. Tiny-dnn has callback mechanism for this purpose. We can use local variables (network, test-data, etc) in callback by using C++11's lambda. ```cpp progress_display disp(static_cast<unsigned long>(train_images.size())); timer t; int minibatch_size = 10; int num_epochs = 30; optimizer.alpha *= static_cast<tiny_dnn::float_t>(std::sqrt(minibatch_size)); // create callback auto on_enumerate_epoch = [&](){ std::cout << t.elapsed() << "s elapsed." << std::endl; tiny_dnn::result res = nn.test(test_images, test_labels); std::cout << res.num_success << "/" << res.num_total << std::endl; disp.restart(static_cast<unsigned long>(train_images.size())); t.restart(); }; auto on_enumerate_minibatch = [&](){ disp += minibatch_size; }; ``` ## Saving/Loading models Just use ```network::save(filename)``` and ```network::load(filename)``` to write your whole model as binary file. ```cpp nn.save("LeNet-model"); nn.load("LeNet-model"); ``` ## Putting it all together train.cpp ```cpp #include <iostream> #include "tiny_dnn/tiny_dnn.h" using namespace tiny_dnn; using namespace tiny_dnn::activation; static void construct_net(network<sequential>& nn) { // connection table, see Table 1 in [LeCun1998] #define O true #define X false static const bool tbl[] = { O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O, O, X, X, X, O, O, O, X, X, O, X, O, O, O, X, O, O, O, X, X, O, O, O, O, X, X, O, X, O, O, X, X, O, O, O, X, X, O, O, O, O, X, O, O, X, O, X, X, X, O, O, O, X, X, O, O, O, O, X, O, O, O }; #undef O #undef X // by default will use backend_t::tiny_dnn unless you compiled // with -DUSE_AVX=ON and your device supports AVX intrinsics core::backend_t backend_type = core::default_engine(); // construct nets // // C : convolution // S : sub-sampling // F : fully connected nn << convolutional_layer(32, 32, 5, 1, 6, // C1, 1@32x32-in, 6@28x28-out padding::valid, true, 1, 1, backend_type) << tanh_layer(28, 28, 6) << average_pooling_layer(28, 28, 6, 2) // S2, 6@28x28-in, 6@14x14-out << tanh_layer(14, 14, 6) << convolutional_layer(14, 14, 5, 6, 16, // C3, 6@14x14-in, 16@10x10-out connection_table(tbl, 6, 16), padding::valid, true, 1, 1, backend_type) << tanh_layer(10, 10, 16) << average_pooling_layer(10, 10, 16, 2) // S4, 16@10x10-in, 16@5x5-out << tanh_layer(5, 5, 16) << convolutional_layer(5, 5, 5, 16, 120, // C5, 16@5x5-in, 120@1x1-out padding::valid, true, 1, 1, backend_type) << tanh_layer(1, 1, 120) << fully_connected_layer(120, 10, true, // F6, 120-in, 10-out backend_type) << tanh_layer(10); } static void train_lenet(const std::string& data_dir_path) { // specify loss-function and learning strategy network<sequential> nn; adagrad optimizer; construct_net(nn); std::cout << "load models..." << std::endl; // load MNIST dataset std::vector<label_t> train_labels, test_labels; std::vector<vec_t> train_images, test_images; parse_mnist_labels(data_dir_path + "/train-labels.idx1-ubyte", &train_labels); parse_mnist_images(data_dir_path + "/train-images.idx3-ubyte", &train_images, -1.0, 1.0, 2, 2); parse_mnist_labels(data_dir_path + "/t10k-labels.idx1-ubyte", &test_labels); parse_mnist_images(data_dir_path + "/t10k-images.idx3-ubyte", &test_images, -1.0, 1.0, 2, 2); std::cout << "start training" << std::endl; progress_display disp(static_cast<unsigned long>(train_images.size())); timer t; int minibatch_size = 10; int num_epochs = 30; optimizer.alpha *= static_cast<tiny_dnn::float_t>(std::sqrt(minibatch_size)); // create callback auto on_enumerate_epoch = [&](){ std::cout << t.elapsed() << "s elapsed." << std::endl; tiny_dnn::result res =