Discover the practical aspects of implementing deep-learning solutions using the rich Python ecosystem. This book bridges the gap between the academic state-of-the-art and the industry state-of-the-practice by introducing you to deep learning frameworks such as Keras, Theano, and Caffe. The practicalities of these frameworks is often acquired by practitioners by reading source code, manuals, and posting questions on community forums, which tends to be a slow and a painful process. Deep Learning with Python allows you to ramp up to such practical know-how in a short period of time and focus more on the domain, models, and algorithms., This book briefly covers the mathematical prerequisites and fundamentals of deep learning, making this book a good starting point for software developers who want to get started in deep learning. A brief survey of deep learning architectures is also included., Deep Learning with Python also introduces you to key concepts of automatic differentiation and GPU computation which, while not central to deep learning, are critical when it comes to conducting large scale experiments., What You Will Learn, Leverage deep learning frameworks in Python namely, Keras, Theano, and CaffeGain the fundamentals of deep learning with mathematical prerequisitesDiscover the practical considerations of large scale experimentsTake deep learning models to production, Who This Book Is ForSoftware developers who want to try out deep learning as a practical solution to a particular problem.Software developers in a data science team who want to take deep learning models developed by data scientists to production.
Deep Learning with Python: A Hands-on Introduction Nikhil Ketkar Bangalore, Karnataka, India ISBN-13(pbk):978-1-4842-2765-7 ISBN-13( electronic:978-1-4842-2766-4 DOI10.1007/978-1-4842-2766-4 Library of Congress Control Number: 2017939734 Copyright o 2017 by Nikhil Ketkar This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark The use in this publication of trade names, trademarks service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights While the advice and information in this book are believed to be true and accurate at the date of publication neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein Managing Director: Welmoed Spahr Editorial director. Todd green Acquisitions Editor: Celestin Suresh John Development Editor: Matthew Moodie and Anila vincent Technical Reviewer: Jojo moolayail Coordinating Editor: Prachi Mehta Copy Editor Larissa Shmailo Compositor: SPi Global Indexer: SPi Global Artist SPi global Cover image designed by Freepik Distributed to the book trade worldwide by Springer Science+ Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax(201)348-4505, e-mai orders-ny@springer-sbm.com,orvisitwww.springeronline.comApressMedia,LlcisaCaliforniaLlc and the sole member (owner)is Springer Science Business Media Finance Inc(SSBM Finance Inc) SSBM Finance Inc is a Delaware corporation Forinformationontranslationspleasee-mailrights@apress.com,orvisithttp://www.apress.com/ rights-permissions press titles may be purchased in bulk for academic, corporate, or promotional use. e Book versions and licenses are also available for most titles. For more information reference our print and ebook bulk sales webpageathttp://www.apress.com/bulk-sales Any source code or other supplementary material referenced by the author in this book is available to readersongIthubviathebooksproductpagelocatedatwww.apress.com/9781484227657.foRmore detailedinformationpleasevisithttp://www.apress.com/source-code Printed on acid-free paper To Aditi Contents at a glance About the author mmm xi About the technical reviewer ■■■■■■■■■■■■口■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Acknowledgments ■■■■■■■■■■ Chapter 1: Introduction to Deep Learning m mm mamiAi 1 Chapter 2: Machine Learning Fundamentals ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 5 Chapter 3: Feed Forward Neural Networks a15 Chapter 4: Introduction to Theano mmammmmmmammammmmmma 33 Chapter 5: Convolutional Neural Networks mmmmmmm. 61 Chapter 6: Recurrent Neural Networks ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■口■■■■■■■■■■■■■■■■■■■■■■■口■■ Chapter 7: Introduction to Keras ■I 95 Chapter 8: Stochastic Gradient Descentaaan ua111 Chapter 9: Automatic Differentiation mmm 131 Chapter 10: Introduction to GPUs g147 Index 157 Contents About the author mmm xi About the technical reviewer ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ Acknowledgments ■■■■■■■ Chapter 1: Introduction to Deep Learning m mm mamiAi 1 Historical Context Advances in Related Fields Prerequisites 334 Overview of Subsequent Chapters Installing the required Libraries Chapter 2: Machine Learning Fundamentals ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■口■■■■■■■■■■■■■ Intuition Binary classification 445556 Regression Generalization Regularization .. Summary 14 Chapter 3: Feed Forward Neural Networks aaa. 15 Unit 15 Overall structure of a neural network Expressing the Neural Network in Vector Form………………………………18 Evaluating the output of the Neural Network Training the Neural Network 21 CONTENTS Deriving cost Functions using Maximum Likelihood 22 Binary cross entropy .. 23 Cross Entropy 23 Squared error 24 Summary of Loss Functions 25 Types of Units/Activation Functions/Layers 25 Linear unit 26 Sigmoid Unit Softmax Layer….., Rectified Linear Unit(relu)......................27 Hyperbolic Tangent 28 Neural Network Hands-on with Autograd 31 Summary 31 Chapter 4: Introduction to Theano.. n33 What is theano …33 Theano hands-On 34 Summary 59 Chapter 5: Convolutional Neural Networks n61 Convolution Operation 61 Pooling Operation 68 Convolution-Detector-Pooling Building block 0 Convolution variants 74 Intuition behind cnns Summary… 56 Chapter 6: Recurrent Neural Networks rnN Basics Training RNNs 82 Bidirecti0 nal rnns,…89 CONTENT Gradient Explosion and vanishing 90 Gradient Clipping DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD 91 Long Short Term Memory.…… 93 Summary 94 Chapter 7: introduction to Keras 95 Summary 109 Chapter 8: Stochastic Gradient Descent ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 111 Optimization problems Method of steepest Descent DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDm 112 Batch, Stochastic(Single and Mini-batch)Descent DDDDDDDDDDDDDDDDDDDDDDDDDDDDDm 113 Batch∴… 114 Stochastic Single Example 144 Stochastic mini- batch.……1144 Batch vs. stochastic.ww114 Challenges with SGD 114 ocal minima 114 Saddle points… 日面口 日面日日自日面日自日 115 Selecting the learning rate .........................116 Slow Progress in Narrow valleys....... 117 Algorithmic variations on SGD 117 Momentun,… 118 Nesterov accelerated gradient(NAS)…… 119 Annealing and Learning Rate Schedules............................119 Adagrad…...,….….,….….,…,….,…,…,…,19 RMSProp….,,120 Adadelta.……121 Adam… 121 Resilient backpropagation 1日重日面日自日日面日日日日面日m日日自日日日面日自日面日自日日面日日自重日日日面日日自日日日面日自日日自日日 121 Equilibrated SGD 122 IX CONTENTS Tricks and Tips for using SGD.. 122 Preprocessing Input Data 122 Choice of activation function 122 Preprocessing Target value . 123 Initializing Parameters 123 Shuffling Data.…....… 123 Batch normalization 123 Early Stopping…..............……,123 Gradient Noise 123 Parallel and distributed sgd,mm., 124 Hogwild...............124 Downpour……..............……124 ands-0 n sgd with downⅢ…125 Summar 130 Chapter 9: Automatic Differentiation. mmmmaamammmamaammn 131 Numerical Differentiation Symbolic Differentiation ,132 Automatic Differentiation fundamentals 133 Forward/Tangent Linear Mode 134 Reverse/Cotangent/Adjoint Linear Mode Implementation of Automatic Differentiation 141 Hands-on Automatic Differentiation with autograd 143 Summary Chapter 10: Introduction to GPUs n147 Summary …156 Index mmmm 157 About the author Nikhil Ketkar currently leads the machine learning platform team at Flipkart, Indias largest e-commerce company He received his PhD from Washington State University. Following that, he conducted postdoctoral research at University of north Carolina at Charotte which was followed by a brief stint in high frequency trading at Trans Market in Chicago. More ecently, he led the data mining team in Guavus, a startup doing big data analytics in the telecom domain and Indix, a startup doing data science in the e-commerce domain. His research interests include machine learning and graph theory.