OpenCV for Secret Agents

Table of Contents Preface 1 Chapter 1: Preparing for the Mission 7 Setting up a development machine 8 Windows 10 OpenCV on Windows with binary installers 12 OpenCV on Windows with CMake and compilers 12 Mac 15 Mac with MacPorts 16 Mac with Homebrew 18 Debian Wheezy and its derivatives, including Raspbian, Ubuntu, and Linux Mint 20 Fedora and its derivatives, including RHEL and CentOS 22 openSUSE and its derivatives 22 Tegra Android Development Pack 23 Building OpenCV Android sample projects with Eclipse 25 Unity 34 Setting up Raspberry Pi 34 Setting up the Raspberry Pi Camera Module 39 Finding OpenCV documentation, help, and updates 40 Alternatives to Raspberry Pi 41 Summary 42 Chapter 2: Searching for Luxury Accommodations Worldwide 43 Planning the Luxocator app 44 Creating, comparing, and storing histograms 45 Training the classifier with reference images 52 Acquiring images from the Web 53 Acquiring images from Bing image search 55 Preparing images and resources for the app 60 Building Luxocator for distribution 71 Summary 74 Chapter 3: Training a Smart Alarm to Recognize the Villain and His Cat 75 Understanding machine learning in general 77 Planning the Interactive Recognizer app 78 Understanding Haar cascades and LBPH 80 Implementing the Interactive Recognizer app 84 Planning the cat detection model 98 Implementing the training script for the cat detection model 100 Planning the Angora Blue app 114 Implementing the Angora Blue app 115 Building Angora Blue for distribution 122 Further fun with finding felines 122 Summary 122 Chapter 4: Controlling a Phone App with Your Suave Gestures 123 Planning the Goldgesture app 124 Understanding optical flow 126 Setting up the Eclipse Workspace 129 Getting a cascade file and audio files 138 Specifying the app's requirements 138 Laying out a camera preview as the main view 139 Tracking back and forth gestures 140 Playing audio clips as questions and answers 143 Capturing images and tracking faces in an activity 147 Summary 164 Chapter 5: Equipping Your Car with a Rearview Camera and Hazard Detection 165 Planning The Living Headlights app 167 Detecting lights as blobs 169 Estimating distances (a cheap approach) 172 Implementing The Living Headlights app 175 Testing The Living Headlights app at home 189 Testing The Living Headlights app in a car 193 Summary 199 Amplifying Camera 201 Planning the Lazy Eyes app 203 Understanding what Eulerian video magnification can do 205 Extracting repeating signals from video using the Fast Fourier Transform (FFT) 206 Choosing and setting up an FFT library 207 Compositing two images using image pyramids 210 Implementing the Lazy Eyes app 211 Configuring and testing the app for various motions 220 Seeing things in another light 228 Summary 229 Chapter 7: Creating a Physics Simulation Based on a Pen and Paper Sketch 231 Planning the Rollingball app 233 Detecting circles and lines 236 Setting up OpenCV for Unity 239 Configuring and building the Unity project 242 Creating the Rollingball scene in Unity 244 Creating Unity assets and adding them to the scene 247 Writing shaders and creating materials 247 Creating physics materials 250 Creating prefabs 252 Writing our first Unity script 256 Writing the main Rollingball script 258 Tidying up and testing 278 Summary 279 Index 281

OpenCV Essentials

Chapter 1, Getting Started, deals with the basic installation steps and introduces the essential concepts of the OpenCV API. The first examples to read/write images and video and capture them from a camera are also provided. Chapter 2, Something We Look At – Graphical User Interfaces, covers user interface capabilities for our OpenCV-based applications. Chapter 3, First Things First – Image Processing, covers the most useful image processing techniques available in OpenCV. Chapter 4, What's in the Image? Segmentation, tackles the all-important problem of image segmentation in OpenCV. Chapter 5, Focusing on the Interesting 2D Features, covers the functions available for extracting keypoints and descriptors from an image. Chapter 6, Where's Wally? Object Detection, describes that object detection is a central problem in computer vision. This chapter explains the functionality available for object detection. Chapter 7, What Is He Doing? Motion, considers more than just a single static image. This chapter deals with motion and tracking in OpenCV. Chapter 8, Advanced Topics, focuses on some advanced topics such as machine learning and GPU-based acceleration.

Instant OpenCV for IOS

Table of Contents Preface 1 Instant OpenCV for iOS 5 Getting started with iOS (Simple) 5 Displaying an image from resources (Simple) 10 Linking OpenCV to an iOS project (Simple) 15 Detecting faces with Cascade Classifier (Intermediate) 21 Printing a postcard (Intermediate) 24 Working with images in Gallery (Intermediate) 30 Applying a retro effect (Intermediate) 37 Taking photos from camera (Intermediate) 41 Creating a static library (Intermediate) 46 Capturing a video from camera (Simple) 50 Control advanced camera settings (Advanced) 55 Applying effects to live video (Intermediate) 60 Saving video from camera (Simple) 63 Optimizing performance with ARM NEON (Advanced) 65 Detecting facial features (Advanced) 72 Using the Accelerate framework (Advanced) 77 Building OpenCV for iOS from sources (Advanced) 81

Android.Application.Programming.with.OpenCV

Preface 1 Chapter 1: Setting Up OpenCV 7 System requirements 8 Setting up a development environment 8 Getting a ready-made development environment – Tegra Android Development Pack (TAPD) 9 Assembling a development environment piece-by-piece 11 Getting the prebuilt OpenCV4Android 14 Building OpenCV4Android from source 14 Building the OpenCV samples with Eclipse 16 Finding documentation and help 30 Summary 30 Chapter 2: Working with Camera Frames 31 Designing our app – Second Sight 31 Creating the Eclipse project 34 Enabling camera and disk access in the manifest 38 Creating menu and string resources 40 Previewing and saving photos in CameraActivity 42 Deleting, editing, and sharing photos in LabActivity 52 Summary 55 Chapter 3: Applying Image Effects 57 Adding files to the project 57 Defining the Filter interface 59 Mixing color channels 60 Making subtle color shifts with curves 64 Processing a neighborhood of pixels with convolution filters 69 Adding the filters to CameraActivity 71 Summary 76 Chapter 4: Recognizing and Tracking Images 77 Adding files to the project 77 Understanding image tracking 79 Writing an image tracking filter 81 Adding the tracker filters to CameraActivity 87 Summary 92 Chapter 5: Combining Image Tracking with 3D Rendering 93 Adding files to the project 93 Defining the ARFilter interface 94 Building projection matrices in CameraProjectionAdapter 95 Modifying ImageDetectionFilter for 3D tracking 99 Rendering the cube in ARCubeRenderer 104 Adding 3D tracking and rendering to CameraActivity 108 Learning more about 3D graphics on Android 112 Summary 112 Index 113

A Practical Introduction to Computer Vision with OpenCV

Contents Preface xiii 1 Introduction 1 1.1 A Difficult Problem 1 1.2 The Human Vision System 2 1.3 Practical Applications of Computer Vision 3 1.4 The Future of Computer Vision 5 1.5 Material in This Textbook 6 1.6 Going Further with Computer Vision 7 2 Images 9 2.1 Cameras 9 2.1.1 The Simple Pinhole Camera Model 9 2.2 Images 10 2.2.1 Sampling 11 2.2.2 Quantisation 11 2.3 Colour Images 13 2.3.1 Red–Green–Blue (RGB) Images 14 2.3.2 Cyan–Magenta–Yellow (CMY) Images 17 2.3.3 YUV Images 17 2.3.4 Hue Luminance Saturation (HLS) Images 18 2.3.5 Other Colour Spaces 20 2.3.6 Some Colour Applications 20 2.4 Noise 22 2.4.1 Types of Noise 23 2.4.2 Noise Models 25 2.4.3 Noise Generation 26 2.4.4 Noise Evaluation 26 2.5 Smoothing 27 2.5.1 Image Averaging 27 2.5.2 Local Averaging and Gaussian Smoothing 28 2.5.3 Rotating Mask 30 2.5.4 Median Filter 31 3 Histograms 35 3.1 1D Histograms 35 3.1.1 Histogram Smoothing 36 3.1.2 Colour Histograms 37 3.2 3D Histograms 39 3.3 Histogram/Image Equalisation 40 3.4 Histogram Comparison 41 3.5 Back-projection 43 3.6 k-means Clustering 44 4 Binary Vision 49 4.1 Thresholding 49 4.1.1 Thresholding Problems 50 4.2 Threshold Detection Methods 51 4.2.1 Bimodal Histogram Analysis 52 4.2.2 Optimal Thresholding 52 4.2.3 Otsu Thresholding 54 4.3 Variations on Thresholding 56 4.3.1 Adaptive Thresholding 56 4.3.2 Band Thresholding 57 4.3.3 Semi-thresholding 58 4.3.4 Multispectral Thresholding 58 4.4 Mathematical Morphology 59 4.4.1 Dilation 60 4.4.2 Erosion 62 4.4.3 Opening and Closing 63 4.4.4 Grey-scale and Colour Morphology 65 4.5 Connectivity 66 4.5.1 Connectedness: Paradoxes and Solutions 66 4.5.2 Connected Components Analysis 67 5 Geometric Transformations 71 5.1 Problem Specification and Algorithm 71 5.2 Affine Transformations 73 5.2.1 Known Affine Transformations 74 5.2.2 Unknown Affine Transformations 75 5.3 Perspective Transformations 76 5.4 Specification of More Complex Transformations 78 5.5 Interpolation 78 5.5.1 Nearest Neighbour Interpolation 79 5.5.2 Bilinear Interpolation 79 5.5.3 Bi-Cubic Interpolation 80 5.6 Modelling and Removing Distortion from Cameras 80 5.6.1 Camera Distortions 81 5.6.2 Camera Calibration and Removing Distortion 82 6 Edges 83 6.1 Edge Detection 83 6.1.1 First Derivative Edge Detectors 85 6.1.2 Second Derivative Edge Detectors 92 6.1.3 Multispectral Edge Detection 97 6.1.4 Image Sharpening 98 6.2 Contour Segmentation 99 6.2.1 Basic Representations of Edge Data 99 6.2.2 Border Detection 102 6.2.3 Extracting Line Segment Representations of Edge Contours 105 6.3 Hough Transform 108 6.3.1 Hough for Lines 109 6.3.2 Hough for Circles 111 6.3.3 Generalised Hough 112 7 Features 115 7.1 Moravec Corner Detection 117 7.2 Harris Corner Detection 118 7.3 FAST Corner Detection 121 7.4 SIFT 122 7.4.1 Scale Space Extrema Detection 123 7.4.2 Accurate Keypoint Location 124 7.4.3 Keypoint Orientation Assignment 126 7.4.4 Keypoint Descriptor 127 7.4.5 Matching Keypoints 127 7.4.6 Recognition 127 7.5 Other Detectors 129 7.5.1 Minimum Eigenvalues 130 7.5.2 SURF 130 8 Recognition 131 8.1 Template Matching 131 8.1.1 Applications 131 8.1.2 Template Matching Algorithm 133 8.1.3 Matching Metrics 134 8.1.4 Finding Local Maxima or Minima 135 8.1.5 Control Strategies for Matching 137 8.2 Chamfer Matching 137 8.2.1 Chamfering Algorithm 137 8.2.2 Chamfer Matching Algorithm 139 8.3 Statistical Pattern Recognition 140 8.3.1 Probability Review 142 8.3.2 Sample Features 143 8.3.3 Statistical Pattern Recognition Technique 149 8.4 Cascade of Haar Classifiers 152 8.4.1 Features 154 8.4.2 Training 156 8.4.3 Classifiers 156 8.4.4 Recognition 158 8.5 Other Recognition Techniques 158 8.5.1 Support Vector Machines (SVM) 158 8.5.2 Histogram of Oriented Gradients (HoG) 159 8.6 Performance 160 8.6.1 Image and Video Datasets 160 8.6.2 Ground Truth 161 8.6.3 Metrics for Assessing Classification Performance 162 8.6.4 Improving Computation Time 165 9 Video 167 9.1 Moving Object Detection 167 9.1.1 Object of Interest 168 9.1.2 Common Problems 168 9.1.3 Difference Images 169 9.1.4 Background Models 171 9.1.5 Shadow Detection 179 9.2 Tracking 180 9.2.1 Exhaustive Search 181 9.2.2 Mean Shift 181 9.2.3 Dense Optical Flow 182 9.2.4 Feature Based Optical Flow 185 9.3 Performance 186 9.3.1 Video Datasets (and Formats) 186 9.3.2 Metrics for Assessing Video Tracking Performance 187 10 Vision Problems 189 10.1 Baby Food 189 10.2 Labels on Glue 190 10.3 O-rings 191 10.4 Staying in Lane 192 10.5 Reading Notices 193 10.6 Mailboxes 194 10.7 Abandoned and Removed Object Detection 195 10.8 Surveillance 196 10.9 Traffic Lights 197 10.10 Real Time Face Tracking 198 10.11 Playing Pool 199 10.12 Open Windows 200 10.13 Modelling Doors 201 10.14 Determining the Time from Analogue Clocks 202 10.15 Which Page 203 10.16 Nut/Bolt/Washer Classification 204 10.17 Road Sign Recognition 205 10.18 License Plates 206 10.19 Counting Bicycles 207 10.20 Recognise Paintings 208 References 209 Index 213

Algorithms for Image Processing and Computer Vision

Preface xxi Chapter 1 Practical Aspects of a Vision System — Image Display, Input/Output, and Library Calls 1 OpenCV 2 The Basic OpenCV Code 2 The IplImage Data Structure 3 Reading and Writing Images 6 Image Display 7 An Example 7 Image Capture 10 Interfacing with the AIPCV Library 14 Website Files 18 References 18 Chapter 2 Edge-Detection Techniques 21 The Purpose of Edge Detection 21 Traditional Approaches and Theory 23 Models of Edges 24 Noise 26 Derivative Operators 30 Template-Based Edge Detection 36 Edge Models: The Marr-Hildreth Edge Detector 39 The Canny Edge Detector 42 The Shen-Castan (ISEF) Edge Detector 48 A Comparison of Two Optimal Edge Detectors 51 Color Edges 53 Source Code for the Marr-Hildreth Edge Detector 58 Source Code for the Canny Edge Detector 62 Source Code for the Shen-Castan Edge Detector 70 Website Files 80 References 82 Chapter 3 Digital Morphology 85 Morphology Defined 85 Connectedness 86 Elements of Digital Morphology—Binary Operations 87 Binary Dilation 88 Implementing Binary Dilation 92 Binary Erosion 94 Implementation of Binary Erosion 100 Opening and Closing 101 MAX—A High-Level Programming Language for Morphology 107 The ‘‘Hit-and-Miss’’ Transform 113 Identifying Region Boundaries 116 Conditional Dilation 116 Counting Regions 119 Grey-Level Morphology 121 Opening and Closing 123 Smoothing 126 Gradient 128 Segmentation of Textures 129 Size Distribution of Objects 130 Color Morphology 131 Website Files 132 References 135 Chapter 4 Grey-Level Segmentation 137 Basics of Grey-Level Segmentation 137 Using Edge Pixels 139 Iterative Selection 140 The Method of Grey-Level Histograms 141 Using Entropy 142 Fuzzy Sets 146 Minimum Error Thresholding 148 Sample Results From Single Threshold Selection 149 The Use of Regional Thresholds 151 Chow and Kaneko 152 Modeling Illumination Using Edges 156 Implementation and Results 159 Comparisons 160 Relaxation Methods 161 Moving Averages 167 Cluster-Based Thresholds 170 Multiple Thresholds 171 Website Files 172 References 173 Chapter 5 Texture and Color 177 Texture and Segmentation 177 A Simple Analysis of Texture in Grey-Level Images 179 Grey-Level Co-Occurrence 182 Maximum Probability 185 Moments 185 Contrast 185 Homogeneity 185 Entropy 186 Results from the GLCM Descriptors 186 Speeding Up the Texture Operators 186 Edges and Texture 188 Energy and Texture 191 Surfaces and Texture 193 Vector Dispersion 193 Surface Curvature 195 Fractal Dimension 198 Color Segmentation 201 Color Textures 205 Website Files 205 References 206 Chapter 6 Thinning 209 What Is a Skeleton? 209 The Medial Axis Transform 210 Iterative Morphological Methods 212 The Use of Contours 221 Choi/Lam/Siu Algorithm 224 Treating the Object as a Polygon 226 Triangulation Methods 227 Force-Based Thinning 228 Definitions 229 Use of a Force Field 230 Subpixel Skeletons 234 Source Code for Zhang-Suen/Stentiford/Holt Combined Algorithm 235 Website Files 246 References 247 Chapter 7 Image Restoration 251 Image Degradations—The Real World 251 The Frequency Domain 253 The Fourier Transform 254 The Fast Fourier Transform 256 The Inverse Fourier Transform 260 Two-Dimensional Fourier Transforms 260 Fourier Transforms in OpenCV 262 Creating Artificial Blur 264 The Inverse Filter 270 The Wiener Filter 271 Structured Noise 273 Motion Blur—A Special Case 276 The Homomorphic Filter—Illumination 277 Frequency Filters in General 278 Isolating Illumination Effects 280 Website Files 281 References 283 Chapter 8 Classification 285 Objects, Patterns, and Statistics 285 Features and Regions 288 Training and Testing 292 Variation: In-Class and Out-Class 295 Minimum Distance Classifiers 299 Distance Metrics 300 Distances Between Features 302 Cross Validation 304 Support Vector Machines 306 Multiple Classifiers—Ensembles 309 Merging Multiple Methods 309 Merging Type 1 Responses 310 Evaluation 311 Converting Between Response Types 312 Merging Type 2 Responses 313 Merging Type 3 Responses 315 Bagging and Boosting 315 Bagging 315 Boosting 316 Website Files 317 References 318 Chapter 9 Symbol Recognition 321 The Problem 321 OCR on Simple Perfect Images 322 OCR on Scanned Images—Segmentation 326 Noise 327 Isolating Individual Glyphs 329 Matching Templates 333 Statistical Recognition 337 OCR on Fax Images—Printed Characters 339 Orientation—Skew Detection 340 The Use of Edges 345 Handprinted Characters 348 Properties of the Character Outline 349 Convex Deficiencies 353 Vector Templates 357 Neural Nets 363 A Simple Neural Net 364 A Backpropagation Net for Digit Recognition 368 The Use of Multiple Classifiers 372 Merging Multiple Methods 372 Results From the Multiple Classifier 375 Printed Music Recognition—A Study 375 Staff Lines 376 Segmentation 378 Music Symbol Recognition 381 Source Code for Neural Net Recognition System 383 Website Files 390 References 392 Chapter 10 Content-Based Search — Finding Images by Example 395 Searching Images 395 Maintaining Collections of Images 396 Features for Query by Example 399 Color Image Features 399 Mean Color 400 Color Quad Tree 400 Hue and Intensity Histograms 401 Comparing Histograms 402 Requantization 403 Results from Simple Color Features 404 Other Color-Based Methods 407 Grey-Level Image Features 408 Grey Histograms 409 Grey Sigma—Moments 409 Edge Density—Boundaries Between Objects 409 Edge Direction 410 Boolean Edge Density 410 Spatial Considerations 411 Overall Regions 411 Rectangular Regions 412 Angular Regions 412 Circular Regions 414 Hybrid Regions 414 Test of Spatial Sampling 414 Additional Considerations 417 Texture 418 Objects, Contours, Boundaries 418 Data Sets 418 Website Files 419 References 420 Systems 424 Chapter 11 High-Performance Computing for Vision and Image Processing 425 Paradigms for Multiple-Processor Computation 426 Shared Memory 426 Message Passing 427 Execution Timing 427 Using clock() 428 Using QueryPerformanceCounter 430 The Message-Passing Interface System 432 Installing MPI 432 Using MPI 433 Inter-Process Communication 434 Running MPI Programs 436 Real Image Computations 437 Using a Computer Network—Cluster Computing 440 A Shared Memory System—Using the PC Graphics Processor 444 GLSL 444 OpenGL Fundamentals 445 Practical Textures in OpenGL 448 Shader Programming Basics 451 Vertex and Fragment Shaders 452 Required GLSL Initializations 453 Reading and Converting the Image 454 Passing Parameters to Shader Programs 456 Putting It All Together 457 Speedup Using the GPU 459 Developing and Testing Shader Code 459 Finding the Needed Software 460 Website Files 461 References 461 Index 465

Emgu CV Essential

Preface 1 Chapter 1: Introduction to Emgu CV 5 What is Emgu CV? 5 Comparing image-processing libraries 6 License agreement 6 Documentation and other material 7 Ease of use 7 Performance 8 Summary of the comparison 9 Advantages of Emgu CV 9 Cross-platform 9 Cross-language support with examples 9 Other advantages 10 Summary 10 Chapter 2: Installing Emgu CV 11 Downloading Emgu CV 11 Installing Emgu CV 11 Installing on Windows 11 Installing on Linux 16 Getting the dependency 16 Building Emgu CV from source 17 Installing on OS X 18 Getting the dependency 19 Building Emgu CV from source 19 Troubleshooting 20 Windows 20 Linux 21 OS X 22 Summary 22 Chapter 3: Hello World 23 Hello World in C# 23 Creating a new project 24 Designing our form 27 Coding 27 Output 29 Hello World in VB.NET 30 Hello World in C++ 31 Summary 32 Chapter 4: Wrapping OpenCV 33 Architecture overview 33 OpenCV 33 Emgu CV 34 Function mapping 36 Structure mapping 36 Enumeration mapping 37 Summary 37 Chapter 5: Working with Images 39 Digital image representation 39 Pixels and data 39 Pixel resolution 40 Color image representation 41 Color depth 43 Working with images 44 Creating an image 44 Loading an image from a file 46 Operations with pixels 47 Method naming rules 49 Using operator overload 50 Generic operations support 51 Garbage collection 51 XML serialization 52 Summary 53 Chapter 6: Working with Matrices 55 Matrix and the Image class 55 Definition and parameters 56 Working with matrices 56 Creating a matrix 57 Operations with elements 58 Summary 59 Chapter 7: Shape Detection 61 Canny Edge Detector 61 Hough transforms 63 Hough Line transform 63 Hough Circle transform 65 Contour 67 Contour finding 68 Representation of contours 68 Sequences of vertexes 68 Free chain codes 69 Drawing contours 69 Polygon approximations 70 A contours example 70 Summary 72 Chapter 8: Face Detection 73 Biometric systems 73 Camera captures 75 Machine learning 76 Face detection or the Haar classifier 77 Boosting theory and supervised learning 78 Haar-like features 78 Code for face detection 81 Summary 83 Chapter 9: License Plate Recognition 85 License Plate Recognition 85 Algorithms for LPR 86 OCR 87 Tesseract-OCR 88 Code for License Plate Recognition 88 Assumption 89 Source code 89 GetWhitePixelMask 90 DetectLicensePlate 91 FindLicensePlate 92 Output 93 Summary 93 Chapter 10: Image Stitching 95 Image stitching 95 Algorithms for image stitching 96 Image matching 96 Image calibration 97 Image blending 97 Code 98 Summary 99 Index 101

Mastering OpenCV with Practical Computer Vision Projects

Preface 1 Chapter 1: Cartoonifier and Skin Changer for Android 7 Accessing the webcam 9 Main camera processing loop for a desktop app 10 Generating a black-and-white sketch 11 Generating a color painting and a cartoon 12 Generating an "evil" mode using edge filters 14 Generating an "alien" mode using skin detection 16 Skin-detection algorithm 16 Showing the user where to put their face 17 Implementation of the skin-color changer 19 Porting from desktop to Android 24 Setting up an Android project that uses OpenCV 24 Color formats used for image processing on Android 25 Input color format from the camera 25 Output color format for display 26 Adding the cartoonifier code to the Android NDK app 28 Reviewing the Android app 30 Cartoonifying the image when the user taps the screen 31 Saving the image to a file and to the Android picture gallery 33 Showing an Android notification message about a saved image 36 Changing cartoon modes through the Android menu bar 37 Reducing the random pepper noise from the sketch image 40 Showing the FPS of the app 43 Using a different camera resolution 43 Customizing the app 44 Summary 45 Chapter 2: Marker-based Augmented Reality on iPhone or iPad 47 Creating an iOS project that uses OpenCV 48 Adding OpenCV framework 49 Including OpenCV headers 51 Application architecture 52 Marker detection 62 Marker identification 64 Grayscale conversion 64 Image binarization 65 Contours detection 66 Candidates search 67 Marker code recognition 72 Reading marker code 72 Marker location refinement 74 Placing a marker in 3D 76 Camera calibration 76 Marker pose estimation 78 Rendering the 3D virtual object 82 Creating the OpenGL rendering layer 82 Rendering an AR scene 85 Summary 92 References 92 Chapter 3: Marker-less Augmented Reality 93 Marker-based versus marker-less AR 94 Using feature descriptors to find an arbitrary image on video 95 Feature extraction 95 Definition of a pattern object 98 Matching of feature points 98 PatternDetector.cpp 99 Outlier removal 100 Cross-match filter 101 Ratio test 101 Homography estimation 102 Homography refinement 104 Putting it all together 107 Pattern pose estimation 108 PatternDetector.cpp 108 Obtaining the camera-intrinsic matrix 110 Pattern.cpp 113 Application infrastructure 114 ARPipeline.hpp 115 ARPipeline.cpp 115 Enabling support for 3D visualization in OpenCV 116 Video capture using OpenCV 118 Rendering augmented reality 119 ARDrawingContext.hpp 119 ARDrawingContext.cpp 120 Demonstration 122 main.cpp 123 Summary 126 References 127 Chapter 4: Exploring Structure from Motion Using OpenCV 129 Structure from Motion concepts 130 Estimating the camera motion from a pair of images 132 Point matching using rich feature descriptors 132 Point matching using optical flow 134 Finding camera matrices 139 Reconstructing the scene 143 Reconstruction from many views 147 Refinement of the reconstruction 151 Visualizing 3D point clouds with PCL 155 Using the example code 158 Summary 159 References 160 Chapter 5: Number Plate Recognition Using SVM and Neural Networks 161 Introduction to ANPR 161 ANPR algorithm 163 Plate detection 166 Segmentation 167 Classification 173 Plate recognition 176 OCR segmentation 177 Feature extraction 178 OCR classification 181 Evaluation 185 Summary 188 Chapter 6: Non-rigid Face Tracking 189 Overview 191 Utilities 191 Object-oriented design 191 Data collection: Image and video annotation 193 Training data types 194 Annotation tool 198 Pre-annotated data (The MUCT dataset) 198 Geometrical constraints 199 Procrustes analysis 202 Linear shape models 205 A combined local-global representation 207 Training and visualization 209 Facial feature detectors 212 Correlation-based patch models 214 Learning discriminative patch models 214 Generative versus discriminative patch models 218 Accounting for global geometric transformations 219 Training and visualization 222 Face detection and initialization 224 Face tracking 228 Face tracker implementation 229 Training and visualization 231 Generic versus person-specific models 232 Summary 233 References 233 Chapter 7: 3D Head Pose Estimation Using AAM and POSIT 235 Active Appearance Models overview 236 Active Shape Models 238 Getting the feel of PCA 240 Triangulation 245 Triangle texture warping 247 Model Instantiation – playing with the Active Appearance Model 249 AAM search and fitting 250 POSIT 253 Diving into POSIT 253 POSIT and head model 256 Tracking from webcam or video file 257 Summary 259 References 260 Chapter 8: Face Recognition using Eigenfaces or Fisherfaces 261 Introduction to face recognition and face detection 261 Step 1: Face detection 263 Implementing face detection using OpenCV 264 Loading a Haar or LBP detector for object or face detection 265 Accessing the webcam 266 Detecting an object using the Haar or LBP Classifier 266 Detecting the face 268 Step 2: Face preprocessing 270 Eye detection 271 Eye search regions 272 Step 3: Collecting faces and learning from them 281 Collecting preprocessed faces for training 283 Training the face recognition system from collected faces 285 Viewing the learned knowledge 287 Average face 289 Eigenvalues, Eigenfaces, and Fisherfaces 290 Step 4: Face recognition 292 Face identification: Recognizing people from their face 292 Face verification: Validating that it is the claimed person 292 Finishing touches: Saving and loading files 295 Finishing touches: Making a nice and interactive GUI 295 Drawing the GUI elements 297 Checking and handling mouse clicks 306 Summary 308 References 309 Index 311

Practical OpenCV

About the Author ...............................................................................................................xiii About the Technical Reviewer ............................................................................................ xv Acknowledgments ............................................................................................................ xvii ■■Part 1: Getting Comfortable .......................................................................... 1 ■■Chapter 1: Introduction to Computer Vision and OpenCV ..................................................3 Why Was This Book Written? .........................................................................................................3 OpenCV ..........................................................................................................................................4 History of OpenCV ..................................................................................................................................................4 Built-in Modules ....................................................................................................................................................4 Summary .......................................................................................................................................5 ■■Chapter 2: Setting up OpenCV on Your Computer ..............................................................7 Operating Systems ........................................................................................................................7 Ubuntu ...................................................................................................................................................................7 Windows ..............................................................................................................................................................12 Mac OSX ..............................................................................................................................................................12 Summary .....................................................................................................................................12 ■■Chapter 3: CV Bling—OpenCV Inbuilt Demos ..................................................................13 Camshift ......................................................................................................................................13 Stereo Matching ..........................................................................................................................16 Homography Estimation in Video ................................................................................................16 Circle and Line Detection ............................................................................................................18 Image Segmentation ...................................................................................................................19 Bounding Box and Circle .............................................................................................................21 Image Inpainting .........................................................................................................................21 Summary .....................................................................................................................................22 ■■Chapter 4: Basic Operations on Images and GUI Windows ..............................................23 Displaying Images from Disk in a Window ..................................................................................23 The cv::Mat Structure ..................................................................................................................24 Creating a cv::Mat ...............................................................................................................................................25 Accessing elements of a cv::Mat ......................................................................................................................... 25 Expressions with cv::Mat ..................................................................................................................................... 25 Converting Between Color-spaces ..............................................................................................26 GUI Track-Bars and Callback Functions ......................................................................................27 Callback Functions .............................................................................................................................................27 ROIs: Cropping a Rectangular Portion out of an Image ...............................................................30 Region of Interest in an Image ............................................................................................................................ 30 Accessing Individual Pixels of an Image .....................................................................................33 Exercise ...............................................................................................................................................................33 Videos ..........................................................................................................................................34 Displaying the Feed from Your Webcam or USB Camera/File .............................................................................. 34 Writing Videos to Disk ..........................................................................................................................................36 Summary .....................................................................................................................................37 ■■ Part 2: Advanced Computer Vision Problems and Coding Them in OpenCV ......................................................................................... 39 ■■Chapter 5: Image Filtering ...............................................................................................41 Image Filters ...............................................................................................................................41 Blurring Images ...................................................................................................................................................45 Resizing Images—Up and Down ......................................................................................................................... 48 Eroding and Dilating Images ................................................................................................................................ 49 Detecting Edges and Corners Efficiently in Images ............................................................................................. 51 Edges ...................................................................................................................................................................51 Canny Edges ........................................................................................................................................................56 Corners ................................................................................................................................................................57 Object Detector App ....................................................................................................................60 Morphological Opening and Closing of Images to Remove Noise ...............................................63 Summary .....................................................................................................................................65 ■■Chapter 6: Shapes in Images ...........................................................................................67 Contours ......................................................................................................................................67 Point Polygon Test ...............................................................................................................................................70 Hough Transform .........................................................................................................................74 Detecting Lines with Hough Transform ................................................................................................................ 74 Detecting Circles with Hough Transform ............................................................................................................. 77 Generalized Hough Transform .....................................................................................................80 RANdom Sample Consensus (RANSAC) .......................................................................................80 Bounding Boxes and Circles ........................................................................................................91 Convex Hulls ................................................................................................................................92 Summary .....................................................................................................................................93 ■■Chapter 7: Image Segmentation and Histograms ............................................................95 Image Segmentation ...................................................................................................................95 Simple Segmentation by Thresholding ................................................................................................................ 96 Floodfill ..............................................................................................................................................................100 Watershed Segmentation .................................................................................................................................. 103 GrabCut Segmentation ......................................................................................................................................110 Histograms ................................................................................................................................111 Equalizing Histograms .......................................................................................................................................111 Histogram Backprojections ............................................................................................................................... 113 Meanshift and Camshift ....................................................................................................................................116 Summary ...................................................................................................................................117 ■■Chapter 8: B asic Machine Learning and Object D etection Based on Keypoints ............119 Keypoints and Keypoint Descriptors: Introduction and Terminology .........................................119 General Terms ....................................................................................................................................................120 How Does the Keypoint-Based Method Work? .................................................................................................. 120 SIFT Keypoints and Descriptors ................................................................................................121 Keypoint Detection and Orientation Estimation ................................................................................................. 121 SIFT Keypoint Descriptors ................................................................................................................................. 125 Matching SIFT Descriptors ................................................................................................................................ 125 SURF Keypoints and Descriptors ...............................................................................................131 SURF Keypoint Detection ...................................................................................................................................131 SURF Descriptor ................................................................................................................................................134 ORB (Oriented FAST and Rotated BRIEF) ...................................................................................136 Oriented FAST Keypoints ...................................................................................................................................137 BRIEF Descriptors ..............................................................................................................................................137 Basic Machine Learning ............................................................................................................140 SVMs ..................................................................................................................................................................141 Object Categorization ................................................................................................................142 Strategy .............................................................................................................................................................142 Organization ......................................................................................................................................................143 Summary ...................................................................................................................................153 ■■ Chapter 9: Affine and Perspective Transformations and Their Applications to Image Panoramas .....................................................................................................155 Affine Transforms ......................................................................................................................155 Applying Affine Transforms ................................................................................................................................ 156 Estimating Affine Transforms ............................................................................................................................. 158 Perspective Transforms .............................................................................................................161 Panoramas ................................................................................................................................166 Summary ...................................................................................................................................172 ■■Chapter 10: 3D Geometry and Stereo Vision .................................................................173 Single Camera Calibration .........................................................................................................173 OpenCV Implementation of Single Camera Calibration ...................................................................................... 176 Stereo Vision .............................................................................................................................179 Triangulation ......................................................................................................................................................179 Calibration .........................................................................................................................................................180 Rectification and Disparity by Matching ............................................................................................................ 186 Summary ...................................................................................................................................200 ■■ Chapter 11: Embedded Computer Vision: Running OpenCV Programs on the Raspberry Pi ................................................................................................................. 201 Raspberry Pi ..............................................................................................................................202 Setting Up Your New Raspberry Pi ............................................................................................202 Installing Raspbian on the Pi ............................................................................................................................. 203 Initial Settings ....................................................................................................................................................204 Installing OpenCV ...............................................................................................................................................205 Camera board ............................................................................................................................206 Camera Board vs. USB Camera ......................................................................................................................... 207 Frame-Rate Comparisons .................................................................................................................................. 214 Usage Examples ........................................................................................................................215 Color-based Object Detector .............................................................................................................................. 216 ORB Keypoint-based Object Detector ................................................................................................................ 217 Summary ...................................................................................................................................218 Index .................................................................................................................................219

OpenCV 2 Computer Vision Application Programming Cookbook

Table of Contents Preface 1 Chapter 1: Playing with Images 7 Introduction 7 Installing the OpenCV library 8 Creating an OpenCV project with MS Visual C++ 11 Creating an OpenCV project with Qt 19 Loading, displaying, and saving images 26 Creating a GUI application using Qt 30 Chapter 2: Manipulating the Pixels 37 Introduction 37 Accessing pixel values 38 Scanning an image with pointers 41 Scanning an image with iterators 49 Writing efficient image scanning loops 51 Scanning an image with neighbor access 55 Performing simple image arithmetic 59 Defining regions of interest 63 Chapter 3: Processing Images with Classes 69 Introduction 69 Using the Strategy pattern in algorithm design 70 Using a Controller to communicate with processing modules 76 Using the Singleton design pattern 80 Using the Model-View-Controller architecture to design an application 82 Converting color spaces 85 Chapter 4: Counting the Pixels with Histograms 89 Introduction 89 Computing the image histogram 89 Applying look-up tables to modify image appearance 96 Equalizing the image histogram 101 Backprojecting a histogram to detect specific image content 103 Using the mean shift algorithm to find an object 108 Retrieving similar images using histogram comparison 112 Chapter 5: Transforming Images with Morphological Operations 117 Introduction 117 Eroding and dilating images using morphological filters 118 Opening and closing images using morphological filters 122 Detecting edges and corners using morphological filters 125 Segmenting images using watersheds 131 Extracting foreground objects with the GrabCut algorithm 137 Chapter 6: Filtering the Images 141 Introduction 141 Filtering images using low-pass filters 142 Filtering images using a median filter 147 Applying directional filters to detect edges 148 Computing the Laplacian of an image 156 Chapter 7: Extracting Lines, Contours, and Components 163 Introduction 163 Detecting image contours with the Canny operator 164 Detecting lines in images with the Hough transform 167 Fitting a line to a set of points 178 Extracting the components' contours 182 Computing components' shape descriptors 186 Chapter 8: Detecting and Matching Interest Points 191 Introduction 191 Detecting Harris corners 192 Detecting FAST features 203 Detecting the scale-invariant SURF features 206 Describing SURF features 212 Chapter 9: Estimating Projective Relations in Images 217 Introduction 217 Calibrating a camera 219 Computing the fundamental matrix of an image pair 228 Matching images using random sample consensus 233 Computing a homography between two images 242 Chapter 10: Processing Video Sequences 247 Introduction 247 Reading video sequences 248 Processing the video frames 251 Writing video sequences 261 Tracking feature points in video 266 Extracting the foreground objects in video 272 Index 279