Bayesian Neural Networks

所需积分/C币:19 2019-02-20 10:45:11 1.06MB PDF
收藏 收藏

Author(s): Herbert K. H. Lee1 University of California, Santa Cruz, Santa Cruz, California 1 Published: 2004 Print ISBN13: 9780898715637 eISBN: 9780898718423 DOI: 10.1137/1.9780898718423 Book Code: SA13 Series: ASA-SIAM Series on Statistics and Applied Probability Pages: x + 93
SAl3 LeeFMA gxd 5/17/2004 10: 21 AM Page 3 Bayesian Nonparametrics via neural networks Herbert K.h. Lee University of California, Santa cruz Santa cruz, California n ASA Society for Industrial and Applied Mathematics American Statistical association Philadelphia, Pennsy/vania Alexandria. Virginia SAl3 LeeFMA gxd 5/17/2004 10: 21 AM Page 4 opyright@ 2004 by the American Statistical Association and the Society for Industrial and Applied Mathematics 0987654321 All rights reserved. Printed in the United States of America. No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher. For information, write to the Society for Industrial and Applied Mathematics, 3 600 University City Science Center, Philadelphia, PA 19104-2688 No warranties, express or implied, are made by the publisher, authors, and their employers that the programs contained in this volume are free of error. they hould not be relied on as the sole basis to solve a problem whose incorrect solution could result in injury to person or property. If the programs are employed in such a manner, it is at the user's own risk and the publisher, authors and their employers disclaim all liability for such misuse Trademarked names may be used in this book without the inclusion of a trademark symbol. These names are used in an editorial context only; no infringement of trademark is intended S-PLUS is a registered trademark of Insightful Corporation SAS is a registered trademark of sas institute Inc This research was suppor ted in part by the national Science Foundation (grants DMS 9803433, 9873275, and 0233710) and the National Institutes of Health (grant RO1 CA54852-08) Library of Congress Cataloging-in-Publication Data Lee, herbert K. H Bayesian nonparametrics via neural networks/ Herbert K H Lee p cm -(ASA-SIAM series on statistics and applied probability Includes bibliographical references and index ISBN0-89871-563-6(pbk.) Bayesian statistical decision theory. 2. Nonparametric statistics. 3. Neural networks( Computer science)I. Title IL Series QA279.5432004 519.5′42-dc22 2004048151 a portion of the royalties from the sale of this book are being placed in a fund to help students attend SIAM meetings and other SlAM-related activities. This fund is administered by slam and qualified individuals encouraged to write directly to sIAM for guidelines is a registered trademark 中2004/5/7 pag Contents List of figures Preface 1 Introduction 1.1 Statistics and Machine Learning 1.2 Outline of the book 2 1.3 Regression Example--Groundlevel Ozone Pollution 1. 4 Classification Example-Loan applications 1. 5 A Simple neural Network example 2 Nonparametric Models 2.1 Nonparametric regression 2. 1. 1 Local method 2.1.2 Regression Using Basis Functions 2269 2.2 Nonparametric classification 2. 3 Neural Networks 20 2.3.1 Neural Networks Are statistical models 2 3.2 A Brief History of Neural Networks 22 .3.3 Multivariate Regression 2. 3. 4 Classification 23 2.3.5 Other Flavors of Neural Networks 26 2.4 The Bayesian Paradigm 28 2.5 Model building 29 3 Priors for Neural networks 31 3. 1 Parameter Interpretation, and lack thereof 31 3.2 Proper Priors 33 3.3 Noninformative Priors 37 3.3.1 Flat priors 38 3.3.2 Jeffreys priors 3.3. 3 Reference priors .46 3.4 Hybrid Priors 46 3.5 Model fitting 48 中2004/5/7 page v Contents 3.6 Example Comparing Priors 51 3.7 Asymptotic Consistency of the posterior .53 Building a model 57 4.1 Model Selection and Model Averaging 57 4.1.1 Modeling versus prediction 59 4.1.2 Bayesian Model selection 4.1.3 Computational Approximations for Model Selection 61 4.1.4 Model averaging 4.1.5 Shrinkage Priors 63 4.1.6 Automatic relevance determination 4.1.7 Bagging 4.2 Searching the Model Space 4.2.1 Greedy algorithms 67 4.2.2 Stochastic Algorithms 4.2.3 Reversible Jump Markov Chain Monte Carlo .70 4.3 Ozone Data analysis 7 4.4 Loan Data Analysis 5 Conclusions a Reference Prior Derivation 81 Glossary Bibliography Index 95 中2004/5/7 page vil List of Figures Pairwise scatterplots for the ozone data Estimated smooths for the ozone data 1.3 Correlated variables: Age vs current residence for loan applicants. 1. 4 A neural network fitted function 45799 1.5 Simple neural network model diagram 2. 1 A tree model for ozone using only wind speed and humidity 14 2.2 Example wavelets from the Haar family. 2.3 Diagram of nonparametric methods 18 2. 4 Neural network model diagram .20 2.5 Multivariate response neural network model diagram 23 2.6 Probability of loan acceptance by only age, 6 nodes 26 2.7 Probability of loan acceptance by only age, 4 nodes 3. 1 Fitted function with a single hidden node .32 3.2 Maximum likelihood fit for a two-hidden node network 34 3.3 Logistic basis functions of the fit in figure 3. 2 34 3. 4 DAG for the muller and rios insua model 35 DAG for the neal model 37 Comparison of priors 4.1 Fitted mean functions for several sizes of networl 58 4.2 Bagging fitted values 4.3 Fitted ozone values by day of year 72 4.4 Fitted ozone values by vertical height, humidity, pressure gradient, and inversion base temperature........ 4.5 Fitted ozone values by actual recorded levels .74 4.6 Probability of loan acceptance by time in current residence 4.7 Probability of loan acceptance by age 4.8 Probability of loan acceptance by income 77 中2004/5/7 pag Preface When I first heard about neural networks and how great they were. I was rather skeptical. Being sold as a magical black box, there was enough hype to make one believe that they could solve the worlds problems. when i tried to learn more about them, I found that most of the literature was written for a machine learning audience, and I had to grapple with a new perspective and a new set of terminology. After some work, I came to see neural networks from a statistical perspective, as a probability model. One of the primary motivations for this book was to write about neural networks for statisticians, addressing issues and concerns of interest to statisticians, and using statistical terminology. Neural networks are a powerful model, and should be treated as such, rather than disdained as a mere algorithm as I have found some statisticians do. Hopefully this book will prove to be illuminating The phrase""Bayesian nonparametrics"means different things to different people. The traditional interpretation usually implies infinite dimensional processes such as dirichlet processes, used for problems in regression, classification. and density estimation. while on the surface this book may not appear to fit that description, it is actually close. One of the themes of this book is that a neural network can be viewed as a finite-dimensional approximation to an infinite-dimensional model, and that this model is useful in practice for problems in regression and classification Thus the first section of this book will focus on introducing neural networks within the statistical context of nonparametric regression and classification. The rest of the book will examine important statistical modeling issues for Bayesian neural networks, particularly the choice of prior and the choice of model While this book will not assume the reader has any prior knowledge about neural networks, neither will it try to be an all-inclusive introduction. Topics will be introduced in a self-contained manner with references provided for further details of the many issues that will not be directly addressed in this book The target audience for this book is practicing statisticians and researchers, as well as students preparing for either or both roles. This book addresses practical and theoretical issues. It is hoped that the users of neural networks will want an understanding of how the model works, which can lead to a better appreciation of knowing when it is working and when it is not. It will be assumed that the reader has already been introduced to the basics of the bayesian approach with only a brief review and additional references provided There are a number of good introductory books on Bayesian statistics available(see Section 2. 4) so it does not seem productive to repeat that material here. It will also be assumed that the reader has a solid background in mathematical statistics and in linear regression, such as 中2004/5/7 page x Preface that which would be acquired as part of a traditional masters degree in statistics. However, there are few formal proofs and much of the text should be accessible even without this background. Computational issues will be discussed at conceptual and algorithmic levels. This work developed from my Ph. D. thesis("Model Selection and Model Averaging for Neural Networks, Carnegie Mellon University, Department of Statistics, 1998).I aIn grateful for all the assistance and know ledge provided by my advisor, Larry Wasserman would also like to acknowledge the many individuals who have contributed to this effort including David Banks, Jim Berger, Roberto Carta, Merlise Clyde, Daniel Cork, Scott Davies, Sigrunn Eliassen, Chris Genovese, Robert Giramacy, Rob Kass, Milovan Krnjajic Meena mani. Daniel merl. Andrew moore Peter muller. Mark schervish, Valerie Ventura and Kert Viele, as well as the staff at siam and a number of anonymous referees and editors, both for this book and for the papers preceding it. At various points during this work, funding has been provided by the National Science Foundation(grants DMS9803433 9873275, and 0233710)and the National Institutes of Health(grant Rol CA54852-08) 中2004/5/7 page I Chapter 1 Introduction The goal of this book is to put neural network models firmly into a statistical framework, treating them with the accompanying rigor normally accorded to statistical models. A neural network is frequently seen as either a magical black box, or purely as a machine learning algorithm, when in fact there is a definite probability model behind it. This book will start by showing how neural networks are indeed a statistical model for doing nonparametric regression or classification. The focus will be on a Bayesian perspective, although many of the topics will apply to frequentist models as well. As much of the literature on neural networks appears in the computer science realm, many standard modeling questions fail to get addressed. In particular, this book will take a hard look at key modeling issues such as choosing an appropriate prior and dealing with model selection. Most of the existing literature deals with neural networks as an algorithm. The hope of this book is to shift the focus back to modeling. 1.1 Statistics and Machine Learning The fields of statistics and machine learning are two approaches tow ard the same goals with much in common. In both cases, the idea is to learn about a problem from data. In most cases, this is either a classification problem, a regression problem, an exploratory data analysis problem, or some combination of the above. Where statistics and machine learnin differ most is in their perspective. It is sort of like two people, one standing outside of an airplane, and one standing inside the same plane, both asked to describe this plane. The person outside might discuss the length, the wingspan, the number of engines and their layout, and so on. The person inside might comment on the number of rows of seats the number of aisles, the seat configurations, the amount of overhead storage space, the number of lavatories, and so on. In the end, they are both describing the same plane. The relationship between statistics and machine learning is much like this situation. As much of the terminology differs between the two fields, towards the end of this book a glossary is provided for translating relevant machine learning terms into statistical terms As a bit of an overgeneralization, the field of statistics and the methods that come out of it are based on probability models. At the heart of almost all analyses, there is some

试读 102P Bayesian Neural Networks
立即下载 低至0.43元/次 身份认证VIP会员低至7折
关注 私信
Bayesian Neural Networks 19积分/C币 立即下载
Bayesian Neural Networks第1页
Bayesian Neural Networks第2页
Bayesian Neural Networks第3页
Bayesian Neural Networks第4页
Bayesian Neural Networks第5页
Bayesian Neural Networks第6页
Bayesian Neural Networks第7页
Bayesian Neural Networks第8页
Bayesian Neural Networks第9页
Bayesian Neural Networks第10页
Bayesian Neural Networks第11页
Bayesian Neural Networks第12页
Bayesian Neural Networks第13页
Bayesian Neural Networks第14页
Bayesian Neural Networks第15页
Bayesian Neural Networks第16页
Bayesian Neural Networks第17页
Bayesian Neural Networks第18页
Bayesian Neural Networks第19页
Bayesian Neural Networks第20页

试读结束, 可继续阅读

19积分/C币 立即下载 >