Python Data Science Handbook (Jupyter Notebook Version)

Python Data Science Handbook

launch

binder

This repository contains the entire Python Data Science Handbook, in the form of (free!) Jupyter notebooks.

How to Use this Book

Read the book in its entirety online at https://jakevdp.github.io/PythonDataScienceHandbook/

Run the code using the Jupyter notebooks available in this repository's notebooks directory.

The book was written and tested with Python 3.5, though other Python versions (including Python 2.7) should work in nearly all cases.

The book introduces the core libraries essential for working with data in Python: particularly IPython, NumPy, Pandas, Matplotlib, Scikit-Learn, and related

packages. Familiarity with Python as a language is assumed; if you need a quick introduction to the language itself, see the free companion project, A

Whirlwind Tour of Python: it's a fast-paced introduction to the Python language aimed at researchers and scientists.

See Index.ipynb for an index of the notebooks available to accompany the text.

Software

The code in the book was tested with Python 3.5, though most (but not all) will also work correctly with Python 2.7 and other older Python versions.

The packages I used to run the code in the book are listed in requirements.txt (Note that some of these exact version numbers may not be available on

your platform: you may have to tweak them for your own use). To install the requirements using conda, run the following at the command-line:

$ conda install --file requirements.txt

The code in this repository, including all code samples in the notebooks listed above, is released under the MIT license. Read more at the Open Source

Initiative.

Text

The text content of the book is released under the CC-BY-NC-ND license. Read more at Creative Commons.

This notebook contains an excerpt from the Python Data Science Handbook by Jake VanderPlas; the content is available on GitHub.

The text is released under the CC-BY-NC-ND license, and code is released under the MIT license. If you find this content useful, please

consider supporting the work by buying the book!

| Contents | IPython: Beyond Normal Python >

Preface

What Is Data Science?

2010:

While some of the intersection labels are a bit tongue-in-cheek, this diagram captures the essence of what I think people mean when they say "data

science": it is fundamentally an interdisciplinary subject. Data science comprises three distinct and overlapping areas: the skills of a statistician who knows

how to model and summarize datasets (which are growing ever larger); the skills of a computer scientist who can design and use algorithms to efficiently

store, process, and visualize this data; and the domain expertise—what we might think of as "classical" training in a subject—necessary both to formulate

the right questions and to put their answers in context.

With this in mind, I would encourage you to think of data science not as a new domain of knowledge to learn, but a new set of skills that you can apply

within your current area of expertise. Whether you are reporting election results, forecasting stock returns, optimizing online ad clicks, identifying

microorganisms in microscope photos, seeking new classes of astronomical objects, or working with data in any other field, the goal of this book is to give

audience is available online, I've long been frustrated by the lack of a single good answer to this question; that is what inspired this book.

The book is not meant to be an introduction to Python or to programming in general; I assume the reader has familiarity with the Python language,

including defining functions, assigning variables, calling methods of objects, controlling the flow of a program, and other basic tasks. Instead it is meant to

help Python users learn to use Python's data science stack–libraries such as IPython, NumPy, Pandas, Matplotlib, Scikit-Learn, and related tools–to

effectively store, manipulate, and gain insight from data.

Why Python?

Python has emerged over the last couple decades as a first-class tool for scientific computing tasks, including the analysis and visualization of large

datasets. This may have come as a surprise to early proponents of the Python language: the language itself was not specifically designed with data

analysis or scientific computing in mind. The usefulness of Python for data science stems primarily from the large and active ecosystem of third-party

packages: NumPy for manipulation of homogeneous array-based data, Pandas for manipulation of heterogeneous and labeled data, SciPy for common

scientific computing tasks, Matplotlib for publication-quality visualizations, IPython for interactive execution and sharing of code, Scikit-Learn for machine

was first released in 2008, adoption has been relatively slow, particularly in the scientific and web development communities. This is primarily because it

took some time for many of the essential third-party packages and toolkits to be made compatible with the new language internals. Since early 2014,

however, stable releases of the most important tools in the data science ecosystem have been fully compatible with both Python 2 and 3, and so this book

Outline of the Book

Each chapter of this book focuses on a particular package or tool that contributes a fundamental piece of the Python Data Sciece story.

Using Code Examples

Supplemental material (code examples, figures, etc.) is available for download at http://github.com/jakevdp/PythonDataScienceHandbook/. This book is

here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not

need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of

code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a

question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into

your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example:

If you feel your use of code examples falls outside fair use or the per‐ mission given above, feel free to contact us at permissions@oreilly.com.

geared toward Python packages, similar in spirit to the apt or yum tools that Linux users might be familiar with.

Anaconda includes both Python and conda, and additionally bundles a suite of other pre-installed packages geared toward scientific computing.

Because of the size of this bundle, expect the installation to consume several gigabytes of disk space.

Any of the packages included with Anaconda can also be installed manually on top of Miniconda; for this reason I suggest starting with Miniconda.

To get started, download and install the Miniconda package–make sure to choose a version with Python 3–and then install the core packages used in this

book:

[~]$ conda install numpy pandas scikit-learn matplotlib seaborn jupyter

Throughout the text, we will also make use of other more specialized tools in Python's scientific ecosystem; installation is usually as easy as typing

conda install packagename. For more information on conda, including information about creating and using conda environments (which I would highly

recommend), refer to conda's online documentation.

There are many options for development environments for Python, and I'm often asked which one I use in my own work. My answer sometimes surprises

people: my preferred environment is IPython plus a text editor (in my case, Emacs or Atom depending on my mood). IPython (short for Interactive Python)

was started in 2001 by Fernando Perez as an enhanced Python interpreter, and has since grown into a project aiming to provide, in Perez's words, "Tools

for the entire life cycle of research computing." If Python is the engine of our data science task, you might think of IPython as the interactive control panel.

As well as being a useful interactive interface to Python, IPython also provides a number of useful syntactic additions to the language; we'll cover the most

useful of these additions here. In addition, IPython is closely tied with the Jupyter project, which provides a browser-based notebook that is useful for

development, collaboration, sharing, and even publication of data science results. The IPython notebook is actually a special case of the broader Jupyter

notebook structure, which encompasses notebooks for Julia, R, and other programming languages. As an example of the usefulness of the notebook

format, look no further than the page you are reading: the entire manuscript for this book was composed as a set of IPython notebooks.

IPython is about using Python effectively for interactive scientific and data-intensive computing. This chapter will start by stepping through some of the

one or the other, we will explicitly state that fact. Before we start, some words on how to launch the IPython shell and IPython notebook.

Launching the IPython Shell

This chapter, like most of this book, is not designed to be absorbed passively. I recommend that as you read through it, you follow along and experiment

with the tools and syntax we cover: the muscle-memory you build through doing this will be far more useful than the simple act of reading about it. Start by

launching the IPython interpreter by typing

a launcher specific to your system (we'll discuss this more fully in Help and Documentation in IPython).

Once you do this, you should see a prompt like the following:

IPython 4.0.1 -- An enhanced Interactive Python.

? -> Introduction and overview of IPython's features.

The text is released under the CC-BY-NC-ND license, and code is released under the MIT license. If you find this content useful, please

consider supporting the work by buying the book!