Preface What Is Data Science? Who Is This Book For? Why Python? Python 2 Versus Python 3 Outline of This Book Using Code Examples Installation Considerations Conventions Used in This Book O’Reilly Safari How to Contact Us 1. IPython: Beyond Normal Python Shell or Notebook? Launching the IPython Shell Launching the Jupyter Notebook Help and Documentation in IPython Accessing Documentation with ? Accessing Source Code with ?? Exploring Modules with Tab Completion Tab completion of object contents Tab completion when importing Beyond tab completion: Wildcard matching Keyboard Shortcuts in the IPython Shell Navigation Shortcuts Text Entry Shortcuts Command History Shortcuts Miscellaneous Shortcuts IPython Magic Commands Pasting Code Blocks: %paste and %cpaste Running External Code: %run Timing Code Execution: %timeit Help on Magic Functions: ?, %magic, and %lsmagic Input and Output History IPython’s In and Out Objects Underscore Shortcuts and Previous Outputs Suppressing Output Related Magic Commands IPython and Shell Commands Quick Introduction to the Shell Shell Commands in IPython Passing Values to and from the Shell Shell-Related Magic Commands Errors and Debugging Controlling Exceptions: %xmode Debugging: When Reading Tracebacks Is Not Enough Partial list of debugging commands Profiling and Timing Code Timing Code Snippets: %timeit and %time Profiling Full Scripts: %prun Line-by-Line Profiling with %lprun Profiling Memory Use: %memit and %mprun More IPython Resources Web Resources Books 2. Introduction to NumPy Understanding Data Types in Python A Python Integer Is More Than Just an Integer A Python List Is More Than Just a List Fixed-Type Arrays in Python Creating Arrays from Python Lists Creating Arrays from Scratch NumPy Standard Data Types The Basics of NumPy Arrays NumPy Array Attributes Array Indexing: Accessing Single Elements Array Slicing: Accessing Subarrays One-dimensional subarrays Multidimensional subarrays Accessing array rows and columns Subarrays as no-copy views Creating copies of arrays Reshaping of Arrays Array Concatenation and Splitting Concatenation of arrays Splitting of arrays Computation on NumPy Arrays: Universal Functions The Slowness of Loops Introducing UFuncs Exploring NumPy’s UFuncs Array arithmetic Absolute value Trigonometric functions Exponents and logarithms Specialized ufuncs Advanced Ufunc Features Specifying output Aggregates Outer products Ufuncs: Learning More Aggregations: Min, Max, and Everything in Between Summing the Values in an Array Minimum and Maximum Multidimensional aggregates Other aggregation functions Example: What Is the Average Height of US Presidents? Computation on Arrays: Broadcasting Introducing Broadcasting Rules of Broadcasting Broadcasting example 1 Broadcasting example 2 Broadcasting example 3 Broadcasting in Practice Centering an array Plotting a two-dimensional function Comparisons, Masks, and Boolean Logic Example: Counting Rainy Days Digging into the data Comparison Operators as ufuncs Working with Boolean Arrays Counting entries Boolean operators Boolean Arrays as Masks Fancy Indexing Exploring Fancy Indexing Combined Indexing Example: Selecting Random Points Modifying Values with Fancy Indexing Example: Binning Data Sorting Arrays Fast Sorting in NumPy: np.sort and np.argsort Sorting along rows or columns Partial Sorts: Partitioning Example: k-Nearest Neighbors Structured Data: NumPy’s Structured Arrays Creating Structured Arrays More Advanced Compound Types RecordArrays: Structured Arrays with a Twist On to Pandas 3. Data Manipulation with Pandas Installing and Using Pandas Introducing Pandas Objects The Pandas Series Object Series as generalized NumPy array Series as specialized dictionary Constructing Series objects The Pandas DataFrame Object DataFrame as a generalized NumPy array DataFrame as specialized dictionary Constructing DataFrame objects From a single Series object From a list of dicts From a dictionary of Series objects From a two-dimensional NumPy array From a NumPy structured array The Pandas Index Object Index as immutable array Index as ordered set Data Indexing and Selection Data Selection in Series Series as dictionary Series as one-dimensional array Indexers: loc, iloc, and ix Data Selection in DataFrame DataFrame as a dictionary DataFrame as two-dimensional array Additional indexing conventions Operating on Data in Pandas Ufuncs: Index Preservation UFuncs: Index Alignment Index alignment in Series Index alignment in DataFrame Ufuncs: Operations Between DataFrame and Series Handling Missing Data Trade-Offs in Missing Data Conventions Missing Data in Pandas None: Pythonic missing data NaN: Missing numerical data NaN and None in Pandas Operating on Null Values Detecting null values Dropping null values Filling null values Hierarchical Indexing A Multiply Indexed Series The bad way The better way: Pandas MultiIndex MultiIndex as extra dimension Methods of MultiIndex Creation Explicit MultiIndex constructors MultiIndex level names MultiIndex for columns Indexing and Slicing a MultiIndex Multiply indexed Series Multiply indexed DataFrames Rearranging Multi-Indices Sorted and unsorted indices Stacking and unstacking indices Index setting and resetting Data Aggregations on Multi-Indices Combining Datasets: Concat and Append Recall: Concatenation of NumPy Arrays Simple Concatenation with pd.concat Duplicate indices Catching the repeats as an error Ignoring the index Adding MultiIndex keys Concatenation with joins The append() method Combining Datasets: Merge and Join Relational Algebra Categories of Joins One-to-one joins Many-to-one joins Many-to-many joins Specification of the Merge Key The on keyword The left_on and right_on keywords The left_index and right_index keywords Specifying Set Arithmetic for Joins Overlapping Column Names: The suffixes Keyword Example: US States Data Aggregation and Grouping Planets Data Simple Aggregation in Pandas GroupBy: Split, Apply, Combine Split, apply, combine The GroupBy object Column indexing Iteration over groups Dispatch methods Aggregate, filter, transform, apply Aggregation Filtering Transformation The apply() method Specifying the split key A list, array, series, or index providing the grouping keys A dictionary or series mapping index to group Any Python function A list of valid keys Grouping example Pivot Tables Motivating Pivot Tables Pivot Tables by Hand Pivot Table Syntax Multilevel pivot tables Additional pivot table options Example: Birthrate Data Further data exploration Vectorized String Operations Introducing Pandas String Operations Tables of Pandas String Methods Methods similar to Python string methods Methods using regular expressions Miscellaneous methods Vectorized item access and slicing Indicator variables Example: Recipe Database A simple recipe recommender Going further with recipes Working with Time Series Dates and Times in Python Native Python dates and times: datetime and dateutil Typed arrays of times: NumPy’s datetime64 Dates and times in Pandas: Best of both worlds Pandas Time Series: Indexing by Time Pandas Time Series Data Structures Regular sequences: pd.date_range() Frequencies and Offsets Resampling, Shifting, and Windowing Resampling and converting frequencies Time-shifts Rolling windows Where to Learn More Example: Visualizing Seattle Bicycle Counts Visualizing the data Digging into the data High-Performance Pandas: eval() and query() Motivating query() and eval(): Compound Expressions pandas.eval() for Efficient Operations Operations supported by pd.eval() Arithmetic operators Comparison operators Bitwise operators Object attributes and indices Other operations DataFrame.eval() for Column-Wise Operations Assignment in DataFrame.eval() Local variables in DataFrame.eval() DataFrame.query() Method Performance: When to Use These Functions Further Resources 4. Visualization with Matplotlib General Matplotlib Tips Importing matplotlib Setting Styles show() or No show()? How to Display Your Plots Plotting from a script Plotting from an IPython shell Plotting from an IPython notebook Saving Figures to File Two Interfaces for the Price of One MATLAB-style interface Object-oriented interface Simple Line Plots Adjusting the Plot: Line Colors and Styles Adjusting the Plot: Axes Limits Labeling Plots Simple Scatter Plots Scatter Plots with plt.plot Scatter Plots with plt.scatter plot Versus scatter: A Note on Efficiency Visualizing Errors Basic Errorbars Continuous Errors Density and Contour Plots Visualizing a Three-Dimensional Function Histograms, Binnings, and Density Two-Dimensional Histograms and Binnings plt.hist2d: Two-dimensional histogram plt.hexbin: Hexagonal binnings Kernel density estimation Customizing Plot Legends Choosing Elements for the Legend Legend for Size of Points Multiple Legends Customizing Colorbars Customizing Colorbars Choosing the colormap Color limits and extensions Discrete colorbars Example: Handwritten Digits Multiple Subplots plt.axes: Subplots by Hand plt.subplot: Simple Grids of Subplots plt.subplots: The Whole Grid in One Go plt.GridSpec: More Complicated Arrangements Text and Annotation Example: Effect of Holidays on US Births Transforms and Text Position Arrows and Annotation Customizing Ticks Major and Minor Ticks Hiding Ticks or Labels Reducing or Increasing the Number of Ticks Fancy Tick Formats Summary of Formatters and Locators Customizing Matplotlib: Configurations and Stylesheets Plot Customization by Hand Changing the Defaults: rcParams Stylesheets Default style FiveThirtyEight style ggplot Bayesian Methods for Hackers style Dark background Grayscale Seaborn style Three-Dimensional Plotting in Matplotlib Three-Dimensional Points and Lines Three-Dimensional Contour Plots Wireframes and Surface Plots Surface Triangulations