SQL query
Interoperating with RDDs
Inferring the schema using reflection
Programmatically specifying the schema
Querying with the DataFrame API
Number of rows
Running filter statements
Querying with SQL
Number of rows
Running filter statements using the where Clauses
DataFrame scenario – on-time flight performance
Preparing the source datasets
Joining flight performance and airports
Visualizing our flight-performance data
Spark Dataset API
Summary
4. Prepare Data for Modeling
Checking for duplicates, missing observations, and outliers
Duplicates
Missing observations
Outliers
Getting familiar with your data
Descriptive statistics
Correlations
Visualization
Histograms
Interactions between features
Summary
5. Introducing MLlib
Overview of the package
Loading and transforming the data
Getting to know your data
Descriptive statistics
Correlations
Statistical testing
Creating the final dataset
