CSS transforms, CSS transitions, and CSS animations are three separate CSS specifi‐ cations. While the three terms sound like they may do the same thing—make some‐ thing move—CSS transitions and animations make things move over time. Transitions and animations let you define the transition between two or more states of an element.
Note that for the generic terms spark and graphs we had to substitute the overly spe- cific Apache Spark and edges and vertices, but the trends can still be seen. A couple of these technologies, machine learning and graphs, have long histories within academic computer science and are attracting new interest in the commercial realm as the avail- ability of Big Data is now mainstreaming these technologies. If you studied these tech- nologies in school as theory, the world is ready now for you to put them into practice.
A lot of companies, including the ones we work for and have worked for in the past, have put Spark—though not necessarily GraphX—into production. This makes it more than just a little convenient when embarking on prototyping graph solutions to try GraphX first. If you have a Spark cluster already, or if you decide to spin up a Spark cluster in the cloud, such as with Databricks or Amazon, you can get started with graphs without having to set up a new graph-specific cluster or technology, and you can use your Spark skills in the GraphX API. As more and more applications of graphs hit the newsstands—from rooting out terrorist networks on Twitter to fraud detection in credit card transaction data—GraphX becomes an easy platform choice for trying them out.
In this book, we simultaneously take on two ambitious goals: to cover everything possible about Spark GraphX, and to assume little to no expertise about any of the technologies represented by the aforementioned buzzwords. The biggest challenge was the hefty amount of prerequisites to get into GraphX—specifically, Spark, Scala, and graphs. Other challenges were the extensive GraphX API and the many different ways graphs can be used. The result is an In Action book that differs a bit from others: it takes a while to get started, with the first five chapters laying the groundwork, and there are a number of interesting examples rather than one that gradually gets built up over the course of the book. In books about other technologies the reader might come with a problem to solve; this book attempts to demystify graphs by showing pre- cisely what problems graphs can solve. And it does so without assuming a lot of back- ground knowledge and experience.
This book is meant for developers who have worked in object-oriented languages such as Java, Ruby, or Python and are interested in improving their craft by learning Scala. Java developers will recognize the core object-oriented, static typing and generic col‐ lections in Scala. However, they may be challenged to switch to Scala’s more expressive and flexible syntax, and the use of immutable data and function literals to solve prob‐ lems. Ruby and Python developers will be familiar with the use of function literals (aka closures or blocks) to work with collections, but may be challenged with its static, generic-supporting type system.
For these and any other developers who want to learn how to develop in the Scala programming language, this book provides an organized and examples-based guide that follows a gradual learning curve.
This book is here to help you get familiar with Apache Kafka and to solve your challenges related to the consumption of millions of messages in publisher-subscriber architectures. It is aimed at getting you started programming with Kafka so that you will have a solid foundation to dive deep into different types of implementations and integrations for Kafka producers and consumers.
In addition to an explanation of Apache Kafka, we also spend a chapter exploring Kafka integration with other technologies such as Apache Hadoop and Apache Storm. Our goal is to give you an understanding not just of what Apache Kafka is, but also how to use it as a part of your broader technical infrastructure. In the end, we will walk you through operationalizing Kafka where we will also talk about administration.
Big data is getting bigger and bigger day by day. And I don't mean tera, peta,
exa, zetta, and yotta bytes of data collected all over the world every day. I refer to complexity and number of components utilized in any decent and respectable big data ecosystem. Never mind the technical nitties gritties—just keeping up with terminologies, new buzzwords, and hypes popping up all the time can be a real challenge in itself. By the time you have mastered them all, and put your hard- earned knowledge to practice, you will discover that half of them are old and inef cient, and nobody uses them anymore. Spark is not one of those "here today, gone tomorrow" fads. Spark is here to stay with us for the foreseeable future, and it is well worth to get your teeth into it in order to get some value out of your data NOW, rather than in some, errr, unforeseeable future. Spark and the technologies built on top of it are the next crucial step in the big data evolution. They offer 100x faster in-memory, and 10x on disk processing speeds in comparison to the traditional Hadoop jobs.
There's no better way of getting to know Spark than by reading this book, written
by Mike Frampton, a colleague of mine, whom I rst met many, many years ago and have kept in touch ever since. Mike's main professional interest has always been data and in pre-big data days, he worked on data warehousing, processing, and analyzing projects for major corporations. He experienced the inef ciencies, poor value, and frustrations that the traditional methodologies of crunching the data offer rst hand. So understanding big data, what it offers, where it is coming from, and where it is heading, and is intrinsically intuitive to him. Mike wholeheartedly embraced big data the moment it arrived, and has been devoted to it ever since. He practices what he preaches, and is not in it for money. He is very active in the big data community, writes books, produces presentations on SlideShare and YouTube, and is always rst to test-drive the new, emerging products.
Mike's passion for big data, as you will nd out, is highly infectious, and he
is always one step ahead, exploring the new and innovative ways big data is
used for. No wonder that in this book, he will teach you how to use Spark in conjunction with the very latest technologies; some of them are still in development stage, such as machine learning and Neural Network. But fear not, Mike will carefully guide you step by step, ensuring that you will have a direct, personal experience of the power and usefulness of these technologies, and are able to put them in practice immediately.
One million Uber rides are booked every day, 10 billion hours of Netflix videos are watched every month, and
$1 trillion are spent on e-commerce web sites every year. The success of these services is underpinned by Big Data and increasingly, real-time analytics. Real-time analytics enable practitioners to put their fingers on the pulse
of consumers and incorporate their wants into critical business decisions. We have only touched the tip of the iceberg so far. Fifty billion devices will be connected to the Internet within the next decade, from smartphones, desktops, and cars to jet engines, refrigerators, and even your kitchen sink. The future is data, and it is becoming increasingly real-time. Now is the right time to ride that wave, and this book will turn you into a pro.
The low-latency stipulation of streaming applications, along with requirements they share with general Big Data systems—scalability, fault-tolerance, and reliability—have led to a new breed of real- time computation. At the vanguard of this movement is Spark Streaming, which treats stream processing as discrete microbatch processing. This enables low-latency computation while retaining the scalability and fault-tolerance properties of Spark along with its simple programming model. In addition, this gives streaming applications access to the wider ecosystem of Spark libraries including Spark SQL, MLlib, SparkR, and GraphX. Moreover, programmers can blend stream processing with batch processing to create applications that use data at rest as well as data in motion. Finally, these applications can use out-of-the- box integrations with other systems such as Kafka, Flume, HBase, and Cassandra. All of these features have turned Spark Streaming into the Swiss Army Knife of real-time Big Data processing. Throughout this book, you will exercise this knife to carve up problems from a number of domains and industries.
This book takes a use-case-first approach: each chapter is dedicated to a particular industry vertical. Real-time Big Data problems from that field are used to drive the discussion and illustrate concepts from Spark Streaming and stream processing in general. Going a step further, a publicly available dataset from that field is used to implement real-world applications in each chapter. In addition, all snippets of code
are ready to be executed. To simplify this process, the code is available online, both on GitHub1 and on the publisher’s web site. Everything in this book is real: real examples, real applications, real data, and real code. The best way to follow the flow of the book is to set up an environment, download the data, and run the applications as you go along. This will give you a taste for these real-world problems and their solutions.
These are exciting times for Spark Streaming and Spark in general. Spark has become the largest open source Big Data processing project in the world, with more than 750 contributors who represent more than
200 organizations. The Spark codebase is rapidly evolving, with almost daily performance improvements and feature additions. For instance, Project Tungsten (first cut in Spark 1.4) has improved the performance of the underlying engine by many orders of magnitude. When I first started writing the book, the latest version of Spark was 1.4. Since then, there have been two more major releases of Spark (1.5 and 1.6). The changes in these releases have included native memory management, more algorithms in MLlib, support for deep learning via TensorFlow, the Dataset API, and session management. On the Spark Streaming front, two major features have been added: mapWithState to maintain state across batches and using back pressure to throttle the input rate in case of queue buildup.2 In addition, managed Spark cloud offerings from the likes of Google, Databricks, and IBM have lowered the barrier to entry for developing and running Spark applications.
Now get ready to add some “Spark” to your skillset!
