Linux−HA Heartbeat System Design
Alan Robertson − SuSE Labs − <alanr@suse.com>
ABSTRACT
One of the most commonly identified features which is felt to be necessary for
Linux
TM
to be considered "enterprise−ready" is High−Availability. High−Availabil−
ity (HA) systems provide increased service availability through clustering techniques.
HA clusters minimize availability interruptions by quickly switching services over
from failed systems to working systems, providing the customer with an illusion of con−
tinuous availability. As such, high−availability features, are vital to mission−critical
systems. Although there are many components to a high−availability system, two of the
key components are heartbeat services and cluster communication services. Heartbeat
services provide notification of when nodes are working, and when they fail. In the
Linux−HA project, the
heartbeat program provides these services and intracluster com−
munication services.
This paper describes the design of the heartbeat program which is part of the
High−Availability Linux Project with particular emphasis on the rationales behind key
design choices, and the results obtained.
Introduction
As Linux
TM1
grows into handling larger server sys−
tems satisfactorily, it will have to provide many of the
same services which these larger servers by Sun,
Compaq, IBM, and others have traditionally provided.
One of the key features which these larger and more
mission−critical servers have provided customers is
high−availability (HA) clustering.
A high−availability cluster is a group of computers
which work together in such a way that the failure of
any single node in the cluster will not cause the serv−
ice to become unavailable. Given this definition, it
seems obvious that it is necessary for the cluster to de−
tect when servers fail, and when they become
available again. This task is performed by code which
is usually called "heartbeat" code. In the case of
Linux−HA, this function is performed by a program
called heartbeat. Heartbeat programs typically send
packets to each machine in the cluster to indicate that
they are still alive.
Another of the most basic functions which any
High−Availability system must perform is cluster
communications. It is often the case that these com−
munications need to communicate between all cluster
members at once in a broadcast or multicast sense.
The Linux−HA heartbeat program takes the ap−
proach that the keepalive messages which it sends are
a specific case of the more general cluster communi−
cations service. In this sense, it treats cluster
membership as joining the communication channel,
1 Linux is a trademark of Linus Torvalds.
and leaving the cluster communication channel as
leaving the cluster. Because of this, the heartbeat
messages which are its namesake are almost a side−
effect of cluster communications, rather than a sepa−
rate standalone facility in the heartbeat program. It
should be emphasized that heartbeat should not be
understood as a complete cluster management solu−
tion, but a basic component providing certain well−
defined low−level services. These services are out−
lined in more detail below.
Heartbeat Design Philosophy
The heartbeat component of the Linux−HA project
[Rob00] is in some senses a simple program. It is one
of the the lowest−level components of the system, and
has the purpose of being reliable, so it is important that
it be simple and straightforward. It should be designed
to run continuously for years without memory leaks, or
bugs. It needs to be easy to understand, easy to debug,
and extremely robust. For this reason, when design
alternatives were considered, the simplest, most
straightforward, and easiest to debug were often cho−
sen.
Even though this low level subsystem is reasonably
simple, there are some non−obvious design decisions
and synergies which were made which appear to be
worth understanding. It is the intent of this paper to
explore some of these elements of the design, and talk
about how it may be extended in the future.