Lessons Learned in Software Testing: A Context-
Driven Approach
byCem Kaner, James BachandBret Pettichord
John Wiley & Sons © 2002 (314 pages) Citation
ISBN:9780471081128
The ultimate resource for software testers, developers, and
managers at every level of expertise, this guidebook reveals
insights on how to do the job well, how to manage it, and how
to steer clear of common misunderstandings in software
testing.
Table of Contents
Lessons Learned in Software Testing—A Context-Driven Approach
Foreword
Preface
Chapter 1 - The Role of the Tester
Chapter 2 - Thinking Like a Tester
Chapter 3 - Testing Techniques
Chapter 4 - Bug Advocacy
Chapter 5 - Automating Testing
Chapter 6 - Documenting Testing
Chapter 7 - Interacting with Programmers
Chapter 8 - Managing the Testing Project
Chapter 9 - Managing the Testing Group
Chapter 10 - Your Career in Software Testing
Chapter 11 - Planning the Testing Strategy
The Context-Driven Approach to Software Testing
Bibliography
Index
List of Figures
List of Tables
List of Sidebars
1
Foreword
Imagine that you are holding a bottle of 50-year-old port. There is a way to drink port. It is not
the only way, but most folks who have enjoyed port for many years have found some
guidelines that help them maximize their port- drinking experience. Here are just a few:
Lesson 1: Don't drink straight from the bottle. If you have no glass, nor any other
possible containers available, pour a tiny amount of port into the palm of your hand
and sip from there. The port aroma should be smelled as you sip. Let the port swirl
over your tongue. Don't gulp down the port.
Lesson 2: Don't drink the entire bottle. If you are drinking because you are thirsty,
put down the port and drink a big glass of water. A small amount of port each time
maximizes the enjoyment of the entire bottle.
Lesson 3: Don't pollute the port. If someone tells you that you ought to try a new
cocktail made with orange juice, seawater, and port, politely refuse. With a big smile,
say, "But I would enjoy a glass of port."
Lesson 4: Don't covet the port. Hoarding your port means that you will never have
the pleasure of gentle conversation while sipping. Port is best shared with friends who
also enjoy a glass. Remember, they have a bottle somewhere, too.
You are not holding a bottle of port in your hands. You are holding Lessons Learned in
Software Testing, a very valuable book about software testing. This book has been ripening to
perfection throughout the 50 work-years of the authors' experiences. While port is for your
taste buds, this book is for your brain. I think that you will find any other differences to be
marginal. I have savored this book and offer you this advice to maximize your reading
experience.
Lesson 1. Don't drink straight from the bottle. Bring your own vessel to the reading of this
book. That is, bring all of your experiences in software development and testing. If you have
never participated in a serious software effort, this book will be too heady for you. It will leave
you confused and unable to operate heavy machinery for some time. If you are experienced,
savor the book's contents in the context of your own projects.
Lesson 2. Don't drink the entire bottle. Don't read this book in one sitting. Read a lesson or
two, close the book, and decide how you respond to the words of Messrs. Kaner, Bach, and
Pettichord. As you will discover, they call their approach to testing "context-driven." Only you
know the context of your own work. You must determine where there is a fit between a given
lesson and your particular work.
Lesson 3. Don't pollute the port. Somebody out there is going to make a list of the titles of
the 293 lessons contained in this book. Please don't let it be you. The heart of the book is the
explanation attached to each of the lessons. Be warned as well that someone else will
immediately try to ISO- ify or CMM-ummify the contents. I can see the article title now,
"Getting to CMM Level 293 using the 293 Lessons of Lessons Learned in Software Testing."
Argh! As the authors explain, "… we don't believe in ‘best practices.’ We believe some
2
practices are more useful than others under some circumstances." Written by pros, those
statements represent the quintessence of attaining the state of Master-crafter of Software
Testing.
Lesson 4. Don't covet the port. If there ever is a book to be read with your colleagues, this
is it. Buy a carton-load and hand a copy out to everyone who tests and to everyone who
thinks that she or he manages testers. Read a few selected lessons at a time and get
together to talk about them over coffee, lunch, even port! Read, ruminate, enjoy. Cheers!
Tim Lister
August 17, 2001
lister@acm.org
The Atlantic Systems Guild, Inc.
New York City
Preface
Overview
The Software Engineering Body of Knowledge (SWEBOK) is being proposed as a suitable
foundation for government licensing, for the regulation of software engineers, and for the
development of university curricula in software engineering. The SWEBOK document claims
to be consensus-based. One would expect such a document to carry the accumulated
knowledge and wisdom (the accumulated lessons) of the field.
Here is everything that SWEBOK has to say about exploratory testing:
Perhaps the most widely practiced technique remains ad hoc testing: tests are derived relying
on the tester skill and intuition ("exploratory" testing), and on his/her experience with similar
programs. While a more systematic approach is advised, ad hoc testing might be useful (but
only if the tester is really expert!) to identify special tests, not easily "captured" by formalized
techniques. Moreover it must be reminded that this technique may yield largely varying
degrees of effectiveness. (SWEBOK 0.95, 2001, 5-9)
How does SWEBOK treat what it acknowledges is the field's most widely practiced
technique? Nothing about how to practice the technique well. Only a statement that
exploration should be done only by real experts, that other approaches are advised, and the
suggestion that other, formalized techniques will yield less varying degrees of effectiveness.
Hah!
We don't pretend to offer a consensus document that describes our field's alleged body of
knowledge, but we do have a lot more to say about our field's most common practices. Rather
than dismissing exploratory testing, this book shows you what testing looks like through the
3
eyes of people who use exploratory methods (and many other methods) in a drive to achieve
excellent testing under real-life conditions.
Welcome to Lessons Learned in Software Testing
This book is about software development as we've experienced it. We've had a combined
total of 50 to 60 years of development experience (depending on how you count). We've been
at this for a while. We've seen a lot of great work and a lot of not-so-great work.
This book is not about how software engineering might be in a tidier and more controlled
world. We're writing about the world as we've worked in it.
In our world, software development teams often work under ambitious deadlines, discovering
what needs to be done at the same time as they're discovering how to do it. Sometimes their
approaches are more formal, sometimes less. It depends on a wide range of circumstances.
We follow the context-driven approach in software testing. We expect that a method that
works wonderfully under some circumstances will not work under others. Rather than talk
about best practices, we talk about practices that are well-suited to the context at hand. We
discuss the context-driven approach at the end of the book, but in essence, context-driven
testing looks at the "what" of testing (the techniques, tools, strategies, and so on) in terms of
"who," "when," "where," "why," and "what if."
Our goal is to match our selection of practices to the circumstances at hand in a way that
achieves excellent testing. We don't expect to achieve great testing by taking over the project,
nor by stamping our feet and telling the project manager (or executives) how Real
Professionals would run the project. We don't expect to achieve great testing by intimidating
programmers, nor by being obsequious to them. We don't expect to do great testing by filling
out thousands of little pieces of paper (or comparable electronic records), nor by wasting
everyone else's time on unnecessary processes.
We don't have a political or a bureaucratic or a formal-methods recipe for great testing.
That recipe doesn't exist!
We think great testing involves skillful technical work (searching for defects) and accurate,
persuasive communication.
The skilled search is always exploratory. There's an infinite amount of testing to do, and a tiny
amount of time in which to do just a little bit of that infinity. Every test we do, every document
we write, every meeting we attend takes time away from running other tests that could expose
a key defect. Facing that constraint, we optimize our testing processes so that they take
advantage of our ever-growing knowledge of the product, its market, its applications, and its
weaknesses. What we learn today gets reflected in more powerful tests tomorrow.
Even if:
4
the product is well specified, and
the specification accurately reflects a requirements document, and
the requirements document accurately represents the actual needs of the product's
stakeholders
(have you ever been on a project where all these "even if's" were true?); we will still learn a lot
about how to test the product while we are testing it. In particular, as we discover errors, we
learn how this group of programmers can go wrong. The specifications tell us about how the
program is supposed to work when it is coded correctly. They don't tell us what mistakes to
anticipate, nor how to design tests to find them. At that task, our key task, we get better from
the start to the end of the project, on every project.
No matter what it looks like from the outside, whenever we are testing with our brains
engaged, our work is exploratory.
Who This Book Is For
This book is for anyone who tests software, anyone who manages testers, and anyone who
has to deal with testers in their software development projects. That includes project
managers and executives.
The primary reader that we have in mind, the "you" in this book, has been testing for a few
years and might recently have been promoted to a supervisory role. We hope that you'll see a
lot in this book that matches your experience, that you'll gain new insights from our lessons,
that you'll find lessons that are useful for quoting to your manager, that you'll like a few of our
statements so much that you'll photocopy them and stick them on the outside of your cubicle,
and maybe that you'll react to at least one statement so strongly that you'll stick a copy on the
middle of your dart board. (We want to stimulate thought, not only agreement.)
Newcomers to testing (and those of you who are just applying for a testing job) won't have as
many occasions to feel that you've already experienced what we're writing about. For you,
this book might provide some early insights and warnings, giving you a good taste of what
issues testers face.
Hint: If you are absolutely new to testing and you are looking for a book to study from in order
to prepare for a job interview, this is the wrong book. If this is the only book available to you,
pay your most careful attention to the chapters on "Testing Techniques" and "Your Career in
Software Testing." If you have a choice of books, we suggest the first five chapters of Testing
Computer Software (Kaner et al., 1993).
Programmers, project managers, and executives who have to work with testers will find this
book a useful source of ideas for setting your expectations of the testing group. We hope that
it will help you evaluate and discuss your concerns with the testing group if you disagree with
their policies or feel that they are spending their time unwisely.
5
