Advanced Bash-Scripting Guide
An in-depth exploration of the art of shell scripting
Mendel Cooper
<thegrendel.abs@gmail.com>
6.3.
30 Apr 2011
Revision History
Revision 6.1 30 Sep 2009 Revised by: mc
'BUFFALOBERRY' release: Minor Update.
Revision 6.2 17 Mar 2010 Revised by: mc
'ROWANBERRY' release
Revision 6.3 27 Apr 2011 Revised by: mc
'SWOZZLEBERRY' release
This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an
intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX® wisdom and
lore. It serves as a textbook, a manual for self-study, and a reference and source of knowledge on shell
scripting techniques. The exercises and heavily-commented examples invite active reader participation, under
the premise that the only way to really learn scripting is to write scripts.
This book is suitable for classroom use as a general introduction to programming concepts.
Dedication
For Anita, the source of all the magic
Table of Contents
Chapter 1. Shell Programming!.........................................................................................................................1
Chapter 2. Starting Off With a Sha-Bang........................................................................................................3
2.1. Invoking the script............................................................................................................................6
2.2. Preliminary Exercises.......................................................................................................................6
Part 2. Basics.......................................................................................................................................................7
Chapter 3. Special Characters...........................................................................................................................8
Chapter 4. Introduction to Variables and Parameters..................................................................................29
4.1. Variable Substitution......................................................................................................................29
4.2. Variable Assignment.......................................................................................................................32
4.3. Bash Variables Are Untyped..........................................................................................................33
4.4. Special Variable Types...................................................................................................................34
Chapter 5. Quoting...........................................................................................................................................40
5.1. Quoting Variables...........................................................................................................................40
5.2. Escaping..........................................................................................................................................42
Chapter 6. Exit and Exit Status.......................................................................................................................50
Chapter 7. Tests................................................................................................................................................53
7.1. Test Constructs...............................................................................................................................53
7.2. File test operators............................................................................................................................60
7.3. Other Comparison Operators..........................................................................................................64
7.4. Nested if/then Condition Tests.......................................................................................................69
7.5. Testing Your Knowledge of Tests..................................................................................................70
Chapter 8. Operations and Related Topics....................................................................................................71
8.1. Operators.........................................................................................................................................71
8.2. Numerical Constants.......................................................................................................................77
8.3. The Double-Parentheses Construct.................................................................................................79
8.4. Operator Precedence.......................................................................................................................80
Part 3. Beyond the Basics.................................................................................................................................83
Chapter 9. Another Look at Variables...........................................................................................................84
9.1. Internal Variables............................................................................................................................84
9.2. Typing variables: declare or typeset.............................................................................................103
9.2.1. Another use for declare.......................................................................................................105
9.3. $RANDOM: generate random integer..........................................................................................105
Chapter 10. Manipulating Variables.............................................................................................................117
10.1. Manipulating Strings...................................................................................................................117
10.1.1. Manipulating strings using awk........................................................................................124
10.1.2. Further Reference..............................................................................................................125
10.2. Parameter Substitution................................................................................................................125
Advanced Bash-Scripting Guide
i
Table of Contents
Chapter 11. Loops and Branches..................................................................................................................135
11.1. Loops..........................................................................................................................................135
11.2. Nested Loops..............................................................................................................................148
11.3. Loop Control...............................................................................................................................149
11.4. Testing and Branching................................................................................................................152
Chapter 12. Command Substitution.............................................................................................................161
Chapter 13. Arithmetic Expansion................................................................................................................167
Chapter 14. Recess Time................................................................................................................................168
Part 4. Commands..........................................................................................................................................169
Chapter 15. Internal Commands and Builtins.............................................................................................177
15.1. Job Control Commands..............................................................................................................206
Chapter 16. External Filters, Programs and Commands...........................................................................211
16.1. Basic Commands........................................................................................................................211
16.2. Complex Commands...................................................................................................................216
16.3. Time / Date Commands..............................................................................................................227
16.4. Text Processing Commands........................................................................................................231
16.5. File and Archiving Commands...................................................................................................253
16.6. Communications Commands......................................................................................................271
16.7. Terminal Control Commands.....................................................................................................285
16.8. Math Commands.........................................................................................................................287
16.9. Miscellaneous Commands..........................................................................................................297
Chapter 17. System and Administrative Commands..................................................................................312
17.1. Analyzing a System Script..........................................................................................................341
Part 5. Advanced Topics.................................................................................................................................343
Chapter 18. Regular Expressions..................................................................................................................345
18.1. A Brief Introduction to Regular Expressions..............................................................................345
18.2. Globbing.....................................................................................................................................349
Chapter 19. Here Documents.........................................................................................................................351
19.1. Here Strings................................................................................................................................361
Chapter 20. I/O Redirection...........................................................................................................................365
20.1. Using exec...................................................................................................................................368
20.2. Redirecting Code Blocks............................................................................................................371
20.3. Applications................................................................................................................................376
Chapter 21. Subshells.....................................................................................................................................378
Advanced Bash-Scripting Guide
ii
Table of Contents
Chapter 22. Restricted Shells.........................................................................................................................383
Chapter 23. Process Substitution...................................................................................................................385
Chapter 24. Functions....................................................................................................................................390
24.1. Complex Functions and Function Complexities.........................................................................394
24.2. Local Variables...........................................................................................................................404
24.2.1. Local variables and recursion............................................................................................406
24.3. Recursion Without Local Variables............................................................................................408
Chapter 25. Aliases.........................................................................................................................................412
Chapter 26. List Constructs...........................................................................................................................415
Chapter 27. Arrays.........................................................................................................................................418
Chapter 28. Indirect References....................................................................................................................447
Chapter 29. /dev and /proc.............................................................................................................................451
29.1. /dev..............................................................................................................................................451
29.2. /proc............................................................................................................................................454
Chapter 30. Network Programming.............................................................................................................460
Chapter 31. Of Zeros and Nulls.....................................................................................................................463
Chapter 32. Debugging...................................................................................................................................467
Chapter 33. Options........................................................................................................................................477
Chapter 34. Gotchas.......................................................................................................................................480
Chapter 35. Scripting With Style..................................................................................................................489
35.1. Unofficial Shell Scripting Stylesheet..........................................................................................489
Chapter 36. Miscellany...................................................................................................................................492
36.1. Interactive and non-interactive shells and scripts.......................................................................492
36.2. Shell Wrappers............................................................................................................................493
36.3. Tests and Comparisons: Alternatives..........................................................................................498
36.4. Recursion: a script calling itself..................................................................................................499
36.5. "Colorizing" Scripts....................................................................................................................501
36.6. Optimizations..............................................................................................................................514
36.7. Assorted Tips..............................................................................................................................515
36.7.1. Ideas for more powerful scripts.........................................................................................515
36.7.2. Widgets..............................................................................................................................525
36.8. Security Issues............................................................................................................................527
36.8.1. Infected Shell Scripts.........................................................................................................527
36.8.2. Hiding Shell Script Source................................................................................................528
Advanced Bash-Scripting Guide
iii
Table of Contents
Chapter 36. Miscellany
36.8.3. Writing Secure Shell Scripts.............................................................................................528
36.9. Portability Issues.........................................................................................................................528
36.9.1. A Test Suite.......................................................................................................................529
36.10. Shell Scripting Under Windows...............................................................................................530
Chapter 37. Bash, versions 2, 3, and 4..........................................................................................................531
37.1. Bash, version 2............................................................................................................................531
37.2. Bash, version 3............................................................................................................................535
37.2.1. Bash, version 3.1...............................................................................................................538
37.2.2. Bash, version 3.2...............................................................................................................538
37.3. Bash, version 4............................................................................................................................539
37.3.1. Bash, version 4.1...............................................................................................................546
37.3.2. Bash, version 4.2...............................................................................................................547
Chapter 38. Endnotes.....................................................................................................................................550
38.1. Author's Note..............................................................................................................................550
38.2. About the Author........................................................................................................................550
38.3. Where to Go For Help.................................................................................................................550
38.4. Tools Used to Produce This Book..............................................................................................551
38.4.1. Hardware...........................................................................................................................551
38.4.2. Software and Printware.....................................................................................................551
38.5. Credits.........................................................................................................................................551
38.6. Disclaimer...................................................................................................................................553
Bibliography....................................................................................................................................................554
Appendix A. Contributed Scripts..................................................................................................................562
Appendix B. Reference Cards........................................................................................................................761
Appendix C. A Sed and Awk Micro-Primer................................................................................................766
C.1. Sed................................................................................................................................................766
C.2. Awk..............................................................................................................................................769
Appendix D. Exit Codes With Special Meanings.........................................................................................772
Appendix E. A Detailed Introduction to I/O and I/O Redirection.............................................................773
Appendix F. Command-Line Options...........................................................................................................775
F.1. Standard Command-Line Options................................................................................................775
F.2. Bash Command-Line Options......................................................................................................776
Appendix G. Important Files.........................................................................................................................778
Appendix H. Important System Directories.................................................................................................779
Advanced Bash-Scripting Guide
iv
