Advanced Bash-Scripting Guide
An in-depth exploration of the art of shell scripting
Mendel Cooper
<thegrendel.abs@gmail.com>
10
10 Mar 2014
Revision History
Revision 6.5 05 Apr 2012 Revised by: mc
'TUNGSTENBERRY' release
Revision 6.6 27 Nov 2012 Revised by: mc
'YTTERBIUMBERRY' release
Revision 10 10 Mar 2014 Revised by: mc
'PUBLICDOMAIN' release
This tutorial assumes no previous knowledge of scripting or programming, yet progresses rapidly toward an
intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX® wisdom and
lore. It serves as a textbook, a manual for self-study, and as a reference and source of knowledge on shell
scripting techniques. The exercises and heavily-commented examples invite active reader participation, under
the premise that the only way to really learn scripting is to write scripts.
This book is suitable for classroom use as a general introduction to programming concepts.
This document is herewith granted to the Public Domain. No copyright!
Dedication
For Anita, the source of all the magic
Table of Contents
Chapter 1. Shell Programming!.........................................................................................................................1
Chapter 2. Starting Off With a Sha-Bang........................................................................................................3
2.1. Invoking the script............................................................................................................................6
2.2. Preliminary Exercises.......................................................................................................................6
Part 2. Basics.......................................................................................................................................................7
Chapter 3. Special Characters...........................................................................................................................8
Chapter 4. Introduction to Variables and Parameters..................................................................................30
4.1. Variable Substitution......................................................................................................................30
4.2. Variable Assignment.......................................................................................................................33
4.3. Bash Variables Are Untyped..........................................................................................................34
4.4. Special Variable Types...................................................................................................................35
Chapter 5. Quoting...........................................................................................................................................41
5.1. Quoting Variables...........................................................................................................................41
5.2. Escaping..........................................................................................................................................43
Chapter 6. Exit and Exit Status.......................................................................................................................51
Chapter 7. Tests................................................................................................................................................54
7.1. Test Constructs...............................................................................................................................54
7.2. File test operators............................................................................................................................62
7.3. Other Comparison Operators..........................................................................................................65
7.4. Nested if/then Condition Tests.......................................................................................................70
7.5. Testing Your Knowledge of Tests..................................................................................................71
Chapter 8. Operations and Related Topics....................................................................................................72
8.1. Operators.........................................................................................................................................72
8.2. Numerical Constants.......................................................................................................................78
8.3. The Double-Parentheses Construct.................................................................................................80
8.4. Operator Precedence.......................................................................................................................81
Part 3. Beyond the Basics.................................................................................................................................84
Chapter 9. Another Look at Variables...........................................................................................................85
9.1. Internal Variables............................................................................................................................85
9.2. Typing variables: declare or typeset.............................................................................................104
9.2.1. Another use for declare.......................................................................................................107
9.3. $RANDOM: generate random integer..........................................................................................107
Chapter 10. Manipulating Variables.............................................................................................................119
10.1. Manipulating Strings...................................................................................................................119
10.1.1. Manipulating strings using awk........................................................................................127
10.1.2. Further Reference..............................................................................................................127
10.2. Parameter Substitution................................................................................................................128
Advanced Bash-Scripting Guide
i
Table of Contents
Chapter 11. Loops and Branches..................................................................................................................138
11.1. Loops..........................................................................................................................................138
11.2. Nested Loops..............................................................................................................................152
11.3. Loop Control...............................................................................................................................153
11.4. Testing and Branching................................................................................................................156
Chapter 12. Command Substitution.............................................................................................................165
Chapter 13. Arithmetic Expansion................................................................................................................171
Chapter 14. Recess Time................................................................................................................................172
Part 4. Commands..........................................................................................................................................173
Chapter 15. Internal Commands and Builtins.............................................................................................181
15.1. Job Control Commands..............................................................................................................210
Chapter 16. External Filters, Programs and Commands...........................................................................215
16.1. Basic Commands........................................................................................................................215
16.2. Complex Commands...................................................................................................................221
16.3. Time / Date Commands..............................................................................................................231
16.4. Text Processing Commands........................................................................................................235
16.5. File and Archiving Commands...................................................................................................258
16.6. Communications Commands......................................................................................................276
16.7. Terminal Control Commands.....................................................................................................291
16.8. Math Commands.........................................................................................................................292
16.9. Miscellaneous Commands..........................................................................................................303
Chapter 17. System and Administrative Commands..................................................................................318
17.1. Analyzing a System Script..........................................................................................................349
Part 5. Advanced Topics.................................................................................................................................351
Chapter 18. Regular Expressions..................................................................................................................353
18.1. A Brief Introduction to Regular Expressions..............................................................................353
18.2. Globbing.....................................................................................................................................357
Chapter 19. Here Documents.........................................................................................................................359
19.1. Here Strings................................................................................................................................369
Chapter 20. I/O Redirection...........................................................................................................................373
20.1. Using exec...................................................................................................................................376
20.2. Redirecting Code Blocks............................................................................................................379
20.3. Applications................................................................................................................................384
Chapter 21. Subshells.....................................................................................................................................386
Advanced Bash-Scripting Guide
ii
Table of Contents
Chapter 22. Restricted Shells.........................................................................................................................391
Chapter 23. Process Substitution...................................................................................................................393
Chapter 24. Functions....................................................................................................................................398
24.1. Complex Functions and Function Complexities.........................................................................402
24.2. Local Variables...........................................................................................................................413
24.2.1. Local variables and recursion............................................................................................414
24.3. Recursion Without Local Variables............................................................................................417
Chapter 25. Aliases.........................................................................................................................................420
Chapter 26. List Constructs...........................................................................................................................423
Chapter 27. Arrays.........................................................................................................................................427
Chapter 28. Indirect References....................................................................................................................456
Chapter 29. /dev and /proc.............................................................................................................................460
29.1. /dev..............................................................................................................................................460
29.2. /proc............................................................................................................................................463
Chapter 30. Network Programming.............................................................................................................469
Chapter 31. Of Zeros and Nulls.....................................................................................................................472
Chapter 32. Debugging...................................................................................................................................476
Chapter 33. Options........................................................................................................................................487
Chapter 34. Gotchas.......................................................................................................................................490
Chapter 35. Scripting With Style..................................................................................................................499
35.1. Unofficial Shell Scripting Stylesheet..........................................................................................499
Chapter 36. Miscellany...................................................................................................................................502
36.1. Interactive and non-interactive shells and scripts.......................................................................502
36.2. Shell Wrappers............................................................................................................................503
36.3. Tests and Comparisons: Alternatives..........................................................................................509
36.4. Recursion: a script calling itself..................................................................................................509
36.5. "Colorizing" Scripts....................................................................................................................512
36.6. Optimizations..............................................................................................................................525
36.7. Assorted Tips..............................................................................................................................528
36.7.1. Ideas for more powerful scripts.........................................................................................528
36.7.2. Widgets..............................................................................................................................539
36.8. Security Issues............................................................................................................................541
36.8.1. Infected Shell Scripts.........................................................................................................541
36.8.2. Hiding Shell Script Source................................................................................................541
Advanced Bash-Scripting Guide
iii
Table of Contents
Chapter 36. Miscellany
36.8.3. Writing Secure Shell Scripts.............................................................................................541
36.9. Portability Issues.........................................................................................................................541
36.9.1. A Test Suite.......................................................................................................................542
36.10. Shell Scripting Under Windows...............................................................................................543
Chapter 37. Bash, versions 2, 3, and 4..........................................................................................................544
37.1. Bash, version 2............................................................................................................................544
37.2. Bash, version 3............................................................................................................................548
37.2.1. Bash, version 3.1...............................................................................................................551
37.2.2. Bash, version 3.2...............................................................................................................552
37.3. Bash, version 4............................................................................................................................552
37.3.1. Bash, version 4.1...............................................................................................................559
37.3.2. Bash, version 4.2...............................................................................................................560
Chapter 38. Endnotes.....................................................................................................................................564
38.1. Author's Note..............................................................................................................................564
38.2. About the Author........................................................................................................................564
38.3. Where to Go For Help.................................................................................................................565
38.4. Tools Used to Produce This Book..............................................................................................565
38.4.1. Hardware...........................................................................................................................565
38.4.2. Software and Printware.....................................................................................................565
38.5. Credits.........................................................................................................................................566
38.6. Disclaimer...................................................................................................................................567
Bibliography....................................................................................................................................................569
Appendix A. Contributed Scripts..................................................................................................................577
Appendix B. Reference Cards........................................................................................................................787
Appendix C. A Sed and Awk Micro-Primer................................................................................................792
C.1. Sed................................................................................................................................................792
C.2. Awk..............................................................................................................................................795
Appendix D. Parsing and Managing Pathnames.........................................................................................798
Appendix E. Exit Codes With Special Meanings.........................................................................................802
Appendix F. A Detailed Introduction to I/O and I/O Redirection.............................................................803
Appendix G. Command-Line Options..........................................................................................................805
G.1. Standard Command-Line Options...............................................................................................805
G.2. Bash Command-Line Options......................................................................................................806
Appendix H. Important Files.........................................................................................................................808
Advanced Bash-Scripting Guide
iv
