Stories
Slash Boxes
Comments

SoylentNews is people

posted by LaminatorX on Sunday October 12 2014, @09:19AM   Printer-friendly
from the Bad-Ass-Script-Host dept.

When I first learned about Linux in the 90’s, I read that it was possible to even write your own commands to use at the command line. Later I learned about bash scripting, and it wasn’t long before I needed to learn how to loop in bash. Looping in bash is one of the fundamental building blocks of bash programming. It isn’t hard to do at all and is worth learning. The main reason to learn looping in bash is to handle doing the same thing over and over again. They’re easy to do even at the command line. Please follow along as we look a couple of basic examples, and how you can expand on them.

http://www.tidbitsfortechs.com/2014/10/looping-in-bash/

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2, Insightful) by Rosco P. Coltrane on Sunday October 12 2014, @09:53AM

    by Rosco P. Coltrane (4757) on Sunday October 12 2014, @09:53AM (#105008)

    Not to nitpick or anything - it's great that people discover new and exciting things - but how is this news, let alone interesting news? This is shell scripting 101...

    I was expecting an article on loopHOLES in Bash, not an introductory course on basic algorithmics.

    • (Score: 0) by Anonymous Coward on Sunday October 12 2014, @10:34AM

      by Anonymous Coward on Sunday October 12 2014, @10:34AM (#105016)

      its interesting and nerdy, and news shouldn't just be about sensationalized bs... there's plenty of other sites for that

      SN is about community as well as news, and sharing tips and tricks, reviews, asking questions etc are all perfect for engaging discussion

      everyone is welcome to also share their opinion of course, which is also great

    • (Score: 0) by Anonymous Coward on Sunday October 12 2014, @10:41AM

      by Anonymous Coward on Sunday October 12 2014, @10:41AM (#105017)

      Not to nitpick or anything - it's great that people discover new and exciting things - but how is this news, let alone interesting news? This is shell scripting 101...

      Indeed. LaminatorX should have done a little bit of research before posting this "news story". A simple google search for "bash script" would have turned up millions of hits. This is not news. Not even close.

      • (Score: 2) by theluggage on Sunday October 12 2014, @11:07AM

        by theluggage (1797) on Sunday October 12 2014, @11:07AM (#105023)

        This is not news. Not even close

        But... "bash" is a trending keyword at the moment, and surely the language every l33t haX0r wants to learn right now. I hope part 2 of this article covers how to embed loops in environment variable declarations.

        P.S. Heartbleed had a logo. Where's the shellshock logo? I thought every security vulnerability got a snappy name and a logo now?

        • (Score: 0) by Anonymous Coward on Sunday October 12 2014, @02:38PM

          by Anonymous Coward on Sunday October 12 2014, @02:38PM (#105057)

          Here you go... [twimg.com]

        • (Score: 0) by Anonymous Coward on Monday October 13 2014, @02:08PM

          by Anonymous Coward on Monday October 13 2014, @02:08PM (#105557)
          Given the security and code quality problems with bash perhaps we should start learning to write stuff so that it's not bash specific.
      • (Score: 2, Insightful) by Anonoob on Sunday October 12 2014, @12:11PM

        by Anonoob (335) on Sunday October 12 2014, @12:11PM (#105034)

        The 'millions of hits' itself is important point here - even the few comments so far have given those of us in the 101 camp some insights, that may not be immediately resolved fishing through those hits. Can readily agree it isn't news, but who really comes to SN for the articles. Sure I can go learn this on other sites, but the opinions it was basic non-news actually peaked my interest, while later comments on different techinques were something the article itself did not go into and shows the true value of this site.
        So I for one, would appreciate this kind of article being posted now and then.

        • (Score: 2) by Phoenix666 on Monday October 13 2014, @01:38PM

          by Phoenix666 (552) on Monday October 13 2014, @01:38PM (#105539) Journal

          while later comments on different techinques were something the article itself did not go into and shows the true value of this site.

          I second this, too. Most of the time when you don't fish in certain waters all the time you chug through a tutorial to get what you need, and move on. This kind of focused feedback gives very valuable context to scripting in general that you would be hard pressed to get any way other than by sitting among the greybeards in the dark rooms for several years.

          --
          Washington DC delenda est.
    • (Score: 5, Insightful) by BsAtHome on Sunday October 12 2014, @11:22AM

      by BsAtHome (889) on Sunday October 12 2014, @11:22AM (#105026)

      While I can relate to the sentiment of this not being news, this site should welcome diversity. Please remember that you also needed to learn the basics at one point...

      The article surely fits into the category "Stuff that matters". New users want to be appreciated too and certainly not mocked for not knowing. They want to learn, you have also learned it and others need to be brought into the breach. We all should politely help each other at the level we can understand and operate. There is no point at making this an exclusive site if you want to reach a broad target audience.

      Just my 2 (euro) cents.

    • (Score: 4, Informative) by maxwell demon on Sunday October 12 2014, @12:15PM

      by maxwell demon (1608) on Sunday October 12 2014, @12:15PM (#105035) Journal

      Indeed, it is not even good as tutorial, as it does several things not advisable.

      First, the constructs he uses are just regular Bourne shell constructs, yet he uses #!/bin/bash as shebang line. Doing so unnecessarily restricts portability (and since the Shellshock bug should also be avoided for security reasons). He should use #!/bin/sh for portability.

      Next, it uses backticks instead of $() for command substitution. Unless you need to run your code on very ancient shells, don't do that. It's too easy to get wrong, and then get unexpected behaviour (a long time ago, I once even inadvertently created a fork bomb this way — I just wanted to output a lot of information from the script, including the script's own name, and an error in the backticks actually caused one of the $0 directly following an opening backtick (which was intended to be a closing one) ...

      Next, he is using variables completely without quotes. That's just asking for disaster when any name has a space inside. Now you may argue that the for construct he uses won't work of he puts $names in quotes. But then, the whole idea of storing the output of cat in a variable via command substitution. The right loop to use here would be a while loop using read to get the names from the file.

      Also, why does he use /bin/true instead of simply true, although he uses cat and not /bin/cat? Obviously /bin is in his path, or cat would not be found. Not to mention that bash provides a built-in true which almost certainly is more efficient to use that /bin/true.

      --
      The Tao of math: The numbers you can count are not the real numbers.
      • (Score: 3, Informative) by toygeek on Sunday October 12 2014, @08:46PM

        by toygeek (28) on Sunday October 12 2014, @08:46PM (#105256) Homepage

        OP here. I can address all of your concerns with two words: Simplicity, Habit. Thanks for reading :)

        --
        There is no Sig. Okay, maybe a short one. http://miscdotgeek.com
    • (Score: 2) by Phoenix666 on Sunday October 12 2014, @12:51PM

      by Phoenix666 (552) on Sunday October 12 2014, @12:51PM (#105040) Journal

      I welcome this kind of thing because bringing back an element of the old LUGs is a good thing. It sets the site apart from other mindless aggregators. Also, when it comes to technology it's always good to get refreshers, even on basic things. Let's say you haven't done any bash scripting in, oh, ten years. It's good to see it again, and it happens also not infrequently that you can learn something new, some trick, that you never knew but wish you had all this time.

      A couple of years ago I had some down time and read a tutorial somewhere about vim scripting. I had once dipped my toe in but since I didn't have a particular use for it at the time never went any further and didn't retain anything. But now I find its use makes me three times more productive and that's awesome.

      --
      Washington DC delenda est.
  • (Score: 3, Interesting) by No.Limit on Sunday October 12 2014, @10:29AM

    by No.Limit (1965) on Sunday October 12 2014, @10:29AM (#105012)

    Just a few things that caught my eye:

    • The examples always use the shebang [wikipedia.org] '#!/bin/bash', whereas it's totally sufficient to use '#!/bin/sh'. bash offers much more functionality over the normal shell, but that's mainly ui functionality. The normal shell is totally sufficient, but much simpler (and thus might leave a smaller attack vector *cough*shellshock*cough*). You could also go for '#!/usr/bin/env sh' because of "portability" [wikipedia.org].
    • Backticks "`" are nice for using embedded shells, but for embedded shells '$(CODE_HERE)' works too and allows for nesting which comes in useful sometimes.
    • For the last example one could also use the 'watch' command which would be much more elegant.

    And as has been said before this isn't really news, but rather some shell scripting tutorial for looping.

    • (Score: 2) by ticho on Sunday October 12 2014, @10:42AM

      by ticho (89) on Sunday October 12 2014, @10:42AM (#105018) Homepage Journal

      I might be remembering this wrong, but aren't backticks considered obsolete, and to-be-replaced by $(code) construct?

      • (Score: 2) by tynin on Sunday October 12 2014, @10:51AM

        by tynin (2013) on Sunday October 12 2014, @10:51AM (#105019) Journal

        Yeah backticks and also seq are considered obsolete but continue to work. But having used them for so long my fingers kept typing them out for doing for loops for cluster management (e.g. for x in `seq 200`; do ssh 10.0.0.$x "hostname"; done). You save yourself having to type a couple extra characters, granted not many. Only recently did one of my coworkers write this simple yet powerful parallel ssh tool that I've stopped bothering with for loops.

    • (Score: 2) by FatPhil on Sunday October 12 2014, @11:41AM

      by FatPhil (863) <pc-soylentNO@SPAMasdf.fi> on Sunday October 12 2014, @11:41AM (#105029) Homepage
      His examples were also not able to cope with names that had spaces in. All $IFS's look the same to a "for x in ...".
      --
      Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
    • (Score: 1) by theCoder on Sunday October 12 2014, @12:42PM

      by theCoder (3583) on Sunday October 12 2014, @12:42PM (#105039)

      I agree that /bin/sh is preferable to /bin/bash. But to your second point, I do not think the $() syntax is part of the original Bourne shell, and thus might not be as supported by the /bin/sh interpreter. A quick test shows that dash (the /bin/sh interpreter on my Mint system) does support it. And RedHat style systems that use bash as /bin/sh will probably allow it, too. So that likely covers most Linux systems, but others like Solaris and the BSDs might have different compatibility. See Portability Issues [tldp.org] for more information and other features that you shouldn't rely on when using /bin/sh.

      • (Score: 3, Informative) by fnj on Sunday October 12 2014, @04:53PM

        by fnj (1654) on Sunday October 12 2014, @04:53PM (#105090)

        You'll have a tough time finding an sh that does not support $(). BSDs and Solaris have supported it for a long time. It's definitely part of POSIX 1003.1. Having said that, it wasn't part of pre-POSIX Bourne sh. It pleases me to limit my scripts to the syntax of the Heirloom Bourne shell [sourceforge.net], which represents state of the art for SVR4, circa 1989. It is easy to install anywhere; I have it as /usr/local/bin/bournesh on every system. Any script you develop and test with it is going to work with any shell you can find on any running system, period. (Your portability is going to end up depending on the vintage of your /bin utilities - it never ends!). But no one would ever use it as an interactive shell. I mean you CAN and it works fine, but with no command line editing or recall whatsoever, you're going to tear your hair out and bash your keyboard to splinters.

        My policy is to never write a bash script; never start with #!/bin/bash. Bash is a terrific interactive shell, but you just don't need $(), (()), [[]], variable arrays, associative arrays, etc. in your scripts. You DEFINITELY don't need "function foo { }". "foo ( ) { }" works fine.

        You can even find the source for V7 (1979) sh [collyer.net] modified to compile on modern systems. That was just too primitive IMHO. It didn't have the # comment character; no functions at all; no unset; no set --; no [] (you have to use test); no ! test negation.

        Or if that is not primitive enough, how about V6 (1975) Thompson sh [v6shell.org]? I'll let you experiment with it on your own, but I'll just say you ARE going to find a lot of missing stuff.

        For that matter, you could try to resurrect the V1 (1971) sh for yourself if you just want to lose your sanity.

        The whole fascinating history [in-ulm.de] makes for a good read.

        It's a discipline. Everyone can decide the portability/readability/reliability/productibity tradeoffs for himself.

      • (Score: 3, Informative) by maxwell demon on Sunday October 12 2014, @05:11PM

        by maxwell demon (1608) on Sunday October 12 2014, @05:11PM (#105099) Journal

        Given that the $() syntax is part of POSIX, I guess unless you need to target very old systems, there's no reason to use backticks.

        Note that on some very old systems, the usual shebang line may not work either, since some old systems checked for the four-byte sequence "#! /", so they would not recognize the commonly found

        #!/bin/sh

        but only

        #! /bin/sh

        --
        The Tao of math: The numbers you can count are not the real numbers.
        • (Score: 1) by theCoder on Sunday October 12 2014, @09:02PM

          by theCoder (3583) on Sunday October 12 2014, @09:02PM (#105264)

          Good to know. I didn't realize support was so widespread and standard. I guess I was burned on some older (maybe Solaris 8?) machines, assumed it was a bash feature only, and never tried again.

    • (Score: 2) by JoeMerchant on Sunday October 12 2014, @02:27PM

      by JoeMerchant (3937) on Sunday October 12 2014, @02:27PM (#105054)

      I'd say the sh bash choice is just that, a choice. If you are writing to bash and you intend to execute in bash, then use bash. If you only need sh, then go for sh.

      What I'd rather work with is a system that uses one, consistently, instead of sh in this file, bash in the next, and csh every so often just to drive you nuts.

      --
      🌻🌻 [google.com]
      • (Score: 2) by fnj on Sunday October 12 2014, @05:01PM

        by fnj (1654) on Sunday October 12 2014, @05:01PM (#105094)

        I'm not religiously against anybody scripting in bash as against sh (I limit my own shell scripts to sh, but I'm not a fascist).

        But you have to ask yourself, given the power of bash and the accompanying bulk, when you get into heavier duty stuff like array variables, associative arrays, shell math, etc. - why not just make the jump to perl or python and do it right?

  • (Score: 2) by The Mighty Buzzard on Sunday October 12 2014, @10:33AM

    by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Sunday October 12 2014, @10:33AM (#105015) Homepage Journal

    It's going on 2015, shouldn't we be using <names.txt or at least $(cat names.txt) by now?

    --
    My rights don't end where your fear begins.
    • (Score: 1) by b on Sunday October 12 2014, @10:59AM

      by b (2121) on Sunday October 12 2014, @10:59AM (#105021)

      Yeah, I thought this felt like useless use of cat too, but I'm not sure how it could be improved. I'm not sure how to remove cat from "for i in $(cat foo.txt)".

      (FWIW I'd use "while read -r" myself anyway.)

      • (Score: 2) by Foobar Bazbot on Monday October 13 2014, @01:00AM

        by Foobar Bazbot (37) on Monday October 13 2014, @01:00AM (#105367) Journal

        9 times out of 10 (i.e. unless one really means to allow multiple iterations per line), the right way to remove cat from "for i in $(cat foo.txt)" is to remove the whole line.

        (FWIW I'd use "while read -r" myself anyway.)

        You and me both.

        • (Score: 1) by b on Monday October 13 2014, @11:42PM

          by b (2121) on Monday October 13 2014, @11:42PM (#105758)
          Agreed. FWIW I worked out better syntax for those occasions when you do want to iterate over words.

          for i in $(<foo); do echo $i; done

  • (Score: 2) by VLM on Sunday October 12 2014, @11:12AM

    by VLM (445) Subscriber Badge on Sunday October 12 2014, @11:12AM (#105024)

    There's a minor problem in the article... by trying to start too simply, other better simpler solutions are available.

    xargs is for more than screwing around with directories with 100K (quantity not length) files.

    Several of the early examples boil down to something equivalent to

    ls | xargs wc -l

    Another interesting strategy not considered is some simple report-ish tasks are very amenable to awk and awk is fast, faster than Perl (and if you've got a truly awful task, whip out a Perl one liner)

    • (Score: 2) by FatPhil on Sunday October 12 2014, @11:43AM

      by FatPhil (863) <pc-soylentNO@SPAMasdf.fi> on Sunday October 12 2014, @11:43AM (#105030) Homepage
      I don't awk, but I do notice that lots of awk one-liners I see ought to be sed-able, and I do sed.
      --
      Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
      • (Score: 0) by Anonymous Coward on Sunday October 12 2014, @12:08PM

        by Anonymous Coward on Sunday October 12 2014, @12:08PM (#105033)

        I think that it would be more correct to state that all sed programs can be implemented in awk. Not the other way around, however for most one-liners I probably agree. AWK is actually a full featured programming language that can do advanced math, has arrays, flow control, file I/O, the works. You can implement most things like bc, sed, grep, in awk.

        • (Score: 2) by VLM on Sunday October 12 2014, @12:27PM

          by VLM (445) Subscriber Badge on Sunday October 12 2014, @12:27PM (#105037)

          "advanced math"

          Well that might be a little optimistic, but awk is pretty awesome at the fairly typical tasks of "gimme the MAX, MIN, AVG, STD DEV" of some set of ... something.

          Reminds me of the project I did some years back importing a medium size engineering dataset into R, then daydreaming of the infinite number of cool things in R, and then nobody wanted anything other than avg, I couldda just left it all in awk...

  • (Score: 2) by Snotnose on Sunday October 12 2014, @01:30PM

    by Snotnose (1623) on Sunday October 12 2014, @01:30PM (#105047)

    is a book with many more examples. My copy has to be 20 years old, wonder if it's still in print?

    --
    Why shouldn't we judge a book by it's cover? It's got the author, title, and a summary of what the book's about.
  • (Score: 3, Informative) by PizzaRollPlinkett on Sunday October 12 2014, @03:42PM

    by PizzaRollPlinkett (4512) on Sunday October 12 2014, @03:42PM (#105075)

    I recommend O'Reilly's "Classic Shell Scripting" book. I usually have to do bash scripting every few years when someone needs a shell script for some reason and doesn't want a Perl or Python program. If you're used to real programming languages, sh and bash are badly designed and confusing and I never really remember them. (I don't care what you say about Perl, bash has words spelled backwards - did Zatanna design it? The syntax looks like a junk drawer of random stuff taken from every programming language in the 70s. Even REXX makes more sense, and some of the commands have open parens without close parens.) The first thing I do is pull down this book and try to read up on what I need to know.

    I have a sample somewhere of looping through an array that I always dig up because it's not obvious. But it's not the hardest thing to remember. There's a "for scalar in array" construct, but the hard part for me has always been getting something into an array.

    Anyhow, bugs or not, bash is harmful to my brain and I try to use a real programming language for any script I possibly can.

    [!si annataZ ohw wonk uoy epoh I]

    --
    (E-mail me if you want a pizza roll!)
    • (Score: 0) by Anonymous Coward on Sunday October 12 2014, @04:10PM

      by Anonymous Coward on Sunday October 12 2014, @04:10PM (#105081)

      O'Reilly's

      You might want to read this http://thebaffler.com/articles/the-meme-hustler [thebaffler.com]

    • (Score: 2, Interesting) by hendrikboom on Sunday October 12 2014, @06:31PM

      by hendrikboom (1125) Subscriber Badge on Sunday October 12 2014, @06:31PM (#105170) Homepage Journal

      The backwards words are an Algol 68 legacy, and are close brackets that match the correcponding forward-spelled words. The variety of brackets makes it easy to detect, diagnose, and correct missing brackets. I contrast this with C's '}', Pascal's ubiquitous 'END' and Lisp's even more irritating ')'.

      -- hendrik

    • (Score: 1) by Qzukk on Sunday October 12 2014, @09:56PM

      by Qzukk (1086) on Sunday October 12 2014, @09:56PM (#105289) Journal

      There's a "for scalar in array" construct

      Rather than thinking of it as "in array" think of it as "in list" where list is a string of things separated by whatever character(s) in $IFS

  • (Score: 2) by hemocyanin on Monday October 13 2014, @03:37PM

    by hemocyanin (186) on Monday October 13 2014, @03:37PM (#105586) Journal

    I consider myself beyond introductory scripting, maybe a weak intermediate. I was sort of confused by the example in the original article because there is a lot of similarity between the ` character and the ' character, especially considering how popular it is use funny single and double quotes in websites these days. At first I just thought this was dealing with a file titled: cat names.txt (with the space) so I gave him kudos for dealing with the space in the filename. Anyway, as others have mentioned, I'm much more used to using "while" for this sort of thing.

    names.txt contains
    Joe
    Jack
    John Doe
    Jill Hart
     
    From the TFA (note that even in the article, the letter count is +1):
    $ names=`cat names.txt`; for name in $names; do echo `echo $name | wc -m` $name; done
    4 Joe
    5 Jack
    5 John
    4 Doe
    5 Jill
    5 Hart
     
    My solution:
    $ while read LINE; do echo $(echo $LINE | wc -m) $LINE; done < names.txt
    4 Joe
    5 Jack
    9 John Doe
    10 Jill Hart
     
    My solution for listing and counting firstnames only and correcting the math to actually match the letter count of each name:
    $ while read LINE; do FNAME=$(echo $LINE | awk '{print $1}'); declare -i CLET; CLET=$(echo $FNAME | wc -m); CLET=$CLET-1; echo "$CLET $FNAME"; done < names.txt
    3 Joe
    4 Jack
    4 John
    4 Jill

    As others have said, people come for the comments. What I got out of the comments was the $() bit -- I too would have defaulted to backticks. My next step is to actually understand what the difference between backticks and $() is. Also to the sed and awk posters -- both are great.

    • (Score: 1) by b on Monday October 13 2014, @11:54PM

      by b (2121) on Monday October 13 2014, @11:54PM (#105763)
      $() allows nesting.

      Also, I'd quote the variables to work more elegantly with spaces (e.g. don't compress multiple spaces in a string). i.e. use "$LINE".

      FWIW, I usually use the -r flag for read, to prevent interpretation of backslashes.

      Finally (a minor point), you can use a heredoc instead of the extraneous echo.

      wc -m <<<"$LINE"

      • (Score: 2) by hemocyanin on Monday October 13 2014, @11:58PM

        by hemocyanin (186) on Monday October 13 2014, @11:58PM (#105766) Journal

        helpful info -- thank you

        • (Score: 1) by b on Tuesday October 14 2014, @12:36AM

          by b (2121) on Tuesday October 14 2014, @12:36AM (#105779)

          No worries. Much of it is fairly minor, for edge cases. Still, your code was a hell of a lot better than the tutorial!