sed!

[Robin Paulson]

..is apparently rather tricky to master. i have the following
situation, and can't figure out how to solve it:

1. search for files matching a certain pattern in a directory
structure, using 'find'
2. carry out some processing (based on imagemagick's combine function)
on each file in the list, against a file in a similar structure

so, say there are these files:
/home/robin/var1/14/8849.png
/home/robin/var1/14/8850.png
/home/robin/var1/14/8851.png
/home/robin/var1/13/784.png
/home/robin/var1/13/785.png
/home/robin/var1/13/786.png
/home/robin/var1/13/786.png

and, for each file which exists in that list i want to process it
against its matching pair in a similar structure, with the following
locations:
/home/robin/var2/14/8849.png
/home/robin/var2/14/8850.png
/home/robin/var2/14/8851.png
/home/robin/var2/13/784.png
/home/robin/var2/13/785.png
/home/robin/var2/13/786.png
/home/robin/var2/13/786.png

to produce files with these locations:
/home/robin/var3/14/8849.png
/home/robin/var3/14/8850.png
/home/robin/var3/14/8851.png
/home/robin/var3/13/784.png
/home/robin/var3/13/785.png
/home/robin/var3/13/786.png
/home/robin/var3/13/786.png

note: there are far more files in the second list than the first - i
only want to process files which exist in the first list (the presence
in the second list can be assumed true)

so, i start off with:

#find all files in the directory structure (there are only .png files)
for i in 'find /home/robin/var1 -f'
#use the imagemagick combine function to overlay a file in folder var1
on top of folder var2
do combine $i <some sed to get from $i to the corresponding filename in
list 2> <some sed to get from $i to the corresponding filename in list
3>
done

and the sed part has me lost. sed appears to want a file, although it
maybe can accept parameters - i'm not sure. man sed is...well, not much
help

and i think i've explained this really badly....

cheers,

[Bruce Kingsbury]

var2=$(echo $var1 | sed -e/.... )

--
Sent from my Ideos!

Post by Robin Paulson
..is apparently rather tricky to master. i have the following
1. search for files matching a certain pattern in a directory
structure, using 'find'
2. carry out some processing (based on imagemagick's combine function)
on each file in the list, against a file in a similar structure
/home/robin/var1/14/8849.png
/home/robin/var1/14/8850.png
/home/robin/var1/14/8851.png
/home/robin/var1/13/784.png
/home/robin/var1/13/785.png
/home/robin/var1/13/786.png
/home/robin/var1/13/786.png
and, for each file which exists in that list i want to process it
against its matching pair in a similar structure, with the following
/home/robin/var2/14/8849.png
/home/robin/var2/14/8850.png
/home/robin/var2/14/8851.png
/home/robin/var2/13/784.png
/home/robin/var2/13/785.png
/home/robin/var2/13/786.png
/home/robin/var2/13/786.png
/home/robin/var3/14/8849.png
/home/robin/var3/14/8850.png
/home/robin/var3/14/8851.png
/home/robin/var3/13/784.png
/home/robin/var3/13/785.png
/home/robin/var3/13/786.png
/home/robin/var3/13/786.png
note: there are far more files in the second list than the first - i
only want to process files which exist in the first list (the presence
in the second list can be assumed true)
#find all files in the directory structure (there are only .png files)
for i in 'find /home/robin/var1 -f'
#use the imagemagick combine function to overlay a file in folder var1
on top of folder var2
do combine $i <some sed to get from $i to the corresponding filename in
list 2> <some sed to get from $i to the corresponding filename in list
3>
done
and the sed part has me lost. sed appears to want a file, although it
maybe can accept parameters - i'm not sure. man sed is...well, not much
help
and i think i've explained this really badly....
cheers,
--
robin
http://bumblepuppy.org/blog/?p=237 - government bill to remove basic
human rights in NZ
_______________________________________________
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Nevyn]

Post by Bruce Kingsbury
var2=$(echo $var1 | sed -e/.... )

for a in $( find /home/robin/var1 -f' ) ; do
combine $a "$( echo $a | sed 's/var1/var2/g' )" -f "$( echo $a | sed
's/var1/var3/g' )"
done

It's a bit rough - won't deal with spaces terribly well.... Assumes
that -f specifies the output file...

Regards,
Nevyn
http://nevsramblings.blogspot.com/

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Volker Kuhlmann]

Post by Nevyn
for a in $( find /home/robin/var1 -f' ) ; do
combine $a "$( echo $a | sed 's/var1/var2/g' )" -f "$( echo $a | sed
's/var1/var3/g' )"
done
It's a bit rough

Yeah, and a bit buggy. Read blows up with a syntax error.

Try this:

find /home/robin/var1 -type f -name \*.png \
| while read -r file; do
cmpfile="$( echo "$file" | sed 's,/var1/,/var2/,g' )"
newfile="$( echo "$file" | sed 's,/var1/,/var3/,g' )"
combine "$file" -f "$cmpfile" "$newfile"
done

People really need to learn to quote in shell scripts.

The combine command and its arguments are your problem.
It's not a command belonging to ImageMagick.

If you use bash, you don't really need to use sed at all:

find /home/robin/var1 -type f -name \*.png \
| while read -r file; do
cmpfile="${file/var1/var2}"
newfile="${file/var1/var3}"
combine "$file" -f "$cmpfile" "$newfile"
done

Both versions will deal with spaces in filenames, but not with newlines
in filenames.

[Robin Paulson]

Post by Volker Kuhlmann
find /home/robin/var1 -type f -name \*.png \
| while read -r file; do
cmpfile="$( echo "$file" | sed 's,/var1/,/var2/,g' )"
newfile="$( echo "$file" | sed 's,/var1/,/var3/,g' )"
combine "$file" -f "$cmpfile" "$newfile"
done

that's what i was after. this and a combination of basename and dirname
(thanks for that atom)

Post by Volker Kuhlmann
The combine command and its arguments are your problem.
It's not a command belonging to ImageMagick.

haha, yeah i meant composite. well spotted...

[Vik Olliver]

Post by Robin Paulson

Post by Volker Kuhlmann
find /home/robin/var1 -type f -name \*.png \
| while read -r file; do
cmpfile="$( echo "$file" | sed 's,/var1/,/var2/,g' )"
newfile="$( echo "$file" | sed 's,/var1/,/var3/,g' )"
combine "$file" -f "$cmpfile" "$newfile"
done

that's what i was after. this and a combination of basename and dirname
(thanks for that atom)

Post by Volker Kuhlmann
The combine command and its arguments are your problem.
It's not a command belonging to ImageMagick.

haha, yeah i meant composite. well spotted...

If you want to spot names that are in both directories, put directory
name listings in files and:

cat file1 file2|sort|uniq -d

Vik :v)

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Robin Paulson
and the sed part has me lost. sed appears to want a file, although it
maybe can accept parameters - i'm not sure. man sed is...well, not much
help

===============

this might be easier to make sense of than sed...

for n in $( find /some/path/ -type f | xargs -n1 basename )

with a recent version of bash you can also do this and see what happens:
for n in $( find /some/path/ -type f )
do
echo ${n##*/}
done

the magic there is "${n##*/}". "${n}" contains the full path to the file,
but "${n##*/}" only substitutes the filename, same as the "basename"
command.

that will give you just filenames without paths, which you can then test
for and act on...

if [ -f /path/2/${n} ] && do-this /path/1/${n} /path/2/${n} /path/3/${n}

or...

if [ -f /path/2/${n##*/} ] && do-this /path/1/${n##*/} /path/2/${n##*/} /path/3/${n##*/}

also, if all of the files your searching for are in the same directory and
there are no sub-directories... this is really simple...

for n in $( cd /some/path/ ; ls )
do
... do stuff ...
done
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

When cryptography is outlawed,
b25seSBvdXRsYXdzIHdpbGwgdXNlIGNyeXB0b2dyYXBoeS4K


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Jim Tittsler]

Post by Robin Paulson
#find all files in the directory structure (there are only .png files)
for i in 'find /home/robin/var1 -f'
#use the imagemagick combine function to overlay a file in folder var1 on
top of folder var2
do combine $i <some sed to get from $i to the corresponding filename in list
2> <some sed to get from $i to the corresponding filename in list 3>
done

do combine $i $(echo $i|sed -e #/var1/#/var2/#) $(echo $i|sed -e
#/var1/#/var3/#)

Is the brute force method, presuming the /var1/ component is really
unique in all "$i"s... or pick a longer prefix that *is* unique.

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Anthony Chapman]

do combine $i <some sed to get from $i to the corresponding filename in list 2> <some sed to get from $i to the corresponding filename in list 3>

If I understand what you're after you may want this:
do combine $i `echo $i|sed -e 's/var1/var2/'` `echo $i|sed -e 's/var1/var3/'`

or:
`echo $i|sed -e 's/^\(.*\)var1\(.*\)$/do combine \1var1\2 \1var2\2 \1var3\2/'`

You might want to run the latter without the backticks, or the former
with "echo" added to the start, to check it does what it is meant to.
If it looks good then either run it again with the backticks, copy and
paste the output as commands, or redirect the output to a file which
can be turned into a shell script and run.

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Martin D Kealey]

Post by Anthony Chapman
do combine $i `echo $i|sed -e 's/var1/var2/'` `echo $i|sed -e 's/var1/var3/'`
`echo $i|sed -e 's/^\(.*\)var1\(.*\)$/do combine \1var1\2 \1var2\2 \1var3\2/'`

Please can we stop recommending `backticks` instead of $(brackets) !?

Brackets are a matched pair and so nest naturally, whereas backticks are a
repeated single characters so you need other machinations to nest them.

And worse, they're barely visually distinguishable from about 4 other
punctuation marks. Backticks were a bad choice typographically right from
the start.

The bracket notation has been usable longer than Linux has been around - the
POSIX standard was published in 1992, after they were already widely
implemented.

In case you think I'm being overly pedantic, I challenge anyone to rewrite
this with backticks instead of brackets:

perl -pi -e 's`#! */bin/sh`#!/bin/bash`;' $(
grep -il 'copyright.*fsf' $(
grep -l '#! */bin/sh' $(
find $( grep PATH /etc/login.defs /etc/default/* |
sed -e '
s`.* *= *``;
s`[^/]*$``;
s`:` `g;
' ) -type f -perm /4000 -print)))

(Oh yes, I'm well aware of xargs, but this is just a quick mock-up to show
how nesting backticks is just horrible.)

Good luck!

-Martin

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Robin Sheat]

Post by Martin D Kealey
Please can we stop recommending `backticks` instead of $(brackets) !?

Some days I wish I could mark emails as 'like' :)

Robin.

[Atom Smasher]

Post by Robin Sheat

Post by Martin D Kealey
Please can we stop recommending `backticks` instead of $(brackets) !?

Some days I wish I could mark emails as 'like' :)

===============

this.

+1

;)
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"Just because something makes perfect sense doesn't mean
it is true."
-- Rory Miller, Meditations on Violence

Corollary: Just because something doesn't make any sense
doesn't mean it isn't true.
-- Atom Smasher


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Volker Kuhlmann]

Post by Martin D Kealey
Please can we stop recommending `backticks` instead of $(brackets) !?

First: Can we please speak English:
Brackets: []
Parentheses: ()
Consult any dictionary if you have a differing opinion.
In particular, programmers should be well aware of such differences
because that kind of fluff like @#$)(&%}({$.</>! is usually of
considerable importance to their profession. Replace parentheses with
brackets in your program, sit back and we watch the entertainment.

Post by Martin D Kealey
Brackets are a matched pair and so nest naturally, whereas backticks are a
repeated single characters so you need other machinations to nest them.

Backticks are easier and faster to type.

The ratio of syntax to content in bash is already way too high.

If your program needs nested backticks, you should probably re-write
your program in a legible fashion. (But then I see you're a perl
programmer, where such thing is hopeless... ;-))) )
Please provide a *real* example where you *need* nested backticks.

The backticks always work on any sh-type shell (even the Solaris /bin/sh
brain death).

Post by Martin D Kealey
The bracket notation has been usable longer than Linux has been around - the
POSIX standard was published in 1992, after they were already widely
implemented.

Hmm interesting, does POSIX require $() to work like `` since 1992 at
the latest? I wasn't aware of that. Are you sure?

Post by Martin D Kealey
In case you think I'm being overly pedantic, I challenge anyone to rewrite

Nope, only good programming examples count. Anyone can go out of their
way and deliberately write obfuscated gobbledegook.

Volker

[Reed Wade]

Post by Volker Kuhlmann
Brackets: []
Parentheses: ()
Consult any dictionary if you have a differing opinion.

Oooh, dude, I hope you're not serious. That kind of talk is the sort
that leads straight to the mcdonaldsi[sz]ation of this country.

You'll want to fix http://en.wikipedia.org/wiki/Bracket and all the
people who pronounce () as brackets (there's really a lot of them). I
won't say this didn't mess with me some when I first came to NZ land
but--it seems kinda harsh to claim there's exactly one English. And on
a region specific mailing list it seems reasonable to use the local
language.

-reed

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Bruce Clement]

On Sun, Jun 12, 2011 at 4:39 PM, Volker Kuhlmann <***@paradise.net.nz>wrote:


Please provide a *real* example where you *need* nested backticks.
I haven't got an example to hand, but I very occasionally hit one. Usually
when I'm doing something like feeding the results of one /usr/bin/find,
filtered through an awk script into the directory list of a second
/usr/bin/find which in turn is ...

I get this as I have some untypical use cases, many stemming from
maintaining around 100 small websites with no CMS & from time -to-time I
need to do bulk changes.

As I say, very rare & the rest of the time I just use backticks as they are
easier to type

Cheers
--
Bruce Clement

Home: http://www.clement.co.nz/
Twitter: http://twitter.com/Bruce_Clement
Directory: http://www.searchme.co.nz/

"Before attempting to create something new, it is vital to have a good
appreciation of everything that already exists in this field." Mikhail
Kalashnikov
_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Volker Kuhlmann
Please provide a *real* example where you *need* nested backticks.

================

very slightly modified from Unix Power Tools, 3rd Edition (Shelley Powers,
Jerry Peek, Tim O'Reilly, Mike Loukides)...

echo "Next year will be $(expr $(date +%Y) + 1)."

can it be done without nested command substitution? sure (this is *nix,
there's almost never only one way to do something, which makes your
challenge a bit strange), but this is more elegant (IMHO) than most other
methods and not too hard to read... i can certanly make it harder to read
without nested command substitution.

it goes on to say "Beginners (and some long-time programmers too) might
argue that you should never nest command substitution because it's too
confusing. I think there are times nesting is clearer. It's more compact
and doesn't need temporary storage. And it's not that hard to understand
once you see what's happening."

there are certainly times when i've used nested command substitution in
shell scripts, and a few times i've used it on the command line. of
course, i also use process substitution and some tricky variable
substitution regularly, often on the command line.

Post by Volker Kuhlmann
The backticks always work on any sh-type shell (even the Solaris /bin/sh
brain death).

=============

solaris doesn't even have a user-land "stat" command. that's one of the
reasons i always push (administratively) to make sure zsh is installed.

from my zshrc (http://smasher.org/zsh/)...

## not all systems (solaris) have a userland stat
[[ -x $(whence -p stat) ]] || { zmodload zsh/stat && alias stat="stat -ronL" }

Post by Volker Kuhlmann
Nope, only good programming examples count. Anyone can go out of their
way and deliberately write obfuscated gobbledegook.

=============

obfuscated gobbledegook can be done with or without nested command
substitution. when i was writing a lot of php i tended to use a lot of
nested command substitution, and it was always clean.

eschew obfuscation. espouse elucidation.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"The future isn't what it used to be."
-- Yogi Berra


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Vik Olliver]

Post by Atom Smasher
very slightly modified from Unix Power Tools, 3rd Edition (Shelley Powers,
Jerry Peek, Tim O'Reilly, Mike Loukides)...
echo "Next year will be $(expr $(date +%Y) + 1)."

The wonders of Open Source mean there's more than one way to do it. This
is how things evolve in the Open Source world. The question to ask is
"Why do people still want to use backticks?"

If there is an advantage, then the practice won't die out and the world
needs to shift accordingly :)

Vik :v)

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Martin D Kealey]

For a quick thing to type, they're great.

For a quick thing to read, they're not.

When offering training/mentoring/advice, IMO one should favour the reader
over the writer.

Which is where I came in.

-Martin

Date: Tue, 14 Jun 2011 09:55:31 +1200
Subject: Re: [nzlug] sed!

Post by Atom Smasher
very slightly modified from Unix Power Tools, 3rd Edition (Shelley Powers,
Jerry Peek, Tim O'Reilly, Mike Loukides)...
echo "Next year will be $(expr $(date +%Y) + 1)."

The wonders of Open Source mean there's more than one way to do it. This
is how things evolve in the Open Source world. The question to ask is
"Why do people still want to use backticks?"
If there is an advantage, then the practice won't die out and the world
needs to shift accordingly :)
Vik :v)
_______________________________________________
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Volker Kuhlmann
Please provide a *real* example where you *need* nested backticks.

==================

very slightly off-topic, but here's an example of nested process
substitution that i just worked out to grab some videos over a wonky
connection...

while [[ $? != 0 ]]
do
youtube-dl -tca <( egrep -vf <( ls | cut -d- -f2- | egrep -v part | cut -d. -f1 ) list )
done

"list" is a file containing youtube URLs.

instead of using nested process substitution, i could have used one
process substitution and piped the output into "xargs youtube-dl -tca" but
this was easier to write, and actually seems easier to read. YMMV.

every now and again i do similar with command substitution, or a
combination of command and process substitution.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"[Commercial radio] is owned by one or two corporations now, and
they're not in the music business. They're in the advertising
business. So let's not kid ourselves. If you want to hear music,
go buy a guitar."
-- Elvis Costello


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Volker Kuhlmann]

Post by Atom Smasher
youtube-dl -tca <( egrep -vf <( ls | cut -d- -f2- | egrep -v part | cut -d. -f1 ) list )
"list" is a file containing youtube URLs.

I'm afraid I fail to see how any of this is legible.

In particular, cmd2 <( cmd1 ) appears to be an obfuscation of
cmd1 | cmd2. We read from left to right; your construction subverts that
(and uses 3 times as many fluff characters). Nesting this takes the cake
in bad programming.

Sometimes you do need to do it your way, if cmd2 is a compound command
which e.g. sets more shell variables. They don't propagate up from
subshells so there's a need to run that in the current shell. This is
not the case in your example.

My opinion.

Volker

[Atom Smasher]

Post by Volker Kuhlmann

Post by Atom Smasher
youtube-dl -tca <( egrep -vf <( ls | cut -d- -f2- | egrep -v part | cut -d. -f1 ) list )
"list" is a file containing youtube URLs.

I'm afraid I fail to see how any of this is legible.
In particular, cmd2 <( cmd1 ) appears to be an obfuscation of cmd1 |
cmd2. We read from left to right; your construction subverts that (and
uses 3 times as many fluff characters). Nesting this takes the cake in
bad programming.

==============

if you can show me how my example works using pipes, i'd be interested to
see the result. especially if it's either more compact or easier to read.

i suppose the outer process substitution could be replaced with a command
substitution... but that doesn't change things too much...

youtube-dl -tc $( grep -vf <( ls | cut -d- -f2- | egrep -v part | cut -d. -f1 ) list )

depending on the shell and how the options are set, that *could* cause
problems interpreting an unquoted "?" which is part of youtube URLs. using
process substitution (as in the original example) eliminates that
possibility.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"Democracy must be something more than two wolves and a sheep
voting on what to have for dinner."
-- James Bovard


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Alexander Taler]

Post by Atom Smasher
if you can show me how my example works using pipes, i'd be interested to
see the result. especially if it's either more compact or easier to read.

How about this?

ls | cut -d- -f2- | egrep -v part | cut -d. -f1 | \
egrep -vf - list | youtube-dl -tca -
--
hsh: The {Human|Happy|History} shell http://www.nongnu.org/hsh/
DEvelopment REFined - Enabling your software team - http://deref.co.nz/
PGP: ID: 0x23DC453B FPR: 42D0 66C2 9FF8 553A 373A B819 4C34 93BA 23DC 453B
the different sources can produce differing forecasts, however we believe [it]
contributes to a more compelling and interesting weather section


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Alexander Taler
How about this?
ls | cut -d- -f2- | egrep -v part | cut -d. -f1 | \
egrep -vf - list | youtube-dl -tca -

==================

nope.

youtube-dl doesn't accept input on stdin; URL's have to be supplied as
arguments or a file (or fifo or process substitution) with "-a". grep
accept stdin as an argument to "-f"; that has to be a file, fifo or a
process substitution.

however... the last part could be
| xargs youtube-dl -tc

grep might be workable with "xargs -I xxx" but things get tricky with
multiple arguments... grep's "-f" considers things per-line, but xargs
doesn't play nice with that. maybe xargs/grep can be made to work in this
case, but i don't think it would be worth the effort and i'm pretty sure
it would be uglier than process substitution.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"All that is necessary for the triumph of evil
is for good men to do nothing."
-- Edmund Burke


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Robin Sheat]

Post by Atom Smasher
youtube-dl doesn't accept input on stdin

but:

youtube-dl -tca <( ..things here... )

is redirecting stuff into it's stdin, unless I'm mistaken.

Robin.

[Robin Sheat]

Post by Robin Sheat
is redirecting stuff into it's stdin, unless I'm mistaken.

Oh, I just checked. I am mistaken, I'd never seen that notation before. Nifty
(if a bit ugly :)

Robin.

[Cliff Pratt]

Post by Robin Sheat

Post by Robin Sheat
is redirecting stuff into it's stdin, unless I'm mistaken.

Oh, I just checked. I am mistaken, I'd never seen that notation before. Nifty
(if a bit ugly :)

eg
diff <(ls dir1) <(ls dir2)

It's not ugly, it's clever.

Cheers,

Cliff

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Volker Kuhlmann]

Post by Atom Smasher
if you can show me how my example works using pipes, i'd be interested to
see the result. especially if it's either more compact or easier to read.

I had already said. It is both easier to read and more compact. Unless I
am mis-understanding the bash man page, process substitution is a fancy
word for piping the output of one process into the input of another (I
use the "pipe" character for that).

Alexander has beaten me to it.

Volker

[Atom Smasher]

Post by Volker Kuhlmann
I had already said. It is both easier to read and more compact. Unless I
am mis-understanding the bash man page, process substitution is a fancy
word for piping the output of one process into the input of another (I
use the "pipe" character for that).
Alexander has beaten me to it.

==============

except his example doesn't work.

a pipe and a fifo are different things. process substitution is (in most
cases) using a fifo... which is different than a pipe (|). a fifo can be
used to simulate a "regular" file but a pipe can't. a pipe just "connects"
stdout from one process to stdin of another.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

Bob Woodward:
"How do you think history will regard the war in Iraq?"
George "dubya" Bush:
"It won't matter. We'll all be dead."


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Alexander Taler]

Post by Atom Smasher

Post by Volker Kuhlmann
Alexander has beaten me to it.

==============
except his example doesn't work.

Did you try it? Perhaps you need to upgrade. I checked the
documentation before I wrote that, although I didn't test it.

Post by Atom Smasher
youtube-dl --version

2011.02.25b

Post by Atom Smasher
youtube-dl --help | egrep -A1 -e '-a '

-a FILE, --batch-file=FILE
file containing URLs to download ('-' for stdin)

Post by Atom Smasher
a pipe and a fifo are different things. process substitution is (in most
cases) using a fifo... which is different than a pipe (|). a fifo can be
used to simulate a "regular" file but a pipe can't. a pipe just "connects"
stdout from one process to stdin of another.

Basically, process substitution is handy when your program isn't
flexible enough in how it accepts input, either not supporting
'-' for stdin, or not accepting multiple input streams, like diff
on the output of two separate processes.

Alex
--
hsh: The {Human|Happy|History} shell http://www.nongnu.org/hsh/
DEvelopment REFined - Enabling your software team - http://deref.co.nz/
PGP: ID: 0x23DC453B FPR: 42D0 66C2 9FF8 553A 373A B819 4C34 93BA 23DC 453B
the different sources can produce differing forecasts, however we believe [it]
contributes to a more compelling and interesting weather section


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Alexander Taler

Post by Atom Smasher
except his example doesn't work.

Did you try it? Perhaps you need to upgrade. I checked the
documentation before I wrote that, although I didn't test it.

============

i'm using an older version, so yeah, that part should work, now. but like
i said, even without that new feature, it's not hard to work that out with
xargs.

this part would still not work as desired:
grep -vf - file

Post by Alexander Taler
Basically, process substitution is handy when your program isn't
flexible enough in how it accepts input, either not supporting '-' for
stdin, or not accepting multiple input streams, like diff on the output
of two separate processes.

=============

who says diff can't handle multiple input streams ;)
diff -u <( gzcat file-1.gz ) <( gzcat file-2.gz )

or even...
paste <( w | tail -n +3 | sort -k 2 ) <( who | sort -k 2 )

sometimes disk space is at a premium, other times disk i/o is limited,
sometimes you may just not want the data to be written to disk (eg,
security/encryption), sometimes you may have to deal with a root file
system that's read-only and, as you've pointed out, some programs don't
deal with stdin as you'd like them to. i'm sure there are other cases,
aside from obfuscating code.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"Until humankind opts for harmony with nature, over
domination, oneness over otherness, the seasons of
death and destruction will only escalate."
-- Mumia Abu-Jamal


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Alexander Taler]

i'm using an older version, so yeah, that part should work, now. but like i
said, even without that new feature, it's not hard to work that out with
xargs.
this part would still not work as desired: grep -vf - file

Sure it does.

echo root\\ndaemon | grep -f - /etc/passwd

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh

who says diff can't handle multiple input streams ;)
diff -u <( gzcat file-1.gz ) <( gzcat file-2.gz )

All I meant was that diff sees filenames, and doesn't care that
they're not regular files. Writing a program which receives
multiple streams of input on configurable file descriptors would
be confusing, ugly and silly and it would require shell
contortions like this:

cat /etc/hostname | (sed 's/^/0: /'
sed 's/^/3: /' 0<&3
sed 's/^/4: /' 0<&4
sed 's/^/5: /' 0<&5
) 5</etc/papersize 4</etc/debian_version 3</etc/host.conf

sometimes disk space is at a premium, other times disk i/o is limited,
sometimes you may just not want the data to be written to disk (eg,
security/encryption), sometimes you may have to deal with a root file
system that's read-only and, as you've pointed out, some programs don't
deal with stdin as you'd like them to. i'm sure there are other cases,
aside from obfuscating code.

Definitely some more good uses for process substituion there.

Thanks for an interesting discussion, it gave me the incentive to
further explore some areas of shell programming.

Alex
--
hsh: The {Human|Happy|History} shell http://www.nongnu.org/hsh/
DEvelopment REFined - Enabling your software team - http://deref.co.nz/
PGP: ID: 0x23DC453B FPR: 42D0 66C2 9FF8 553A 373A B819 4C34 93BA 23DC 453B
the different sources can produce differing forecasts, however we believe [it]
contributes to a more compelling and interesting weather section


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Alexander Taler

this part would still not work as desired: grep -vf - file

Sure it does.

=============

hey, it does work! i thought i even tested it before posting.

Post by Alexander Taler
All I meant was that diff sees filenames, and doesn't care that they're
not regular files. Writing a program which receives multiple streams of
input on configurable file descriptors would be confusing, ugly and
cat /etc/hostname | (sed 's/^/0: /'
sed 's/^/3: /' 0<&3
sed 's/^/4: /' 0<&4
sed 's/^/5: /' 0<&5
) 5</etc/papersize 4</etc/debian_version 3</etc/host.conf

==============

intuitively, that looks like all four seds are reading (or at least
capable of reading) stdin, but stdin is being "consumed" by the first sed.

using zsh (not gonna work in bash), you *can* send stdout into multiple
processes... although there's no guarantee which process(es) will finish
first (in this example)...

{ date } > >( tr '[a-z]' '[A-Z]' ) > >( rot13 ) > >( openssl base64 )

* on ubuntu, rot13 is available as part of the "bsdgames" package. i like
it because it's a quick and easy way to demonstrate pipelines.

that takes the output from "date" and passes it into three processes; each
process will output data to the terminal. it won't return until all of the
processes are done.

on gnu/linux, you can use "date +%c' '%N" to verify that all three
processes acting on the output are reading input at the same nanosecond.

install zsh; man zshmisc; scroll down to "MULTIOS".

if you *really* want to do that in bash...

date | tee >( tr '[a-z]' '[A-Z]' > /dev/tty ) \
| tee >( rot13 >/dev/tty ) \
| tee >( openssl base64 > /dev/tty ) > /dev/null

seriously, there are times when i find things like this useful and
practical.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"The Army is the Indian's best friend."
-- General George Armstron Custer, 1870


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Volker Kuhlmann]

Post by Atom Smasher
date | tee >( tr '[a-z]' '[A-Z]' > /dev/tty ) \
| tee >( rot13 >/dev/tty ) \
| tee >( openssl base64 > /dev/tty ) > /dev/null

You have a WEIRD sense of humour... ;-))

You're right, reading from file descriptors is seekable (unless fed by a
pipe), reading from a pipe is not. The bash manual doesn't mention that
difference. A program going belly-up reading its 1-per-line URL input is
broken though. Thanks for the interesting input.

Volker

[Martin D Kealey]

Post by Atom Smasher
date | tee >( tr '[a-z]' '[A-Z]' > /dev/tty ) \
| tee >( rot13 >/dev/tty ) \
| tee >( openssl base64 > /dev/tty ) > /dev/null

A single redirection outside the command substitutions will suffice. And
the "tee" command can take multiple output filenames, so you only need
one instance of tee:

date |
tee >( tr '[a-z]' '[A-Z]' ) \

Post by Atom Smasher
( rot13 ) \
( openssl base64 ) > /dev/tty

However that will return after "tee" finishes writing to all output
streams, so the output may be displayed after the next shell prompt
(when run interactively), or after the next command begins (when inside
a script).

That can be fixed by having something else to wait on:

date |
tee >( tr '[a-z]' '[A-Z]' ) \

Post by Atom Smasher
( rot13 ) \
( openssl base64 ) |

cat > /dev/tty

This latter waits on "cat", which means it waits until all 5 of the
preceding processes have finished producing output.

Also note the lack of "\" after the "|" -- they're not needed, and the spec
& manual are quite explicit about that. Same goes for all
simple-command-terminating punctuation.

-Martin

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Martin D Kealey
A single redirection outside the command substitutions will suffice. And
the "tee" command can take multiple output filenames, so you only need
date |
tee >( tr '[a-z]' '[A-Z]' ) \

Post by Atom Smasher
( rot13 ) \
( openssl base64 ) > /dev/tty

=============

or use zsh ;)

date > >( tr '[a-z]' '[A-Z]' ) > >( rot13 ) > >( openssl base64 )

Post by Martin D Kealey
However that will return after "tee" finishes writing to all output
streams, so the output may be displayed after the next shell prompt
(when run interactively), or after the next command begins (when inside
a script).
date |
tee >( tr '[a-z]' '[A-Z]' ) \

Post by Atom Smasher
( rot13 ) \
( openssl base64 ) |

cat > /dev/tty
This latter waits on "cat", which means it waits until all 5 of the
preceding processes have finished producing output.

==============

in zsh, this can be fixed by placing curly-braces around "date"...

{ date } > >( tr '[a-z]' '[A-Z]' ) > >( rot13 ) > >( openssl base64 )

the problem with your [ba]sh examples are that after writing to multiple
files (which are, for anyone who's trying to keep up, really fifos) tee
will still send it's stdin to stdout...

$ echo test | tee >( rot13 )
test
grfg

in order to suppress the duplication of input on the output, output from
tee has to be closed or sent someplace else...

$ echo test | tee >( rot13 ) > /dev/null
grfg

the problem then... how do you get rid of that output and still "cat" the
rest of the output? well... the output of the fifos is implicitly
connected to the controlling terminal *after* tee... so...

date | tee >( rot13 ; sleep 2 ) > /dev/null | cat -

to clarify... this puts tee and the fifos in a subshell that redirects
output from tee into /dev/null and works as above:

date | ( tee >( rot13 ) >( base64 ; sleep 2 ) > /dev/null ) | cat -

but this... sends the output of the subshell into /dev/null and produces
no output:

date | ( tee >( rot13 ) >( base64 ; sleep 2 ) ) > /dev/null | cat -

so, that can send the output from tee into /dev/null and only cats the
output from the fifos. the sleep command makes it clear that the shell is
waiting for cat, and cat is waiting for something else.

Post by Martin D Kealey
Also note the lack of "\" after the "|" -- they're not needed, and the
spec & manual are quite explicit about that. Same goes for all
simple-command-terminating punctuation.

=============

it's a matter of style... like putting curly-braces around ${variables} or
a dash after "cat" when it's catting stdin.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"If you take out the killings, Washington
actually has a very very low crime rate."
-- M. Barry, Mayor of Washington, DC


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Martin D Kealey]

Post by Volker Kuhlmann
I had already said. It is both easier to read and more compact. Unless I
am mis-understanding the bash man page, process substitution is a fancy
word for piping the output of one process into the input of another (I
use the "pipe" character for that).

Only subtly misunderstood.

Command substitution doesn't connect stdin or stdout, it connects a "random"
file descriptor, and at the point in the command where the ">()" or "<()"
appears, instead places something like "/dev/fd/43".

So something like this:

cat <(echo Hello)

translates into something like this:

[create pipe /dev/fd/3 /dev/fd/4]
echo Hello > /dev/fd/4 &
cat /dev/fd/3

You can put an additional "<" or ">" before the "<()" or ">()" to have the
shell perform the redirection before invoking the command. Unless you do so,
no, it's not quite the same as using a '|'' pipe.

-Martin

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Wayne Rooney]

Post by Robin Paulson
..is apparently rather tricky to master. i have the following

Tried cut?

#!/bin/bash
for I in $( find /home/robin/var1 -f | cut -f 5-6 -d/) ; do
combine /home/robin/var1/"$I" /home/robin/var2/"$I" /home/robin/var3/"$I"
done

Wayne

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Cliff Pratt]

..is apparently rather tricky to master. i have the following situation,
1. search for files matching a certain pattern in a directory structure,
using 'find'
2. carry out some processing (based on imagemagick's combine function)
on each file in the list, against a file in a similar structure
/home/robin/var1/14/8849.png
/home/robin/var1/14/8850.png
/home/robin/var1/14/8851.png
/home/robin/var1/13/784.png
/home/robin/var1/13/785.png
/home/robin/var1/13/786.png
/home/robin/var1/13/786.png
and, for each file which exists in that list i want to process it
against its matching pair in a similar structure, with the following
/home/robin/var2/14/8849.png
/home/robin/var2/14/8850.png
/home/robin/var2/14/8851.png
/home/robin/var2/13/784.png
/home/robin/var2/13/785.png
/home/robin/var2/13/786.png
/home/robin/var2/13/786.png
/home/robin/var3/14/8849.png
/home/robin/var3/14/8850.png
/home/robin/var3/14/8851.png
/home/robin/var3/13/784.png
/home/robin/var3/13/785.png
/home/robin/var3/13/786.png
/home/robin/var3/13/786.png
note: there are far more files in the second list than the first - i
only want to process files which exist in the first list (the presence
in the second list can be assumed true)
#find all files in the directory structure (there are only .png files)
for i in 'find /home/robin/var1 -f'
#use the imagemagick combine function to overlay a file in folder var1
on top of folder var2
do combine $i <some sed to get from $i to the corresponding filename in
list 2> <some sed to get from $i to the corresponding filename in list 3>
done
and the sed part has me lost. sed appears to want a file, although it
maybe can accept parameters - i'm not sure. man sed is...well, not much
help
and i think i've explained this really badly....

I think that you would be better off with a perl or python script rather
than bash or whatever shell you are using. Which is not really an answer
to your question..

Cheers,

Cliff

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug