bogus ${var/#pattern/replacement} or ${var/%pattern/replacement} when var is unset

[stephane-chazelas]

$ ksh -c 'echo "${a/#foo/$bar}"'
#foo/$bar
$ ksh -c 'echo "${a/%foo/$bar}"'
%foo/$bar

Expected: an empty line of output. See also how $var was not expanded.

Same for empty arrays:

$ ksh -c 'typeset -a a=(); echo "${a[@]/#foo/$bar}"'
#foo/$bar

[McDutchie]

Thanks for the report.

This bug is also in 93u+ 2012-08-01 and ksh2020, so we'll have to find a fix ourselves again.

[McDutchie]

Found a patch already.

Since we've already errored out at this point if the nounset option is set, we can simply treat a nonexistent value as an empty one.

diff --git a/src/cmd/ksh93/sh/macro.c b/src/cmd/ksh93/sh/macro.c
index 0765fda60..694411796 100644
--- a/src/cmd/ksh93/sh/macro.c
+++ b/src/cmd/ksh93/sh/macro.c
@@ -1744,6 +1744,8 @@ retry1:
 					mp->assign &= ~1;
 					mp->arith = 0;
 					newquote = 0;
+					if(!v)
+						v = Empty;
 				}
 				else if(c=='?' || c=='=')
 					mp->split = mp->pattern = 0;

[stephane-chazelas]

Thanks, but the behaviour of ${array[@]/#/prefix} (often used as replacement of rc/fish's prefix$array or zsh's prefix$^array) is still undesirable and misaligned with bash's or zsh's:

$ bar=xxx ksh -c 'typeset -a a=(); echo "${a[@]/#/$bar}"'
xxx
$ bar=xxx bash -c 'typeset -a a=(); echo "${a[@]/#/$bar}"'

$ bar=xxx zsh -c 'typeset -a a=(); echo "${a[@]/#/$bar}"'

$

One would expect echo would be passed an empty list for a substitution applied to an empty list.

[McDutchie]

Thanks for testing!

Patch attempt 2: if there is no value, disable the #/% handling by setting type to 0.

diff --git a/src/cmd/ksh93/sh/macro.c b/src/cmd/ksh93/sh/macro.c
index 0765fda60..239073f6e 100644
--- a/src/cmd/ksh93/sh/macro.c
+++ b/src/cmd/ksh93/sh/macro.c
@@ -1734,7 +1734,7 @@ retry1:
 					{
 						ssize_t d = fcmbget(&LEN);
 						fcseek(-LEN);
-						if(d=='(')
+						if(d=='(' || !v)
 							type = 0;
 					}
 					fcseek(-1);

[stephane-chazelas]

Thanks, now back to unset scalar expansion, that aligns with bash rather than zsh:

$ bash -c 'echo "${a/#/x}"'

$ ksh -c 'echo "${a/#/x}"'

$ zsh -c 'echo "${a/#/x}"'
x

But aligning with zsh would make it more consistent with:

$ ksh -c 'echo "${a/@()/x}"'
x

Or:

$ ksh -c 'echo "${a/~(E:^)/x}"'
x

Or:

$ ksh -c 'echo "${a/""/x}"'
x

Having said that, with or without the patch applied:

$ ksh -c 'typeset -a a=(); echo "${a[@]/*/<\0>}"'
<>

Would also be wrong IMO.

"${a[@]}" expands to 0 element if the $a array is empty and "$a" (short for "${a[0]}") expands to one element whether ${a[0]} is set or not. Expansion for unset scalars usually expands like for scalars assigned an empty value, so I'd expect it to be the same for ${a[@]/pattern/replacement} (expand to 0 element if $a is an empty array) and ${a/pattern/replacement} (expands the same for unset $a and empty $a).

[McDutchie]

Thanks, now back to unset scalar expansion, that aligns with bash rather than zsh:

$ bash -c 'echo "${a/#/x}"'

$ ksh -c 'echo "${a/#/x}"'

$ zsh -c 'echo "${a/#/x}"'
x

I can live with this. The inconsistencies are deep in obscure corner case territory and have little real-world relevance, IMO. I did just spend a few hours trying to find a better fix than the second patch above, but I was unsuccessful.

As for the last reproducer, you wrote:

"${a[@]}" expands to 0 element if the $a array is empty and "$a" (short for "${a[0]}") expands to one element whether ${a[0]} is set or not.

I don't think that's right:

$ ksh -uc 'typeset -a a=(); echo ${a[0]}' 
ksh: a[0]: parameter not set
$ zsh --emulate ksh -uc 'typeset -a a=(); echo ${a[0]}'
zsh:1: a[0]: parameter not set
$ LANG=C bash -uc 'typeset -a a=(); echo ${a[0]}'
bash: line 1: a[0]: unbound variable

If an array is empty, there is no zeroth element, therefore that element is unset and has no value. Without -u, the empty value is substituted by default (which is where this bug is).

So, I think the current behaviour for the last reproducer is correct . * is a non-empty glob pattern, so it will match the default empty string. This then gets replaced by a string with a back reference, \0, which expands to that empty string.

[stephane-chazelas]

Thanks, now back to unset scalar expansion, that aligns with bash rather than zsh:

$ bash -c 'echo "${a/#/x}"'

$ ksh -c 'echo "${a/#/x}"'

$ zsh -c 'echo "${a/#/x}"'
x

I can live with this. The inconsistencies are deep in obscure corner case territory and have little real-world relevance, IMO. I did just spend a few hours trying to find a better fix than the second patch above, but I was unsuccessful.

I can live with that as well even. It's wrong IMO, but it's less likely one would want to apply a transformation to an unset variable.

As for the last reproducer, you wrote:

"${a[@]}" expands to 0 element if the $a array is empty and "$a" (short for "${a[0]}") expands to one element whether ${a[0]} is set or not.

I don't think that's right:

$ ksh -c 'f() { echo "$# ${#1}"; }; f "$a"'
1 0

f was passed 1 0-length argument, same as when $a is set to an empty string.

$ ksh -c 'f() { echo "$#"; }; f "${a[@]}"'
0

f was passed no argument.

Is the point I was trying to make.

$ ksh -uc 'typeset -a a=(); echo ${a[0]}' 
ksh: a[0]: parameter not set
$ zsh --emulate ksh -uc 'typeset -a a=(); echo ${a[0]}'
zsh:1: a[0]: parameter not set
$ LANG=C bash -uc 'typeset -a a=(); echo ${a[0]}'
bash: line 1: a[0]: unbound variable

If an array is empty, there is no zeroth element, therefore that element is unset and has no value. Without -u, the empty value is substituted by default (which is where this bug is).

nounset is a linting tool to catch programming error, not sure how that's relevant.

So, I think the current behaviour for the last reproducer is correct . * is a non-empty glob pattern, so it will match the default empty string. This then gets replaced by a string with a back reference, \0, which expands to that empty string.

I firmly disagree here, and you agreed earlier that ${a[@]/#/prefix} on an empty array should expand to 0 element.

It's the same here. If you use "${array[@]/*/PREFIX\0SUFFIX}" you want to both prefix and suffix all elements of the array, you don't want it to invent one empty element when the array is empty.

${array[@]/#/prefix} can be written ${array[@]/""/prefix} and ${array[@]/%/suffix} ${array[@]/*/\0suffix} and they current differ in behaviour in the case of an empty array (one with no element).