Announcing: KornShell 93u+m 1.0.0-beta.1
https://github.com/ksh93/ksh

In May 2020, when every KornShell (ksh93) development project was
abandoned, development was rebooted in a new fork based on the last
stable AT&T version: ksh 93u+. Now, one year and hundreds of bug fixes
later, the first beta version is ready, and KornShell lives again. This
new fork is called ksh 93u+m as a permanent nod to its origin; a standard
semantic version number is added starting at 1.0.0-beta.1. Please test
the beta and report any bugs you find, or help us fix known bugs.

Main developers: Martijn Dekker, Johnothan King, hyenias

Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull,
Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Sterling Jensen

HOW TO GET IT

Please download the source code tarball from our GitHub releases page:

https://github.com/ksh93/ksh/releases

To build, follow the instructions in README.md or src/cmd/ksh93/README.
Hopefully, OS/distro packagers will make ksh 93u+m packages available soon.
If you would like to have a binary for your OS from us, ask and we'll try
to build one and add it to the releases page.

HOW TO GET INVOLVED

To report a bug, please open an issue at our GitHub page (see above).
Alternatively, email me at martijn@inlv.org with your report.
To get involved in development, read the brief policy information in
README.md and then jump right in with a pull request or email a patch.
See the TODO file in the top-level directory for a to-do list.

MAIN CHANGES SINCE KSH 93U+ 2012-08-01

Hundreds of bugs have been fixed, including many serious/critical bugs.
This includes upstreamed patches from OpenSUSE, Red Hat, and Solaris, fixes
backported from the abandoned 93v- beta and ksh2020 fork, as well as many
new fixes from the community. See the NEWS file for more information, and
the git commit log for complete documentation of every fix. Incompatible
changes have been minimised, but not at the expense of fixing bugs. For a
list of potentially incompatible changes, see src/cmd/ksh93/COMPATIBILITY.

Though there was a "no new features, bugfixes only" policy, some new
features were found necessary, either to fix serious design flaws or to
complete functionality that was evidently intended, but not finished.
Below is a summary of these new features.

New command line editor features:

- The forward-delete and End keys are now handled as expected in the
emacs and vi built-in line editors.

- In the vi and emacs line editors, repeat count parameters can now also
be used for the arrow keys and the forward-delete key. E.g., in emacs
mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to
the left. In vi control mode, this would be entered as: 7 <left-arrow>.

New shell language features:

- The &>file redirection shorthand (for >file 2>&1) is now available for
all scripts and interactive sessions and not only for profile/login
scripts, bringing ksh 93u+m in line with mksh, bash, and zsh.

- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
never matches the special navigational names '.' (current directory)
and '..' (parent directory). This change makes a pattern like .*
useful; it now matches all hidden files (dotfiles) in the current
directory, without the harmful inclusion of '.' and '..'.

- Tilde expansion can now be extended or modified by defining a
.sh.tilde.get or .sh.tilde.set discipline function. This replaces a
2004 undocumented attempt to add this functionality via a .sh.tilde
command, which never worked and crashed the shell. See the manual for
details on the new method.

New features in built-in commands:

- Usage error messages now show the --help/--man self-documentation options.

- Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by
invoking the canonical path, so the following will now work as expected:
$ /opt/ast/bin/cat --version
version cat (AT&T Research) 2012-05-31

- 'command -x' now looks for external commands only, skipping built-ins.
In addition, its xargs-like functionality no longer freezes the shell on
Linux and macOS, making it effectively a new feature on these systems.

- 'redirect' now checks if all arguments are valid redirections before
performing them. If an error occurs, it issues an error message instead
of terminating the shell.

- 'suspend' now refuses to suspend a login shell, as there is probably no
parent shell to return to and the login session would freeze.

- 'times' now gives high precision output in a POSIX compliant format.

- 'typeset' now gives an informative error message if an incompatible
combination of options is given.

- 'whence -v/-a' now reports the location of autoloadable functions.

New features in shell options:

- A new --globcasedetect shell option is added on OSs where we can
check for a case-insensitive file system (currently Windows/Cygwin,
macOS, Linux and QNX 7.0+). When this option is turned on, file name
generation (globbing), as well as file name tab completion on
interactive shells, automatically become case-insensitive on file
systems where the difference between upper and lower case is ignored
for file names. This is transparently determined for each directory, so
a path pattern that spans multiple file systems can be part
case-sensitive and part case-insensitive.

- A new --nobackslashctrl shell option disables the special escaping
behaviour of the backslash character in the emacs and vi built-in
editors. Particularly in the emacs editor, this makes it much easier to
go backward, insert a forgotten backslash into a command, and then
continue editing without having your next cursor key replace your
backslash with garbage. Note that Ctrl+V (or whatever other character
was set using 'stty lnext') always escapes all control characters in
either editing mode.

- A new --posix shell option has been added to ksh 93u+m that makes the
ksh language more compatible with other shells by following the POSIX
standard more closely. See the manual page for details. It is enabled by
default if ksh is invoked as sh, otherwise it is disabled by default.

- Enhancement to -G/--globstar: symbolic links to directories are now
followed if they match a normal (non-**) glob pattern. For example, if
'/lnk' is a symlink to a directory, '/lnk/**' and '/l?k/**' now work as
you would expect.

Martijn Dekker <martijn@inlv.demon.nl> writes:

> - A new --posix shell option has been added to ksh 93u+m that makes the
> ksh language more compatible with other shells by following the POSIX
> standard more closely. See the manual page for details. It is enabled by
> default if ksh is invoked as sh, otherwise it is disabled by default.

Sounds good. Does that mean that if invoked as "sh" or with "ksh
--posix" it will complain in one way or the other about non-posix stuff
like

if [[ 3 != 4 ]]; then echo true; else echo false; fi

?

From the sh.1 in the github repository I conclude that

echo {a,b}{c,d}

will then output

{a,b}{c,d}

not like, when invoked as "ksh":

ac ad bc bd

Correct?

Thanks for your work and best regards

Axel

Op 10-05-21 om 20:23 schreef Axel Reichert:
> Martijn Dekker <martijn@inlv.demon.nl> writes:
>
>> - A new --posix shell option has been added to ksh 93u+m that makes the
>> ksh language more compatible with other shells by following the POSIX
>> standard more closely. See the manual page for details. It is enabled by
>> default if ksh is invoked as sh, otherwise it is disabled by default.
>
> Sounds good. Does that mean that if invoked as "sh" or with "ksh
> --posix" it will complain in one way or the other about non-posix stuff
> like
>
> if [[ 3 != 4 ]]; then echo true; else echo false; fi
>
> ?

No. The POSIX mode makes the minimum changes necessary to maximise
compatibility with the standard. It does not disable extensions that are
compatible with it. Since the behaviour of '[[' is explicitly
'unspecified' in the standard, shells can do what they want.

> From the sh.1 in the github repository I conclude that
>
> echo {a,b}{c,d}
>
> will then output
>
> {a,b}{c,d}
>
> not like, when invoked as "ksh":
>
> ac ad bc bd
>
> Correct?

Correct. This can also be controlled separately by enabling or disabling
the bracceexpand shell option.

Here's a quote from the man page that explains everything else the posix
option currently changes.

posix Enables the POSIX standard mode for maximum compatibility with
other compliant shells. At the moment that the posix option is
turned on, it also turns on letoctal and turns off -B/braceexpand;
the reverse is done when posix is turned back off. (These options
can still be controlled independently in between.) Furthermore, the
posix option is automatically turned on upon invocation if ksh is
invoked as sh or rsh. In that case, or if the option is turned on
by specifying -o posix on the invocation command line, the invoked
shell will not set the preset aliases even if interactive, and will
not import type attributes for variables (such as integer or read-
only) from the environment.
In addition, while on, the posix option
o disables exporting variable type attributes to the environment
for other ksh processes to import;
o causes file descriptors > 2 to be left open when invoking
another program;
o disables the &> redirection shorthand;
o makes the <> redirection operator default to redirecting stan-
dard input if no file descriptor number precedes it; and
o disables a hack that makes test -t ([ -t ]) equivalent to test
-t 1 ([ -t 1 ]).

Whoops. Already found a small omission there: it still says "such as
integer or read-only"... ksh 93u+m no longer imports the read-only
attribute from the environment in any case (POSIX or not) because there
is no way for a script to unset or otherwise undo that, so it's a
security risk with no way to mitigate it. I'll update the manual for the
next release.

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

On 10.05.2021 19:01, Martijn Dekker wrote:
> Announcing: KornShell 93u+m 1.0.0-beta.1
> https://github.com/ksh93/ksh

My first reflex was to look onto the calendar and see whether we have
April 1st today, but luckily no. :-) Thanks for all your effort!

> In May 2020, when every KornShell (ksh93) development project was
> abandoned, development was rebooted in a new fork based on the last
> stable AT&T version: ksh 93u+. Now, one year and hundreds of bug fixes
> later, the first beta version is ready, and KornShell lives again. This
> new fork is called ksh 93u+m as a permanent nod to its origin; a standard
> semantic version number is added starting at 1.0.0-beta.1. Please test
> the beta and report any bugs you find, or help us fix known bugs.

I just downloaded and started a standard (unmodified) configuration as
described at the git page by cd'ing to the top directory and running
'bin/package make' which runs smoothly quite some time and finally ends

....
/usr/bin/ar: two different operation options specified
mamake [lib/libcmd]: *** exit code 1 making sumlib.o
mamake: *** exit code 1 making lib/libcmd
mamake: *** exit code 1 making all
package: make done at Mon May 10 21:55:49 CEST 2021 in
/home/myuid/ksh93u+m/arch/linux.i386-64

Any idea what's wrong?

Janis

Martijn Dekker <martijn@inlv.demon.nl> writes:

> Since the behaviour of '[[' is explicitly 'unspecified' in the
> standard, shells can do what they want.

O.K., I did not know this. Thanks for the clarification!

> Correct. This can also be controlled separately by enabling or
> disabling the bracceexpand shell option.

Yes, I saw this.

Thanks!

Axel

Op 10-05-21 om 21:05 schreef Janis Papanagnou:
> I just downloaded and started a standard (unmodified) configuration as
> described at the git page by cd'ing to the top directory and running
> 'bin/package make' which runs smoothly quite some time and finally ends
>
> ...
> /usr/bin/ar: two different operation options specified
> mamake [lib/libcmd]: *** exit code 1 making sumlib.o
> mamake: *** exit code 1 making lib/libcmd
> mamake: *** exit code 1 making all
> package: make done at Mon May 10 21:55:49 CEST 2021 in
> /home/myuid/ksh93u+m/arch/linux.i386-64
>
> Any idea what's wrong?

I haven't the foggiest. Several of us have been building it on various
Linux distros without issue. What exact distro, GCC version, binutils
version, C library version, etc. are you using? If I can replicate your
system configuration in a VM, I can investigate the problem.

Thanks,

- Martijn

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

Martijn Dekker <martijn@inlv.demon.nl> writes:
> Op 10-05-21 om 20:23 schreef Axel Reichert:
>> Martijn Dekker <martijn@inlv.demon.nl> writes:
>>
>>> - A new --posix shell option has been added to ksh 93u+m that makes the
>>> ksh language more compatible with other shells by following the POSIX
>>> standard more closely. See the manual page for details. It is enabled by
>>> default if ksh is invoked as sh, otherwise it is disabled by default.
>>
>> Sounds good. Does that mean that if invoked as "sh" or with "ksh
>> --posix" it will complain in one way or the other about non-posix stuff
>> like
>>
>> if [[ 3 != 4 ]]; then echo true; else echo false; fi
>>
>> ?
>
> No. The POSIX mode makes the minimum changes necessary to maximise
> compatibility with the standard. It does not disable extensions that
> are compatible with it. Since the behaviour of '[[' is explicitly
> 'unspecified' in the standard, shells can do what they want.

Just curious, how is the behavior of '[[' *explicitly* unspecified?

There's no mention of '[[' in
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html

You could always have a command by that name in $PATH, but then it seems
to me that '[[' has the same status as 'btfsplk'.

(It's very likely I'm missing something.)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:

> There's no mention of '[[' in
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_04

states

The following words may be recognized as reserved words on some
implementations (when none of the characters are quoted), causing
unspecified results:

[[ ]] function select

Best regards

Axel

On 2021-05-10, Martijn Dekker <martijn@inlv.demon.nl> wrote:
> Op 10-05-21 om 21:05 schreef Janis Papanagnou:
>> I just downloaded and started a standard (unmodified) configuration as
>> described at the git page by cd'ing to the top directory and running
>> 'bin/package make' which runs smoothly quite some time and finally ends
>>
>> ...
>> /usr/bin/ar: two different operation options specified
>> mamake [lib/libcmd]: *** exit code 1 making sumlib.o
>> mamake: *** exit code 1 making lib/libcmd
>> mamake: *** exit code 1 making all
>> package: make done at Mon May 10 21:55:49 CEST 2021 in
>> /home/myuid/ksh93u+m/arch/linux.i386-64
>>
>> Any idea what's wrong?
>
> I haven't the foggiest. Several of us have been building it on various

The build is likely feeding duplicate commands to ar, maybe via some
combination of directly and from some AR options variable:

These combinations repro the error with the GNU Binutils 2.30 ar:

$ ar -rc -rc foo.ar file
ar: two different operation options specified

$ ar rc -rc foo.ar file
ar: two different operation options specified

These combinations don't:

$ ar rc rc foo.ar file
$ ar -rc rc foo.ar file

Here, the archive is just assumed to be "rc".

In other words, GNU ar is diagnosing the issue of the archive looking
like a clump of options, even if it follows a non-option argument.
But it's diagnosed only if it starts with a dash.

Or something like that.

Axel Reichert <mail@axel-reichert.de> writes:
> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>
>> There's no mention of '[[' in
>> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_04
>
> states
>
> The following words may be recognized as reserved words on some
> implementations (when none of the characters are quoted), causing
> unspecified results:
>
> [[ ]] function select

Thanks.

Following that, it says:

Words that are the concatenation of a name and a <colon> ( ':' ) are
reserved; their use produces unspecified results.

Are there examples of that syntax being used by some shell?

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */

On 11.05.2021 02:29, Keith Thompson wrote:
>
> Following that, it says:
>
> Words that are the concatenation of a name and a <colon> ( ':' ) are
> reserved; their use produces unspecified results.
>
> Are there examples of that syntax being used by some shell?

My faint memories seem to recall a conversation with David Korn
many years ago where he pondered about a goto-label syntax,
something like

label:
...
break label

It seems this feature had been dismissed (unless I am just
misremembering in the first place).

Do other POSIX-shells support such a feature?

Janis

On 10.05.2021 23:43, Martijn Dekker wrote:
> Op 10-05-21 om 21:05 schreef Janis Papanagnou:
>> I just downloaded and started a standard (unmodified) configuration as
>> described at the git page by cd'ing to the top directory and running
>> 'bin/package make' which runs smoothly quite some time and finally ends
>>
>> ...
>> /usr/bin/ar: two different operation options specified
>> mamake [lib/libcmd]: *** exit code 1 making sumlib.o
>> mamake: *** exit code 1 making lib/libcmd
>> mamake: *** exit code 1 making all
>> package: make done at Mon May 10 21:55:49 CEST 2021 in
>> /home/myuid/ksh93u+m/arch/linux.i386-64
>>
>> Any idea what's wrong?
>
> I haven't the foggiest. Several of us have been building it on various
> Linux distros without issue. What exact distro, GCC version, binutils
> version, C library version, etc. are you using?

I understand the reflex to ask for every detail of my standard distro
and version of every tool (including the final "etc." tool/library you
mention) but isn't the first line of the posted error log sufficient?

/usr/bin/ar: two different operation options specified

Myself I would try yet another build-run on another Linux platform to
see whether I get the same result. (I'll inform you about that later.)

Thanks.

Janis

> If I can replicate your
> system configuration in a VM, I can investigate the problem.
>
> Thanks,
>
> - Martijn
>

On 2021-05-11, Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Axel Reichert <mail@axel-reichert.de> writes:
>> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>>
>>> There's no mention of '[[' in
>>> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html
>>
>> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_04
>>
>> states
>>
>> The following words may be recognized as reserved words on some
>> implementations (when none of the characters are quoted), causing
>> unspecified results:
>>
>> [[ ]] function select
>
> Thanks.
>
> Following that, it says:
>
> Words that are the concatenation of a name and a <colon> ( ':' ) are
> reserved; their use produces unspecified results.
>
> Are there examples of that syntax being used by some shell?

Even if there aren't, here is hypothesis: maybe some shells historically
parsed the : command separately even if it was glued to a previous word,
similarly to how, say tokens like & or ; are treated.

So that is to say, imagine that some shell had historically allowed

if true; then
:
fi

which, on one line is:

if foo; then : ; fi

and can be condensed to

if foo;then :;fi

to be further condensed to this:

if foo;then:;fi

The rule which says "if you glue : to a word, the results are
unspecified" is then portability advice.

When we feed the above to Bash, we get:

bash: syntax error near unexpected token `fi'

and that can be regarded as a consequence of "unspecified result".

When the command works as expected on that hypothetical shell, that is
also an "unspecified result".

If that's the reason, it's a bad one; it's okay to let deviant old cruft
become nonconforming. Sometimes such rules are justifiable as
transitional measures.

Maybe that's what this is; just nobody remembered to remove it.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On 11.05.2021 13:04, Janis Papanagnou wrote:
>>> ...
>>> /usr/bin/ar: two different operation options specified
>>> mamake [lib/libcmd]: *** exit code 1 making sumlib.o
>>> mamake: *** exit code 1 making lib/libcmd
>>> mamake: *** exit code 1 making all
>>> package: make done at Mon May 10 21:55:49 CEST 2021 in
>>> /home/myuid/ksh93u+m/arch/linux.i386-64
>
> Myself I would try yet another build-run on another Linux platform to
> see whether I get the same result. (I'll inform you about that later.)

On a newer version of the Ubuntu it compiles to the end without errors.
The shell executable runs. An annoying tilde completion bug in ksh93u+
(my first test case) seems to work correctly in ksh93u+m. Great. Thanks
again!

Janis