Announcing: KornShell 93u+m 1.0.0-beta.2
https://github.com/ksh93/ksh

In May 2020, when every KornShell (ksh93) development project was
abandoned, development was rebooted in a new fork based on the last
stable AT&T version: ksh 93u+. This new fork is called ksh 93u+m as a
permanent nod to its origin. We're restarting it at version 1.0. Seven
months after the first beta, the second one is ready. Please test this
second beta and report any bugs you find, or help us fix known bugs.

We're now the default ksh93 in some OS distributions, at least Debian
and Slackware! Even though we don't think it's stable release quality
yet, the consensus seems to be that 93u+m is already much better than
the last AT&T release.

Main developers: Martijn Dekker, Johnothan King, hyenias

Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull,
Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Ryan Schmidt,
Sterling Jensen

HOW TO GET IT

Please download the source code tarball from our GitHub releases page:

https://github.com/ksh93/ksh/releases

To build, follow the instructions in README.md or src/cmd/ksh93/README.

HOW TO GET INVOLVED

To report a bug, please open an issue at our GitHub page (see above).
Alternatively, email me at martijn@inlv.org with your report.
To get involved in development, read the brief policy information in
README.md and then jump right in with a pull request or email a patch.
See the TODO file in the top-level directory for a to-do list.

### MAIN CHANGES between 1.0.0-beta.1 and 1.0.0-beta.2 ###

New features in built-in commands:

- 'cd' now supports an -e option that, when combined with -P, verifies
that $PWD is correct after changing directories; this helps detect
access permission problems. See:
https://www.austingroupbugs.net/view.php?id=253

- 'printf' now supports a -v option as in bash. This assigns formatted
output directly to variables, which is very fast and will not strip
final newline (\n) characters.

- The 'return' command, when used to return from a function, can now
return any status value in the 32-bit signed integer range, like on
zsh. However, due to a traditional Unix kernel limitation, $? is
still trimmed to its least significant 8 bits whenever leaving a
(sub)shell environment.

- 'test'/'[' now supports all the same operators as [[ (including =~,
\<, \>) except for the different 'and'/'or' operators. Note that
'test'/'[' remains deprecated due to its unfixable pitfalls;
[[ ... ]] is recommended instead.

Shell language changes:

- Several improvements were made to the --noexec shell code linter.

- Arithmetic expressions in native ksh mode no longer interpret a
number with a leading zero as octal in any context. Use 8#octalnumber
instead (e.g. 8#400 == 256). Arithmetic expressions now also behave
identically within and outside ((...)) and $((...)).

- POSIX compatibility mode fixes (only applicable with the --posix shell
option on):
- A leading zero is now consistently recognised as introducing an octal
number in all arithmetic contexts.
- $((inf)) and $((nan)) are now interpreted as regular variables.
- The '.' built-in no longer runs ksh functions and now only runs
files.

Bugs fixed:

- '.' and '..' are now once again completed by tab completion.

- If SIGINT is set to ignore, the interactive shell no longer exits on
Ctrl+C.

- ksh now builds and runs on Apple's new M1 hardware.

- The 'return' and 'exit' commands no longer risk triggering actual
signals by returning or exiting with a status > 256.

- Ksh no longer behaves badly when parsing a type definition command
('typeset -T' or 'enum') without executing it or when executing it in
a subshell. Types can now safely be defined in subshells and defined
conditionally as in 'if condition; then enum ...; fi'.

- Discipline functions, especially those applied to PS2 or .sh.tilde,
will no longer crash your shell upon being interrupted or throwing an
error.

- Fixed a bug that could corrupt output if standard output is closed
upon initialising the shell.

- Fixed a bug in the [[ ... ]] compound command: the '!' logical
negation operator now correctly negates another '!', e.g.,
[[ ! ! 1 -eq 1 ]] now returns 0/true. Note that this has always been
the case for 'test'/'['.

- Fixed SHLVL so that replacing ksh by itself (exec ksh) will not
increase it.

- Arithmetic expressions are no longer allowed to assign out-of-range
values to variables of types declared with enum.

- The 'time' keyword no longer makes the --errexit shell option
ineffective.

- Various bugs in libcmd built-in commands (those bound to the
/opt/ast/bin path by default) have been fixed.

- Various other crashing bugs have been fixed.

Fixes for the shcomp byte code compiler:

- shcomp is now able to compile scripts that define types using enum.

- shcomp now refuses to mess up your terminal by writing bytecode
to it.

### MAIN CHANGES between ksh 93u+ 2012-08-01 and 93u+m 1.0.0-beta.1 ###

Hundreds of bugs have been fixed, including many serious/critical bugs.
This includes upstreamed patches from OpenSUSE, Red Hat, and Solaris, fixes
backported from the abandoned 93v- beta and ksh2020 fork, as well as many
new fixes from the community. See the NEWS file for more information, and
the git commit log for complete documentation of every fix. Incompatible
changes have been minimised, but not at the expense of fixing bugs. For a
list of potentially incompatible changes, see src/cmd/ksh93/COMPATIBILITY.

Though there was a "no new features, bugfixes only" policy, some new
features were found necessary, either to fix serious design flaws or to
complete functionality that was evidently intended, but not finished.
Below is a summary of these new features.

New command line editor features:

- The forward-delete and End keys are now handled as expected in the
emacs and vi built-in line editors.

- In the vi and emacs line editors, repeat count parameters can now also
be used for the arrow keys and the forward-delete key. E.g., in emacs
mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to
the left. In vi control mode, this would be entered as: 7 <left-arrow>.

New shell language features:

- The &>file redirection shorthand (for >file 2>&1) is now available for
all scripts and interactive sessions and not only for profile/login
scripts, bringing ksh 93u+m in line with mksh, bash, and zsh.

- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
never matches the special navigational names '.' (current directory)
and '..' (parent directory). This change makes a pattern like .*
useful; it now matches all hidden files (dotfiles) in the current
directory, without the harmful inclusion of '.' and '..'.

- Tilde expansion can now be extended or modified by defining a
.sh.tilde.get or .sh.tilde.set discipline function. This replaces a
2004 undocumented attempt to add this functionality via a .sh.tilde
command, which never worked and crashed the shell. See the manual for
details on the new method.

New features in built-in commands:

- Usage error messages now show the --help/--man self-documentation options.

- Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by
invoking the canonical path, so the following will now work as expected:
$ /opt/ast/bin/cat --version
version cat (AT&T Research) 2012-05-31

- 'command -x' now looks for external commands only, skipping built-ins.
In addition, its xargs-like functionality no longer freezes the shell on
Linux and macOS, making it effectively a new feature on these systems.

- 'redirect' now checks if all arguments are valid redirections before
performing them. If an error occurs, it issues an error message instead
of terminating the shell.

- 'suspend' now refuses to suspend a login shell, as there is probably no
parent shell to return to and the login session would freeze.

- 'times' now gives high precision output in a POSIX compliant format.

- 'typeset' now gives an informative error message if an incompatible
combination of options is given.

- 'whence -v/-a' now reports the location of autoloadable functions.

New features in shell options:

- A new --globcasedetect shell option is added on OSs where we can
check for a case-insensitive file system (currently Windows/Cygwin,
macOS, Linux and QNX 7.0+). When this option is turned on, file name
generation (globbing), as well as file name tab completion on
interactive shells, automatically become case-insensitive on file
systems where the difference between upper and lower case is ignored
for file names. This is transparently determined for each directory, so
a path pattern that spans multiple file systems can be part
case-sensitive and part case-insensitive.

On 17.12.2021 04:38, Martijn Dekker wrote:
> Announcing: KornShell 93u+m 1.0.0-beta.2
> https://github.com/ksh93/ksh

Great, thanks!

I'll have a look into the details soon, but this one irritated me;
but maybe I just misunderstood it:

> - The 'return' command, when used to return from a function, can now
> return any status value in the 32-bit signed integer range, like on
> zsh. However, due to a traditional Unix kernel limitation, $? is
> still trimmed to its least significant 8 bits whenever leaving a
> (sub)shell environment.

If in Kornshell I terminate a process with a signal I get a number
returned that is beyond the 8 bit range; e.g. a kill (with SIGTERM)
results in $? evaluating to 271.

Janis

Op 17-12-21 om 09:43 schreef Janis Papanagnou:
> I'll have a look into the details soon, but this one irritated me;
> but maybe I just misunderstood it:
>
>> - The 'return' command, when used to return from a function, can now
>> return any status value in the 32-bit signed integer range, like on
>> zsh. However, due to a traditional Unix kernel limitation, $? is
>> still trimmed to its least significant 8 bits whenever leaving a
>> (sub)shell environment.
>
> If in Kornshell I terminate a process with a signal I get a number
> returned that is beyond the 8 bit range; e.g. a kill (with SIGTERM)
> results in $? evaluating to 271.

Yes, that has not changed.

If a child process of ksh's is terminated by a signal, ksh doesn't
inherit the exit status via the kernel as that process didn't exit
normally. Instead, ksh detects that the process was terminated by a
signal (see 'man 2 wait' for how that works) and sets $? to the signal
number with the 9th bit set in the current process -- which is not
subject to the kernel's 8-bit exit status limitation as that all happens
in the parent process.

What the announcement item above means instead is this:

$ myfunc() { return 12345; }
$ myfunc; echo $?
12345
$ (myfunc); echo $?
57
$ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
57

i.e., the exit status of a subshell or child shell that invokes
'myfunc', and does nothing to overwrite $? before exiting, is the least
significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.

For consistency's sake, ksh 93u+m enforces this for virtual
(non-forking) subshells as well, because a script should never be able
to notice a difference between a virtual subshell and a subshell that
forked a separate process due to executing something like 'ulimit'.

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

On Fri, 17 Dec 2021 11:18:26 +0100
Martijn Dekker <martijn@inlv.demon.nl> wrote:
> What the announcement item above means instead is this:
>
> $ myfunc() { return 12345; }
> $ myfunc; echo $?
> 12345
> $ (myfunc); echo $?
> 57
> $ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
> 57
>
> i.e., the exit status of a subshell or child shell that invokes
> 'myfunc', and does nothing to overwrite $? before exiting, is the least
> significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.

For the least significant 8 bits of 12345 the calculation you want is
12345 % 256 == 57 .Regardless , 12345 % 0xFF == 105 .

In article <JGDDG6C28aBHRfrQ6@bongo-ra.co>,
Spiros Bousbouras <spibou@gmail.com> wrote:
>On Fri, 17 Dec 2021 11:18:26 +0100
>Martijn Dekker <martijn@inlv.demon.nl> wrote:
>> What the announcement item above means instead is this:
>>
>> $ myfunc() { return 12345; }
>> $ myfunc; echo $?
>> 12345
>> $ (myfunc); echo $?
>> 57
>> $ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
>> 57
>>
>> i.e., the exit status of a subshell or child shell that invokes
>> 'myfunc', and does nothing to overwrite $? before exiting, is the least
>> significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.
>
>For the least significant 8 bits of 12345 the calculation you want is
>12345 % 256 == 57 .Regardless , 12345 % 0xFF == 105 .

If you want to use 255 (i.e., hex FF), then you have to use an 'and'
operator.

i.e., either of:

12345 % 256

or:

12345 & 255

--
If the automobile had followed the same development cycle as the
computer, a Rolls-Royce today would cost $100, get a million miles to
the gallon, and explode once every few weeks, killing everyone inside.

Op 17-12-21 om 12:50 schreef Spiros Bousbouras:
> On Fri, 17 Dec 2021 11:18:26 +0100
> Martijn Dekker <martijn@inlv.demon.nl> wrote:
>> What the announcement item above means instead is this:
>>
>> $ myfunc() { return 12345; }
>> $ myfunc; echo $?
>> 12345
>> $ (myfunc); echo $?
>> 57
>> $ ksh -c 'myfunc() { return 12345; }; myfunc'; echo $?
>> 57
>>
>> i.e., the exit status of a subshell or child shell that invokes
>> 'myfunc', and does nothing to overwrite $? before exiting, is the least
>> significant 8 bits of 12345, i.e., 12345 % 0xFF == 210.
>
> For the least significant 8 bits of 12345 the calculation you want is
> 12345 % 256 == 57 .Regardless , 12345 % 0xFF == 105 .

Indeed. Sorry. I was travelling and insufficiently caffeinated while
writing that. What I *meant* was: 12345 & 0xFF == 57

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

On 17.12.2021 14:10, Kenny McCormack wrote:
>
> If you want to use 255 (i.e., hex FF), then you have to use an 'and'
> operator.
> i.e., either of:
> 12345 % 256
> or:
> 12345 & 255

Yep, just a typical well known careless mistake. In the same vein
as implementing the integer division
x div 2 # x/2
by
x shr 1 # x>>1
but writing by accident x>>2 (a careless mistake I made 35 years
ago in a test and still recall); a favorable reader will understand
the intention, but a computer will merciless execute it.

Janis

On 17.12.2021 04:38, Martijn Dekker wrote:
> Announcing: KornShell 93u+m 1.0.0-beta.2
> https://github.com/ksh93/ksh
> [...]
> Please download the source code tarball from our GitHub releases page:
>
> https://github.com/ksh93/ksh/releases
>
> To build, follow the instructions in README.md or src/cmd/ksh93/README.

I tried a build of the shell, and after a long seamless running compile
got an error. (I seem to recall that I detected it already in a previous
release.)

Note: in the logs below I replaced the actual top level directory by ~ .

+ ~/ksh/arch/linux.i386-64/bin/ar x -lsum sumlib.o
/usr/bin/ar: two different operation options specified
mamake [lib/libcmd]: *** exit code 1 making sumlib.o
mamake: *** exit code 1 making lib/libcmd
mamake: *** exit code 1 making all

Got the usage info from the packaged ar (no idea where the -U thing
comes from, and why a different path for ar is displayed)

~/ksh/arch/linux.i386-64/bin/ar --help
/usr/bin/ar: invalid option -- 'U'
Usage: /usr/bin/ar [emulation options] [-]{dmpqrstx}[abcDfilMNoPsSTuvV]
[--plugin <name>] [member-name] [count] archive-file file...
/usr/bin/ar -M [<mri-script]

but it seems the call, instead of
ar x -lsum sumlib.o
should probably rather be something like
ar xl sum sumlib.o

An individual call reproduces the error

~/ksh/arch/linux.i386-64/bin/ar x -lsum sumlib.o
/usr/bin/ar: two different operation options specified

while changing the separately started call just complains about the file

~/ksh/arch/linux.i386-64/bin/ar xl sum sumlib.o
/usr/bin/ar: sum: No such file or directory

(which is understandable, since I wasn't executing the test call from
within the build system, probably being in a wrong path or environment
variables may be unset).

Is there any fix for that?

Janis

Op 18-12-21 om 14:07 schreef Janis Papanagnou:
> On 17.12.2021 04:38, Martijn Dekker wrote:
>> Announcing: KornShell 93u+m 1.0.0-beta.2
>> https://github.com/ksh93/ksh
>> [...]
>> Please download the source code tarball from our GitHub releases page:
>>
>> https://github.com/ksh93/ksh/releases
>>
>> To build, follow the instructions in README.md or src/cmd/ksh93/README.
>
> I tried a build of the shell, and after a long seamless running compile
> got an error. (I seem to recall that I detected it already in a previous
> release.)

Sorry, I'd forgotten about that.

> Note: in the logs below I replaced the actual top level directory by ~ .
>
> + ~/ksh/arch/linux.i386-64/bin/ar x -lsum sumlib.o
> /usr/bin/ar: two different operation options specified
> mamake [lib/libcmd]: *** exit code 1 making sumlib.o
> mamake: *** exit code 1 making lib/libcmd
> mamake: *** exit code 1 making all
>
> Got the usage info from the packaged ar (no idea where the -U thing
> comes from, and why a different path for ar is displayed)
>
> ~/ksh/arch/linux.i386-64/bin/ar --help
> /usr/bin/ar: invalid option -- 'U'
> Usage: /usr/bin/ar [emulation options] [-]{dmpqrstx}[abcDfilMNoPsSTuvV]
> [--plugin <name>] [member-name] [count] archive-file file...
> /usr/bin/ar -M [<mri-script]
[...]

The cause is probably this ar wrapper script:

$ cat src/cmd/INIT/ar.linux.i386-64

: linux.i386-64 ar wrapper

case $1 in
*x*) /usr/bin/ar "$@" ;;
*) /usr/bin/ar U"$@" ;;
esac

This gets copied into your package hierarchy as
arch/linux.i386-64/bin/ar and put in $PATH.

If you delete the wrapper script in both places

$ rm src/cmd/INIT/ar.linux.i386-64 arch/linux.i386-64/bin/ar

and then try 'bin/package make' again, does it build?

I have no idea what that U thing is for. The AT&T folks must have had
some reason for adding it.

'man ar' on CentOS x86_64 documents:

A number of modifiers (mod) may immediately follow the p
keyletter, to specify variations on an operation's behavior:
[...]
D Operate in deterministic mode. When adding files and the
archive index use zero for UIDs, GIDs, timestamps, and use
consistent file modes for all files. When this option is
used, if ar is used with identical options and identical
input files, multiple runs will create identical output
files regardless of the input files' owners, groups, file
modes, or modification times.
[...]
U Do not operate in deterministic mode. This is the inverse
of the D modifier, above: added files and the archive index
will get their actual UID, GID, timestamp, and file mode
values.

This is the default unless binutils was configured with
--enable-deterministic-archives.

So the question is, why would we want to disable deterministic mode? It
sounds like a good thing.

If you can confirm that deleting the wrapper script fixes your build
failure, I'll get rid of it and see if anything breaks for anyone else.

(In fact in the long term I plan to get rid of all such wrapper scripts.
It's a very flawed way of making a build work. Just overriding the
default compiler or linker using the appropriate failures will disable
the wrappers so the necessary flags aren't added. There are better ways
of doing this.)

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

Op 19-12-21 om 21:00 schreef Martijn Dekker:
> If you can confirm that deleting the wrapper script fixes your build
> failure, I'll get rid of it and see if anything breaks for anyone else.

In the meantime I got rid of that wrapper script and some others on the
1.0 branch: https://github.com/ksh93/ksh/commit/24fc1bbc

--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh

On 19.12.2021 22:00, Martijn Dekker wrote:
> [...]
> If you delete the wrapper script in both places
>
> $ rm src/cmd/INIT/ar.linux.i386-64 arch/linux.i386-64/bin/ar
>
> and then try 'bin/package make' again, does it build?

I'll try this and also your github changes after the holidays.

Thank you!

Janis