Continuing from the other thread which got hijacked into talking about
binfmt_misc...

I find this binfmt_misc stuff very interesting and intriguing, but I
question whether it is actually of any real value. It seems pretty clear
that it was implemented in order to make WINE work sensibly - that is, to
be able to just type in the name of a DOS/Windows executable and have it
work like it does in DOS/Windows. Nothing wrong with that, of course...

But, that said, I have a question. On one system, I find:

$ nl /proc/sys/fs/binfmt_misc/ksh
1 enabled
2 interpreter /bin/ksh
3 flags:
4 offset 0
5 magic 0b1308

What exactly does the "magic" value 0b1308 mean? According to the docs,
since there is no \x, this is a literal string. But it sure looks like a
hex string, doesn't it?

And what would 0b1308 mean if it was a hex string? Not a string likely to
be found in a ksh script in either case.

Also, why setup a special entry for ksh scripts anyway? Aren't ksh scripts
already directly executable via the usual #! mechanism?

--
Politics is show business for ugly people.

Sports is politics for stupid people.

On Mon, 20 Sep 2021 12:34:25 +0000, Kenny McCormack wrote:

> Continuing from the other thread which got hijacked into talking about
> binfmt_misc...
>
> I find this binfmt_misc stuff very interesting and intriguing, but I
> question whether it is actually of any real value. It seems pretty clear
> that it was implemented in order to make WINE work sensibly - that is, to
> be able to just type in the name of a DOS/Windows executable and have it
> work like it does in DOS/Windows. Nothing wrong with that, of course...
>
> But, that said, I have a question. On one system, I find:
>
> $ nl /proc/sys/fs/binfmt_misc/ksh
> 1 enabled
> 2 interpreter /bin/ksh
> 3 flags:
> 4 offset 0
> 5 magic 0b1308
>
> What exactly does the "magic" value 0b1308 mean? According to the docs,
> since there is no \x, this is a literal string. But it sure looks like a
> hex string, doesn't it?

It /is/ a hex string. My example of
echo :audio:M::RIFF::/usr/bin/paplay: >/proc/sys/fs/binfmt_misc/register
produces
# nl /proc/sys/fs/binfmt_misc/audio
1 enabled
2 interpreter /usr/bin/paplay
3 flags:
4 offset 0
5 magic 52494646
and 0x52 = ascii 'R'
0x49 = ascii 'I'
0x46 = ascii 'F'

> And what would 0b1308 mean if it was a hex string? Not a string likely to
> be found in a ksh script in either case.

Remember, binfmt_misc allows you to associate an "interpreter" to a magic number.
There's no reason that, for the purposes of whomever set up your ksh binfmt entry,
the magic number 0x0b1308 /cant/ trigger ksh scripts.

What would ksh do if it encountered a line beginning with a Vertical Tab (0x0b),
a DC3 (0x13), and a Backspace (0x08)?

> Also, why setup a special entry for ksh scripts anyway?

I can only guess that, for some system purpose, whomever sourced that binfmt_misc
entry didn't want to use shebangs in certain scripts. Perhaps to disguise certain
ksh scripts so that file(1) doesn't report them as scripts?

> Aren't ksh scripts
> already directly executable via the usual #! mechanism?

Yes, so.... obfuscation?

--
Lew Pitcher
"In Skills, We Trust"

In article <sia3f8$gcm$1@dont-email.me>,
Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
....
>> What exactly does the "magic" value 0b1308 mean? According to the docs,
>> since there is no \x, this is a literal string. But it sure looks like a
>> hex string, doesn't it?
>
>It /is/ a hex string. My example of
> echo :audio:M::RIFF::/usr/bin/paplay: >/proc/sys/fs/binfmt_misc/register

Thanks for confirming that. It pretty much had to be.

>> And what would 0b1308 mean if it was a hex string? Not a string likely to
>> be found in a ksh script in either case.
>
>Remember, binfmt_misc allows you to associate an "interpreter" to a
>magic number. There's no reason that, for the purposes of whomever set
>up your ksh binfmt entry, the magic number 0x0b1308 /cant/ trigger ksh
>scripts.

But no good reason why it should, either.

>What would ksh do if it encountered a line beginning with a Vertical Tab (0x0b),
>a DC3 (0x13), and a Backspace (0x08)?
>
>> Also, why setup a special entry for ksh scripts anyway?
>
>I can only guess that, for some system purpose, whomever sourced that binfmt_misc
>entry didn't want to use shebangs in certain scripts. Perhaps to disguise certain
>ksh scripts so that file(1) doesn't report them as scripts?

Yup. And, yet, it is.

>> Aren't ksh scripts
>> already directly executable via the usual #! mechanism?
>
>Yes, so.... obfuscation?

Well, I think that ksh might have some kind of compiled mode script kinda
thing. I don't know much about ksh. Maybe one of the ksh experts on the
group will chime in.

On that same system, there is also a similarly bizarre setup for Python. I
think Python might also have some kind of compiled-script thing, that needs
to be triggered like this.

In any case, the character sequence represented by 0b1308 is indeed
strange, and very unlikely to be found in any sort of text file.

--
Men rarely (if ever) manage to dream up a God superior to themselves.
Most Gods have the manners and morals of a spoiled child.

On 2021-09-20, Kenny McCormack <gazelle@shell.xmission.com> wrote:
> Continuing from the other thread which got hijacked into talking about
> binfmt_misc...
>
> I find this binfmt_misc stuff very interesting and intriguing, but I
> question whether it is actually of any real value.

For example, with binfmt_misc, in combination with QEMU, you can run an
ARM executable on an x86 box. If you have an ARM sysroot for it that
it can chroot into, then it will have a complete environment with shared
libs and all.

You can use this to create an ARM build environment for compiling
programs for ARM even if they don't support cross-compiling; they think
they are being built on an ARM box.

It's pretty useful.

> that it was implemented in order to make WINE work sensibly - that is, to
> be able to just type in the name of a DOS/Windows executable and have it
> work like it does in DOS/Windows. Nothing wrong with that, of course...
>
> But, that said, I have a question. On one system, I find:
>
> $ nl /proc/sys/fs/binfmt_misc/ksh
> 1 enabled
> 2 interpreter /bin/ksh
> 3 flags:
> 4 offset 0
> 5 magic 0b1308

This is for compiled Korn shell scripts. ksh93 has a command called shcomp for
compiling shell scripts:

The man page for this claims that it is Copyright 1982-2012 David Korn,
so this thing has a long history.

0:[0923:214942]:sun-go:~$ ksh
$ shcomp | od -tx1 # compile standard input and hex dump compiled output
echo 42
0000000 0b 13 08 00 03 00 00 40 00 02 05 65 63 68 6f 03
0000020 34 32 00 01
0000024

As you can see, shcomp cranks out this byte sequence.

However, the byte sequence occurs nowhere in the ksh93 executable
as a byte sequence. TXR Lisp:

1> (search (file-get-buf "/bin/ksh93") #b'0b1308')
nil
2> (search (file-get-buf "/bin/ksh93") #b'08130b')
nil

We try it in both byte orders, just in case; no hit either way.

It's probably encoded into code, along the lines of this:

/* recognition */
if (byte[0] == 0x08 && byte[1] == 0x13 ....) ...

/* generation */
putc(0x08, outfile);
putc(0x13, outfile);
putc(0x0D, outfile);

C code like this will very unlikely compile to an image which
contains those three numbers as a byte string.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On 2021-09-20, Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> On Mon, 20 Sep 2021 12:34:25 +0000, Kenny McCormack wrote:
>
>> Continuing from the other thread which got hijacked into talking about
>> binfmt_misc...
>>
>> I find this binfmt_misc stuff very interesting and intriguing, but I
>> question whether it is actually of any real value. It seems pretty clear
>> that it was implemented in order to make WINE work sensibly - that is, to
>> be able to just type in the name of a DOS/Windows executable and have it
>> work like it does in DOS/Windows. Nothing wrong with that, of course...
>>
>> But, that said, I have a question. On one system, I find:
>>
>> $ nl /proc/sys/fs/binfmt_misc/ksh
>> 1 enabled
>> 2 interpreter /bin/ksh
>> 3 flags:
>> 4 offset 0
>> 5 magic 0b1308
>>
>> What exactly does the "magic" value 0b1308 mean? According to the docs,
>> since there is no \x, this is a literal string. But it sure looks like a
>> hex string, doesn't it?
>
> It /is/ a hex string. My example of
> echo :audio:M::RIFF::/usr/bin/paplay: >/proc/sys/fs/binfmt_misc/register
> produces
> # nl /proc/sys/fs/binfmt_misc/audio
> 1 enabled
> 2 interpreter /usr/bin/paplay
> 3 flags:
> 4 offset 0
> 5 magic 52494646
> and 0x52 = ascii 'R'
> 0x49 = ascii 'I'
> 0x46 = ascii 'F'
>
>> And what would 0b1308 mean if it was a hex string? Not a string likely to
>> be found in a ksh script in either case.
>
> Remember, binfmt_misc allows you to associate an "interpreter" to a magic number.
> There's no reason that, for the purposes of whomever set up your ksh binfmt entry,
> the magic number 0x0b1308 /cant/ trigger ksh scripts.
>
> What would ksh do if it encountered a line beginning with a Vertical Tab (0x0b),
> a DC3 (0x13), and a Backspace (0x08)?
>
>> Also, why setup a special entry for ksh scripts anyway?
>
> I can only guess that, for some system purpose, whomever sourced that binfmt_misc
> entry didn't want to use shebangs in certain scripts. Perhaps to disguise certain
> ksh scripts so that file(1) doesn't report them as scripts?
>
>> Aren't ksh scripts
>> already directly executable via the usual #! mechanism?
>
> Yes, so.... obfuscation?

If you were implementing a shell compiler today, you probably wouldn't
think twice of just using hash bang. This could reference a different
binary, or the same one, but with some additional hacks to distinguish
the compiled code.

Evidently, David Korn's shcomp thing has been around since 1982.

Probably, one motivation was to make scripts as small as possible. Hard
disks were so tiny. Encoding the interpreter path into every program is
at odds with "as small as possible".

Furthermore, with a custom magic number, you can assume hard-coded,
fixed offsets for any elements in the area that follows the magic
number, making it easy to process. Or, at the very least, you can
assume a fixed offset for whatever follows. Hash bang means the
interpreter has to scan for a newline to skip the hash bang line,
before doing anything else.

You can pad: say, start your payload 32 bytes from the start of the
file, so the newline-terminated hash bang line has to fit into this 32
bytes (followed by padding nulls, say, to bring it to 32). That also
goes against the spirit of making the files as small as possible.

In article <20210923213007.701@kylheku.com>,
Kaz Kylheku <480-992-1380@kylheku.com> wrote:
....
>This is for compiled Korn shell scripts. ksh93 has a command called shcomp for
>compiling shell scripts:

Yeah, that's what I thought. (See OP)

Thanks for the writeup.

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/Windows

Kaz Kylheku <480-992-1380@kylheku.com> writes:
[...]
> This is for compiled Korn shell scripts. ksh93 has a command called shcomp for
> compiling shell scripts:
>
> The man page for this claims that it is Copyright 1982-2012 David Korn,
> so this thing has a long history.
>
> 0:[0923:214942]:sun-go:~$ ksh
> $ shcomp | od -tx1 # compile standard input and hex dump compiled output
> echo 42
> 0000000 0b 13 08 00 03 00 00 40 00 02 05 65 63 68 6f 03
> 0000020 34 32 00 01
> 0000024
>
> As you can see, shcomp cranks out this byte sequence.
>
> However, the byte sequence occurs nowhere in the ksh93 executable
> as a byte sequence. TXR Lisp:
>
> 1> (search (file-get-buf "/bin/ksh93") #b'0b1308')
> nil
> 2> (search (file-get-buf "/bin/ksh93") #b'08130b')
> nil
>
> We try it in both byte orders, just in case; no hit either way.
>
> It's probably encoded into code, along the lines of this:
>
> /* recognition */
> if (byte[0] == 0x08 && byte[1] == 0x13 ....) ...
>
> /* generation */
> putc(0x08, outfile);
> putc(0x13, outfile);
> putc(0x0D, outfile);
>
> C code like this will very unlikely compile to an image which
> contains those three numbers as a byte string.

In ksh's parse.c:

if( cp[0]==CNTL('k') && cp[1]==CNTL('s') && cp[2]==CNTL('h') && cp[3]==0)

In shcomp.c:

#define CNTL(x) ((x)&037)
static const char header[6] = { CNTL('k'),CNTL('s'),CNTL('h'),0,SHCOMP_HDR_VERSION,0 };
...
if(!dflag)
sfwrite(out,header,sizeof(header));

so the byte sequence is likely to appear in the shcomp binary
executable.

And yes, it's Ctrl-K, Ctrl-S, Ctrl-H.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

On Fri, 24 Sep 2021 04:56:30 +0000, Kaz Kylheku wrote:

> On 2021-09-20, Kenny McCormack <gazelle@shell.xmission.com> wrote:
>> Continuing from the other thread which got hijacked into talking about
>> binfmt_misc...
>>
>> I find this binfmt_misc stuff very interesting and intriguing, but I
>> question whether it is actually of any real value.
>
> For example, with binfmt_misc, in combination with QEMU, you can run an
> ARM executable on an x86 box.

Or, with binfmt_misc in combination with Wine, you can directly run Microsoft
Windows .EXE files.

[snip]

--
Lew Pitcher
"In Skills, We Trust"

On 25.09.2021 at 19:07, Lew Pitcher scribbled:

> On Fri, 24 Sep 2021 04:56:30 +0000, Kaz Kylheku wrote:
>
> > On 2021-09-20, Kenny McCormack <gazelle@shell.xmission.com> wrote:
> >> Continuing from the other thread which got hijacked into talking
> >> about binfmt_misc...
> >>
> >> I find this binfmt_misc stuff very interesting and intriguing, but
> >> I question whether it is actually of any real value.
> >
> > For example, with binfmt_misc, in combination with QEMU, you can
> > run an ARM executable on an x86 box.
>
> Or, with binfmt_misc in combination with Wine, you can directly run
> Microsoft Windows .EXE files.

Just what the world needs... :p

(Yes, that was sarcasm. :p)

--
With respect,
= Aragorn =