We have added multiple sections [ertl-sections16] to Gforth. Sections
are the same concept as in assembly language and binary formats
(e.g. ELF): Classical Forth has one section=dictionary: A stretch of
memory from the start to HERE, with the address space behind available
for future allocation. Each section works like the dictionary: It has
a stretch of allocated memory, and behind that there is memory
available for future allocation.

Gforth offers the following interface for sections: There is a current
section (by default the ordinary dictionary), and HERE and UNUSED
produce the results for the current section, and ALLOT etc. work on
that section. You can temporarily change the section.

We have a stack of sections starting with the ordinary dictionary. We
also have named sections (not section stacks, unstackable named
sections are good enough for now). E.g., when I type .SECTIONS after
starting gforth, I get:

start size +dp name
7F7FFB5D8000 4000 518 locals-headers
> 7F7FF9B81040 800000 898F8 Forth
7F7FF9980000 200000 4DE8 noname
7F7FF98FF000 80000 5B8 noname
7F7FFB401000 20000 100 noname

The "size" column indicates the maximum size of that section, "+dp"
the ALLOTed memory in that section. "Forth" is the traditional
dictionary, and the ">" indicates that it is the current section; the
noname sections below are the stack of sections associated with the
dictionary. "locals-headers" is a section that contains the names of
currently visible locals.

Some major uses of sections are for storing quotations and string
literals that are compiled into a colon definition: We switch to the
next section in the stack, put the quotation or string there, then
switch back, and compile some literal pointing to the quotation or
string in the other section. This avoids the need to branch around
the quotation or string compiled into the same section.

I would have liked to quantify the benefit that this use of sections
gives us by using the old branch-around scheme and comparing the
results, but a quick attempt resulted in a crash, so apparently there
is code in gforth that relies on sections being separate. However,
what I have seen is that just by loading a few Gforth files after the
sections code rather than before, the native code size in gforth-fast
shrank by 224 bytes.

Another (recent) use of sections is for storing the locals headers.
Originally Gforth had a 1000-Byte buffer for that, which served like a
section before we introduced sections. But that buffer was stored in
its whole length in the image, unlike sections, of which only the
alloted part is stored, so we did not want to make it really large.
In 2011 (I guess when we introduced quotations), we felt that
something less limited was needed, and we ALLOCATEd the names of the
locals (stored in a linked list for deallocation). Since then, we
have added sections, so yesterday I changed allocation of locals names
to use a section LOCALS-HEADERS instead of using ALLOCATE.

This allowed me to rip out the code for ALLOCATEing and FREEing the
locals names, and for allowing to CREATE names in that allocated
memory. The result is that the source code shrank by 40 lines:

git log --shortstat b3118b84...f634044881a
....
2 files changed, 25 insertions(+), 65 deletions(-)

The threaded code (on a 64-bit system) shrank by 2608 bytes, and the
native code on AMD64 gforth-fast shrank by 1744 bytes.

The paper below is surprisingly (to me) up-to-date. I originally
envisioned having stacks for all named sections, but they have not
been needed yet.

@InProceedings{ertl-sections16,
author = {M. Anton Ertl},
title = {Sections},
crossref = {euroforth16},
pages = {55--57},
url = {http://www.complang.tuwien.ac.at/papers/ertl-sections16.pdf},
video = {https://wiki.forth-ev.de/lib/exe/fetch.php/events:sections.mp4},
OPTnote = {not refereed},
abstract = {A section is a contiguous region of memory, to which
data or code can be appended (like the Forth
dictionary). Assembly languages and linkers have
supported multiple sections for a long time. This
paper describes the benefits of supporting multiple
sections in Forth, interfaces and implementation
techniques.}
}

@Proceedings{euroforth16,
title = {32nd EuroForth Conference},
booktitle = {32nd EuroForth Conference},
year = {2016},
key = {EuroForth'16},
url = {http://www.complang.tuwien.ac.at/anton/euroforth/ef16/papers/proceedings.pdf}
}

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

On Friday, August 6, 2021 at 10:56:05 AM UTC+2, Anton Ertl wrote:
> We have added multiple sections [ertl-sections16] to Gforth. Sections
> are the same concept as in assembly language and binary formats
> (e.g. ELF): Classical Forth has one section=dictionary: A stretch of
> memory from the start to HERE, with the address space behind available
> for future allocation. Each section works like the dictionary: It has
> a stretch of allocated memory, and behind that there is memory
> available for future allocation.
[..]
Doesn't this scheme complicate inlining?

-marcel

Marcel Hendrix <mhx@iae.nl> writes:
>On Friday, August 6, 2021 at 10:56:05 AM UTC+2, Anton Ertl wrote:
>> We have added multiple sections [ertl-sections16] to Gforth. Sections
>> are the same concept as in assembly language and binary formats
>> (e.g. ELF): Classical Forth has one section=dictionary: A stretch of
>> memory from the start to HERE, with the address space behind available
>> for future allocation. Each section works like the dictionary: It has
>> a stretch of allocated memory, and behind that there is memory
>> available for future allocation.
>[..]
>Doesn't this scheme complicate inlining?

Gforth does not have inlining, but I don't see why it would. If we
had inlining at the threaded-code level (for which we have sections),
it would just compile the contents of the called colon definition
instead of the call, all in the current section, so sections don't
come into play here.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>We have added multiple sections [ertl-sections16] to Gforth. Sections
>are the same concept as in assembly language and binary formats
>(e.g. ELF): Classical Forth has one section=dictionary: A stretch of
>memory from the start to HERE, with the address space behind available
>for future allocation. Each section works like the dictionary: It has
>a stretch of allocated memory, and behind that there is memory
>available for future allocation.
>
>Gforth offers the following interface for sections: There is a current
>section (by default the ordinary dictionary), and HERE and UNUSED
>produce the results for the current section, and ALLOT etc. work on
>that section. You can temporarily change the section.
>
>We have a stack of sections starting with the ordinary dictionary. We
>also have named sections (not section stacks, unstackable named
>sections are good enough for now). E.g., when I type .SECTIONS after
>starting gforth, I get:
>
> start size +dp name
> 7F7FFB5D8000 4000 518 locals-headers
>> 7F7FF9B81040 800000 898F8 Forth
> 7F7FF9980000 200000 4DE8 noname
> 7F7FF98FF000 80000 5B8 noname
> 7F7FFB401000 20000 100 noname
>
>The "size" column indicates the maximum size of that section, "+dp"
>the ALLOTed memory in that section. "Forth" is the traditional
>dictionary, and the ">" indicates that it is the current section; the
>noname sections below are the stack of sections associated with the
>dictionary. "locals-headers" is a section that contains the names of
>currently visible locals.
>
>Some major uses of sections are for storing quotations and string
>literals that are compiled into a colon definition: We switch to the
>next section in the stack, put the quotation or string there, then
>switch back, and compile some literal pointing to the quotation or
>string in the other section. This avoids the need to branch around
>the quotation or string compiled into the same section.
>
>I would have liked to quantify the benefit that this use of sections
>gives us by using the old branch-around scheme and comparing the
>results, but a quick attempt resulted in a crash, so apparently there
>is code in gforth that relies on sections being separate.

I have now taken a different approach: Instead of measuring the
difference by having both versions and comparing the sizes, I now
count how many branches-around would have been generated, and
computing the size difference from that.

I count 550 calls to NEXT-SECTION (each of which replaces one branch
around). On the threaded-code level this replaces 8800 bytes on a
64-bit system.

On the native-code level, things are more complicated: Each
threaded-code branch is expanded into 16 bytes (in AMD64 gforth-fast).
But because we don't have sections on the native-code level, we still
make sure that a NEXT (5 bytes) is generated before a quotation. I
did not count the quotations, and anyway, there may have been a NEXT
generated before for other reasons. The bottom line is that between
(16-5)*550=6050 bytes and 16*550=8800 bytes of additional native code
would be generated if we did not use sections.

If the explanation for the native code is not clear to you, never
mind. Just consider how much these branches would cost in your
favourite implementation model.

>I changed allocation of locals names
>to use a section LOCALS-HEADERS instead of using ALLOCATE.
>
>This allowed me to rip out the code for ALLOCATEing and FREEing the
>locals names, and for allowing to CREATE names in that allocated
>memory. The result is that the source code shrank by 40 lines:
>
>git log --shortstat b3118b84...f634044881a
>...
>2 files changed, 25 insertions(+), 65 deletions(-)
>
>The threaded code (on a 64-bit system) shrank by 2608 bytes, and the
>native code on AMD64 gforth-fast shrank by 1744 bytes.

Of course the code for dealing with sections also has a cost: It costs
123 lines (without file header, test code and .SECTIONS), 4424 bytes
of dictionary (threaded code, etc) on a 64-bit system, and 4794 bytes
of native code.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

There must be complications that arise from having sections e.g.
dictionary trimming. How do you handle those?

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>>We have added multiple sections [ertl-sections16] to Gforth. Sections
>>are the same concept as in assembly language and binary formats
>>(e.g. ELF): Classical Forth has one section=dictionary: A stretch of
>>memory from the start to HERE, with the address space behind available
>>for future allocation. Each section works like the dictionary: It has
>>a stretch of allocated memory, and behind that there is memory
>>available for future allocation.
>>
>>Gforth offers the following interface for sections: There is a current
>>section (by default the ordinary dictionary), and HERE and UNUSED
>>produce the results for the current section, and ALLOT etc. work on
>>that section. You can temporarily change the section.
>>
>>We have a stack of sections starting with the ordinary dictionary. We
>>also have named sections (not section stacks, unstackable named
>>sections are good enough for now). E.g., when I type .SECTIONS after
>>starting gforth, I get:
>>
>> start size +dp name
>> 7F7FFB5D8000 4000 518 locals-headers
>>> 7F7FF9B81040 800000 898F8 Forth
>> 7F7FF9980000 200000 4DE8 noname
>> 7F7FF98FF000 80000 5B8 noname
>> 7F7FFB401000 20000 100 noname
>>
>>The "size" column indicates the maximum size of that section, "+dp"
>>the ALLOTed memory in that section. "Forth" is the traditional
>>dictionary, and the ">" indicates that it is the current section; the
>>noname sections below are the stack of sections associated with the
>>dictionary. "locals-headers" is a section that contains the names of
>>currently visible locals.
>>
>>Some major uses of sections are for storing quotations and string
>>literals that are compiled into a colon definition: We switch to the
>>next section in the stack, put the quotation or string there, then
>>switch back, and compile some literal pointing to the quotation or
>>string in the other section. This avoids the need to branch around
>>the quotation or string compiled into the same section.
>>
>>I would have liked to quantify the benefit that this use of sections
>>gives us by using the old branch-around scheme and comparing the
>>results, but a quick attempt resulted in a crash, so apparently there
>>is code in gforth that relies on sections being separate.
>
>I have now taken a different approach: Instead of measuring the
>difference by having both versions and comparing the sizes, I now
>count how many branches-around would have been generated, and
>computing the size difference from that.
>
>I count 550 calls to NEXT-SECTION (each of which replaces one branch
>around). On the threaded-code level this replaces 8800 bytes on a
>64-bit system.
>
>On the native-code level, things are more complicated: Each
>threaded-code branch is expanded into 16 bytes (in AMD64 gforth-fast).
>But because we don't have sections on the native-code level, we still
>make sure that a NEXT (5 bytes) is generated before a quotation. I
>did not count the quotations, and anyway, there may have been a NEXT
>generated before for other reasons. The bottom line is that between
>(16-5)*550=6050 bytes and 16*550=8800 bytes of additional native code
>would be generated if we did not use sections.

I found another way to measure this: Insert a threaded-code branch
where the before-section code would have done so, but still switch the
section. As a result, I see a difference of 8800 bytes in threaded
code (as predicted) and 9456 bytes in native code. The native-code
result is more than the upper bound above. I tested some explanations
for that, but came up empty-handed.

I did the same measurement on ARM64. The Forth code is only partially
the same, because on AMD64 we have an AMD64 assembler and
disassembler, and likewise on ARM64. I see a difference of 8448 bytes
of threaded code and 9120 bytes of native code.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

dxforth <dxforth@gmail.com> writes:
>There must be complications that arise from having sections e.g.
>dictionary trimming. How do you handle those?

What is dictionary trimming?

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

Anton Ertl schrieb am Freitag, 6. August 2021 um 10:56:05 UTC+2:
> We have added multiple sections [ertl-sections16] to Gforth. Sections
> are the same concept as in assembly language and binary formats
> (e.g. ELF): Classical Forth has one section=dictionary: A stretch of
> memory from the start to HERE, with the address space behind available
> for future allocation. Each section works like the dictionary: It has
> a stretch of allocated memory, and behind that there is memory
> available for future allocation. <snip>

This is gforth-specific and therefore no critique here.

For my part I use one (sometimes several) heap(s) for data objects such
as measured sensor data (time series). So they don't clutter the
dictionary, and access is controlled by a small set of heap management
words.

Headers for locals are temporarily compiled into the dictionary and
discarded/overwritten later so that they don't eat up space. I have not
yet found good reasons to compartmentalize the dictionary, except
keeping headers and code in different sections.

On 7/08/2021 02:54, Anton Ertl wrote:
> dxforth <dxforth@gmail.com> writes:
>>There must be complications that arise from having sections e.g.
>>dictionary trimming. How do you handle those?
>
> What is dictionary trimming?

FORGET MARKER EMPTY

It's a complication of Forth other languages have largely avoided.
If there's one thing in forth that's become a monster and dubious
benefit it's this. Inherently messy and complicated, sections only
make it worse. While I didn't have a choice, others do.

Regarding Forth 'in-lining' the classic example is CREATE DOES>.
Attempting to 'section off' the run-time colon definition can
actually cost more in space - as I discovered when implementing
my assembler.

Which brings me to quotations. When I looked at the possibility of
separating the quotation from the parent, I found it problematic to
the point of not pursuing.

Sections bring with them a good deal of complication. IMO one has
to have a substantial reason for using them that counter-balances
the negatives. That 'other languages have them' doesn't cut it.
Other languages don't have users screwing around with dictionaries.

dxforth <dxforth@gmail.com> writes:
>On 7/08/2021 02:54, Anton Ertl wrote:
>> dxforth <dxforth@gmail.com> writes:
>>>There must be complications that arise from having sections e.g.
>>>dictionary trimming. How do you handle those?
>>
>> What is dictionary trimming?
>
>FORGET MARKER EMPTY

MARKER needs to store the HERE of all sections, and when the marker is
called, the HEREs have to be restored. Any sections that did not
exist at MARKER time have to be eliminated.

Gforth does not have FORGET, nor EMPTY.

>It's a complication of Forth other languages have largely avoided.

FORGET is TOOLS EXT and therefore optional in the standard. It is
still marked as obsolescent, although I doubt that there will be
consensus for destandardizing it.

MARKER is CORE EXT (why not TOOLS EXT?) and therefore optional in the
standard.

EMPTY is non-standard.

So if you shy away from the complications of implementing these words,
the standard is certainly no obstacle to that (and given your
rethorics wrt standardization, the standard should be no obstacle
anyway).

My personal take is that BYE and restart is the appropriate way to
clean up the dictionary. It was different on the C64 when reloading
the Forth system and the application would cost a minute or several.

>If there's one thing in forth that's become a monster and dubious
>benefit it's this. Inherently messy and complicated, sections only
>make it worse.

Sure, although not much worse.

>Regarding Forth 'in-lining' the classic example is CREATE DOES>.

CREATE DOES> does not do any inlining, certainly not in classic Forth
implementations (where such words are represented by a CFA and
compilation just ","s this CFA), and not in Gforth (where such words
are represented by their CFA and compilation compiles a primitive
DOES-XT followed by the CFA.

>Attempting to 'section off' the run-time colon definition can
>actually cost more in space - as I discovered when implementing
>my assembler.

What do you mean by 'section off'?

>Sections bring with them a good deal of complication.

In addition to what I described earlier, the image saver (SAVESYSTEM),
the relocation data generator, and the image loader became
section-aware and more complicated. One could avoid these
complications by avoiding relocation and saving everything as one big
image (unused space included), and then loading it to the same
address.

>That 'other languages have them' doesn't cut it.

It's also a straw man.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

"minf...@arcor.de" <minforth@arcor.de> writes:
>Headers for locals are temporarily compiled into the dictionary and
>discarded/overwritten later so that they don't eat up space. I have not
>yet found good reasons to compartmentalize the dictionary, except
>keeping headers and code in different sections.

Ok, so you can keep the locals headers in the normal dictionary
section because you compile the code in a different section. In
either case, the locals headers are stored in a different section than
the code.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

Anton Ertl schrieb am Samstag, 7. August 2021 um 09:35:01 UTC+2:
> "minf...@arcor.de" <minf...@arcor.de> writes:
> >Headers for locals are temporarily compiled into the dictionary and
> >discarded/overwritten later so that they don't eat up space. I have not
> >yet found good reasons to compartmentalize the dictionary, except
> >keeping headers and code in different sections.
> Ok, so you can keep the locals headers in the normal dictionary
> section because you compile the code in a different section. In
> either case, the locals headers are stored in a different section than
> the code.

No, code is compiled into dictionary space, but relocatible and moved upwards
when temporary local names are no longer needed "to close the gap".

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
> Each section works like the dictionary: It has a stretch of allocated
> memory, and behind that there is memory available for future
> allocation.

Is this also a good way to implement the dictionaries and stacks in a
multitasking system?

On 8/08/2021 07:29, Paul Rubin wrote:
> anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> Each section works like the dictionary: It has a stretch of allocated
>> memory, and behind that there is memory available for future
>> allocation.
>
> Is this also a good way to implement the dictionaries and stacks in a
> multitasking system?
>

Multi-user? There have been traditional ways of implementing that in
Forth. As for 'good' alternate ways, how to know unless one tried it?
More importantly would you be satisfied with an answer from someone
who hadn't? Most who ask questions aren't vitally interested, else
they'd find out for themselves instead of pursuing opinion. AFAIK
Chuck didn't consult authorities before embarking upon Forth.

dxforth <dxforth@gmail.com> writes:
>> Is this also a good way to implement the dictionaries and stacks in a
>> multitasking system?
>>
> Multi-user? There have been traditional ways of implementing that in
> Forth. As for 'good' alternate ways, how to know unless one tried it?

I hadn't been thinking of multi-user, but maybe in a traditional single
user multitasking system, the tasks share a single dictionary, and only
the stacks and user variable areas would be in segments.

> More importantly would you be satisfied with an answer from someone
> who hadn't?

Anton implemented the segment scheme and has tons of Forth knowledge, so
of course I'd be satisfied with his answer.

> Most who ask questions aren't vitally interested,

I don't claim to be vitally interested and don't feel that I have such a
duty. It still seemed worth asking. The world is not binary and
neither are interest levels.

Paul Rubin <no.email@nospam.invalid> writes:
>anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> Each section works like the dictionary: It has a stretch of allocated
>> memory, and behind that there is memory available for future
>> allocation.
>
>Is this also a good way to implement the dictionaries and stacks in a
>multitasking system?

Concerning the stacks, they are not in the sections. Every task has
its own set of stacks, and its own user area.

Concerning the dictionary, AFAIK Chuck Moore's (and later Forth,
Inc.'s) multitasking multi-user Forth systems had a dictionary for
each user. So he thought that was a good idea, and that feature was
continued in polyForth, so it appears to have been a success.

A difference from Gforth's sections is that the idea was to present an
unsectioned dictionary to each user, so it did not have the API for
dealing with sections nor did it make use of sections for its own
internal purposes. It would have been a bad idea on the machines of
the time: No paging MMU to allow sections to use only as much physical
RAM as is actually used (plus padding to the next page boundary), so
you would have to size the sections precisely to avoid wasting RAM,
which would diminish their utility greatly. Plus, sections cost RAM
for the management software, and with so little RAM overall, it's
questionable if that cost can be amortized.

By contrast, Gforth's multi-tasking is not multi-user, and a task
normally cannot do dictionary allocation. In particular, the compiler
is not designed for letting multiple tasks compile. The sections are
there for the main task (that can compile), although it's possible to
create a section, and let the CURRENT-SECTION of a task point to it,
and do dictionary allocation (ALLOT and ,) there (but no other task
should allocate to that section then).

Not sure if that answers your question.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

Paul Rubin <no.email@nospam.invalid> writes:
>dxforth <dxforth@gmail.com> writes:
>>> Is this also a good way to implement the dictionaries and stacks in a
>>> multitasking system?
>>>
>> Multi-user? There have been traditional ways of implementing that in
>> Forth. As for 'good' alternate ways, how to know unless one tried it?
>
>I hadn't been thinking of multi-user, but maybe in a traditional single
>user multitasking system, the tasks share a single dictionary, and only
>the stacks and user variable areas would be in segments.

In multi-tasking Forths of course each task has its own stacks and
user area. Early Forths also had multi-user capabilities, with each
user having a dictionary:

From
<https://www.forth.com/resources/forth-programming-language/#21_Forth_at_NRAO>:
|Multiuser operation of the sort that enabled NRAO’s astronomers to
|graphically analyze data while an operator controlled the telescope
|and live data was flowing in, was unheard-of.

Given that Gforth runs hosted on a multi-user OS, and Gforth EC runs
on 0-1 terminal-user embedded controllers, this feature is not
necessary in Gforth.

And I think the same is true in most other Forth systems today. AFAIK
Greg Bailey was still providing native (i.e., no OS below Forth) Forth
systems a while ago, not sure if he or his successors still do.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
> Concerning the stacks, they are not in the sections. Every task has
> its own set of stacks, and its own user area.

Is there a particular reason to not use sections to allocate the stacks
and user areas? Or does it just not help significantly?

Is it usual in multitasking Forths to create and destroy tasks
dynamically (like a network server might do, as connections come and
go), or is there more commonly a fixed set of tasks through the
program's entire execution?

> The sections are there for the main task (that can compile)

Ah ok, if only the main task can compile, that is reasonable and
simplifies things. I guess there is some way to have separate names for
the user variables in the different tasks: I'll look at eforth to see
how it does that.

> Not sure if that answers your question.

Yes I think so. Actually in a system as big as gforth, maybe it is
reasonable to use garbage collection in the compiler, at least in a
hypothetical fancy analytical compiler of the future that could benefit
from it.

Paul Rubin <no.email@nospam.invalid> writes:
> dxforth <dxforth@gmail.com> writes:
> >> Is this also a good way to implement the dictionaries and stacks in a
> >> multitasking system?
> > Multi-user? There have been traditional ways of implementing that in
> > Forth. As for 'good' alternate ways, how to know unless one tried it?
> I hadn't been thinking of multi-user, but maybe in a traditional single
> user multitasking system, the tasks share a single dictionary, and only
> the stacks and user variable areas would be in segments.

You have to thinka about the heap. In ForthOS, each task has its own heap,
and every other comparable Forth system I've studied did this also. A
consequence of this is that only the primary task can do compilations;
making all the edge cases work nicely to compile on other tasks was--for
me--not worth the effort.

Andy Valencia
Home page: https://www.vsta.org/andy/
To contact me: https://www.vsta.org/contact/andy.html

Paul Rubin <no.email@nospam.invalid> writes:
>anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> Concerning the stacks, they are not in the sections. Every task has
>> its own set of stacks, and its own user area.
>
>Is there a particular reason to not use sections to allocate the stacks
>and user areas? Or does it just not help significantly?

It does not help. Sections are for when you need to allocate
piecewise. Each stack and the user area are each allocated in one
chunk.

>Is it usual in multitasking Forths to create and destroy tasks
>dynamically (like a network server might do, as connections come and
>go), or is there more commonly a fixed set of tasks through the
>program's entire execution?

Gforth's multi-tasking certainly allows the former. My guess is that
on small systems you would have a fixed set of tasks in order to avoid
out-of-memory conditions.

>> The sections are there for the main task (that can compile)
>
>Ah ok, if only the main task can compile, that is reasonable and
>simplifies things. I guess there is some way to have separate names for
>the user variables in the different tasks: I'll look at eforth to see
>how it does that.

Every user variable has an instance in every task, so you tend to be
stingy with them. If you want thread-local data that is not
replicated across all threads, you have to do it differently. E.g.,
Gforth has user objects that take only one user-variable cell (the
address of the object), and you can dynamically allocate the object
(or not allocate it).

>> Not sure if that answers your question.
>
>Yes I think so. Actually in a system as big as gforth, maybe it is
>reasonable to use garbage collection in the compiler, at least in a
>hypothetical fancy analytical compiler of the future that could benefit
>from it.

I have plans for an analytical compiler, but no plans for using
garbage collection for it. Region-based memory allocation looks like
a good fit for a compiler.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

On 06/08/2021 08:50, Anton Ertl wrote:
> We have added multiple sections [ertl-sections16] to Gforth. Sections
> are the same concept as in assembly language and binary formats
> (e.g. ELF): Classical Forth has one section=dictionary: A stretch of
> memory from the start to HERE, with the address space behind available
> for future allocation. Each section works like the dictionary: It has
> a stretch of allocated memory, and behind that there is memory
> available for future allocation.
>
> Gforth offers the following interface for sections: There is a current
> section (by default the ordinary dictionary), and HERE and UNUSED
> produce the results for the current section, and ALLOT etc. work on
> that section. You can temporarily change the section.
>
> We have a stack of sections starting with the ordinary dictionary. We
> also have named sections (not section stacks, unstackable named
> sections are good enough for now). E.g., when I type .SECTIONS after
> starting gforth, I get:
>
> start size +dp name
> 7F7FFB5D8000 4000 518 locals-headers
>> 7F7FF9B81040 800000 898F8 Forth
> 7F7FF9980000 200000 4DE8 noname
> 7F7FF98FF000 80000 5B8 noname
> 7F7FFB401000 20000 100 noname
>
> The "size" column indicates the maximum size of that section, "+dp"
> the ALLOTed memory in that section. "Forth" is the traditional
> dictionary, and the ">" indicates that it is the current section; the
> noname sections below are the stack of sections associated with the
> dictionary. "locals-headers" is a section that contains the names of
> currently visible locals.
>
> Some major uses of sections are for storing quotations and string
> literals that are compiled into a colon definition: We switch to the
> next section in the stack, put the quotation or string there, then
> switch back, and compile some literal pointing to the quotation or
> string in the other section. This avoids the need to branch around
> the quotation or string compiled into the same section.
>
> I would have liked to quantify the benefit that this use of sections
> gives us by using the old branch-around scheme and comparing the
> results, but a quick attempt resulted in a crash, so apparently there
> is code in gforth that relies on sections being separate. However,
> what I have seen is that just by loading a few Gforth files after the
> sections code rather than before, the native code size in gforth-fast
> shrank by 224 bytes.
>
> Another (recent) use of sections is for storing the locals headers.
> Originally Gforth had a 1000-Byte buffer for that, which served like a
> section before we introduced sections. But that buffer was stored in
> its whole length in the image, unlike sections, of which only the
> alloted part is stored, so we did not want to make it really large.
> In 2011 (I guess when we introduced quotations), we felt that
> something less limited was needed, and we ALLOCATEd the names of the
> locals (stored in a linked list for deallocation). Since then, we
> have added sections, so yesterday I changed allocation of locals names
> to use a section LOCALS-HEADERS instead of using ALLOCATE.
>
> This allowed me to rip out the code for ALLOCATEing and FREEing the
> locals names, and for allowing to CREATE names in that allocated
> memory. The result is that the source code shrank by 40 lines:
>
> git log --shortstat b3118b84...f634044881a
> ...
> 2 files changed, 25 insertions(+), 65 deletions(-)
>
> The threaded code (on a 64-bit system) shrank by 2608 bytes, and the
> native code on AMD64 gforth-fast shrank by 1744 bytes.
>

That's interesting. My system has had sections since I developed it in
approx 2005. Sections are named and mostly provide several dataspaces.
My motivation was to be able to use dataspace handling words such as
HERE , C, ALLOT ALIGN UNUSED in different areas of memory. For example
it is nice to be able to ALLOT or ALLOCATE a buffer as a section and use
C, to save characters in it. Sometimes a word may be defined to do a
similar but different action e.g define , to compile a string (ca u).
There is a current-section pointer - I've not considered a section stack
but would guess that having one would simplify a few things. The system
uses a few sections, these include headers, headers for locals and other
objects, code, string literals. I haven't used multiple sections for
code but in principle that is possible (I think!) but I didn't think of
doing so at the time - I'm not sure what benefit that would give me e.g.
quotations don't cost any extra space at present.

--
Gerry

Gerry Jackson <do-not-use@swldwa.uk> writes:
>On 06/08/2021 08:50, Anton Ertl wrote:
>I haven't used multiple sections for
>code but in principle that is possible (I think!) but I didn't think of
>doing so at the time - I'm not sure what benefit that would give me e.g.
>quotations don't cost any extra space at present.

How does the code compiled for, e.g.,

: foo dup [: swap - ;] swap ! ;

look in your system. Before sections, Gforth compiles this code

\ header of foo
$7F4EDFCE84B8 dup
$7F4EDFCE84C0 branch
$7F4EDFCE84C8 <foo+$40>
$7F4EDFCE84D0 <(UValue)+$A8> \ header of a nameless word
$7F4EDFCE84D8 $4048C8 \ header of a nameless word
$7F4EDFCE84E0 swap
$7F4EDFCE84E8 -
$7F4EDFCE84F0 ;s
$7F4EDFCE84F8 lit
$7F4EDFCE8500 <foo+$20>
$7F4EDFCE8508 swap
$7F4EDFCE8510 !
$7F4EDFCE8518 ;s ok

With sections, it compiles this code:

For foo:
\ header of foo
$7FE8DC25ECD8 dup
$7FE8DC25ECE0 lit
$7FE8DC25ECE8 $7FE8DBFD8CB0
$7FE8DC25ECF0 swap
$7FE8DC25ECF8 !
$7FE8DC25ED00 ;s ok

For the quotation:
\ header of a nameless word
$7FE8DBFD8CB8 swap
$7FE8DBFD8CC0 -
$7FE8DBFD8CC8 ;s ok

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

On 09/08/2021 10:00, Anton Ertl wrote:
> Gerry Jackson <do-not-use@swldwa.uk> writes:
>> On 06/08/2021 08:50, Anton Ertl wrote:
>> I haven't used multiple sections for
>> code but in principle that is possible (I think!) but I didn't think of
>> doing so at the time - I'm not sure what benefit that would give me e.g.
>> quotations don't cost any extra space at present.
>
> How does the code compiled for, e.g.,
>
> : foo dup [: swap - ;] swap ! ;
>
> look in your system. Before sections, Gforth compiles this code
>
> \ header of foo
> $7F4EDFCE84B8 dup
> $7F4EDFCE84C0 branch
> $7F4EDFCE84C8 <foo+$40>
> $7F4EDFCE84D0 <(UValue)+$A8> \ header of a nameless word
> $7F4EDFCE84D8 $4048C8 \ header of a nameless word
> $7F4EDFCE84E0 swap
> $7F4EDFCE84E8 -
> $7F4EDFCE84F0 ;s
> $7F4EDFCE84F8 lit
> $7F4EDFCE8500 <foo+$20>
> $7F4EDFCE8508 swap
> $7F4EDFCE8510 !
> $7F4EDFCE8518 ;s ok
>
> With sections, it compiles this code:
>
> For foo:
> \ header of foo
> $7FE8DC25ECD8 dup
> $7FE8DC25ECE0 lit
> $7FE8DC25ECE8 $7FE8DBFD8CB0
> $7FE8DC25ECF0 swap
> $7FE8DC25ECF8 !
> $7FE8DC25ED00 ;s ok
>
> For the quotation:
> \ header of a nameless word
> $7FE8DBFD8CB8 swap
> $7FE8DBFD8CC0 -
> $7FE8DBFD8CC8 ;s ok
>

\ Header Section
$11DD124 \ Header for FOO with xt = $11DAAE4

\ Code Section
$11DAAD4 Start of quotation (colon def semantics pointer)
$11DAAD8 swap
$11DAADC -
$11DAAE0 ;s
$11DAAE4 Start of FOO (colon def semantics pointer)
$11DAAE8 dup
$11DAAEC (lit)
$11DAAF0 $11DAAD4 xt of quotation
$11DAAF4 swap
$11DAAF8 !
$11DAAFC ;s

\ Strings section
$11D8A8B "3foo" Counted string

The quotation code precedes that for FOO because my system initially
compiles to an intermediate code in a buffer until ; is reached when it
lays down the final code. When the quotation is encountered compilation
of FOO is suspended, the quotation compiled until complete, then
compilation of FOO is resumed. So there is no need for a branch over the
quotation. If quotations are nested the innermost is completed first.
This also permits colon definitions, variables etc to be nested inside
colon definitions.

THe Semantics pointer points to a structure with pointers to code to
interpret, compile the word etc. The same semantics structure is used
for all colon definitions with default semantics including quotations
and :NONAMEs

Why does Gforth need a header for a nameless word?

--
Gerry

On Monday, August 9, 2021 at 2:46:27 PM UTC+2, Gerry Jackson wrote:
> Why does Gforth need a header for a nameless word?

I wouldn't know. Just branch over the shebang.

Addr| Opcode Operand

0| branch 10
1| dup 0
2| literal 4
3| branch 7
4| swap 0
5| - 0
6| exit 0
7| swap 0
8| ! 0
9| exit 0

You guys do have AHEAD, don't you?

Hans Bezemer

On 09/08/2021 16:49, Hans Bezemer wrote:
> On Monday, August 9, 2021 at 2:46:27 PM UTC+2, Gerry Jackson wrote:
>> Why does Gforth need a header for a nameless word?
>
> I wouldn't know. Just branch over the shebang.

How does that answer the question?

>
> Addr| Opcode Operand
>
> 0| branch 10
> 1| dup 0
> 2| literal 4
> 3| branch 7
> 4| swap 0
> 5| - 0
> 6| exit 0
> 7| swap 0
> 8| ! 0
> 9| exit 0
>
> You guys do have AHEAD, don't you?

How do you think Gforth generated the code before Anton introduced the
use of sections?

Why would I use AHEAD when my system's design meant I didn't need to?

--
Gerry