I am overhauling the string parsing words in the kForth String Words
library (strings.4th). The existing words are not named as clearly as
they should be, and have inefficient implementations:

\ PARSE_TOKEN ( a u -- arem urem atok utok )
\ PARSE_LINE ( a u -- a1 u1 a2 u2 ... an un n )
\ PARSE_ARGS ( a u -- n ) ( F: r1 ... rn )
\ PARSE_CSV ( a u -- n ) ( F: r1 ... rn )

PARSE_TOKEN skips leading spaces and parses the next blank(s)-delimited
substring, atok utok, and also returns the remaining portion of the string.

PARSE_LINE applies PARSE_TOKEN repeatedly to place the token strings and
token count on the stack.

PARSE_ARGS parses a string with blank delimited numbers, converting each
substring to a floating point number, returning the n floating point
numbers and the count.

PARSE_CSV is a hack which replaces each comma in the string with a space
and performs the same function as PARSE_ARGS.

These words need clearer names and an upgrade (particularly PARSE_CSV).
My proposed replacements are

\ NEXT-BS-TOKEN ( a u -- atok utok arem urem )
\ NEXT-CS-TOKEN ( a u -- atok utok arem urem )
\ PARSED-BSV ( a u -- a1 u1 a2 u2 ... an un n )
\ PARSED-CSV ( a u -- a1 u1 a2 u2 ... an un n )
\ ITH-PARSED ( a1 u1 ... an un n i -- a1 u1 ... an un n ai ui )
\ DROP-PARSED ( a1 u1 ... an un n -- )
\ PARSED>FLOATS ( a1 u1 ... an un n -- n ) ( F: r1 ... rn )

NEXT-BS-TOKEN is a slightly more efficient version of PARSE_TOKEN, but
leaves the token and remaining strings in a swapped order. The name
indicates that it is parsing the next "blank(s)-separated" token in the
string.

NEXT-CS-TOKEN parses the next "comma-separated" token in the string and
returns the token and remaining substrings.

PARSED-BSV starts with an input string and repeatedly applies
NEXT-BS-TOKEN until the remaining string is null and returns all of the
substrings and the token count on the stack.

PARSED-CSV starts with an input string and repeatedly applies
NEXT-CS-TOKEN until the remaining string is null. It returns all of the
substrings and the token count on the stack. Unlike PARSED-BSV, each
comma delimiter will mark a separte token string, e.g. if the input
string is ",," there will be three token strings, each of length zero.

ITH-PARSED provides a way to PICK the ith substring returned by
PARSED-BSV and PARSED-CSV.

DROP-PARSED may be used to discard the n token substrings returned by
PARSED-BSV and PARSED-CSV.

PARSED>FLOATS converts each of the token substrings returned by
PARSED-BSV or PARSED-CSV to n floating point numbers. If the string to
float conversion fails, the floating point value NAN will be returned.
If a substring has zero length, the corresponding fp value will also be NAN.

Comments or suggestions?

--
Krishna Myneni

The previous set of words will be redefined using the new words (for
running existing source code); however, they will be considered deprecated.

On Sunday, 13 March 2022 at 01:12:12 UTC+1, Krishna Myneni wrote:
> I am overhauling the string parsing words in the kForth String Words
> library (strings.4th). The existing words are not named as clearly as
> they should be, and have inefficient implementations:
>
> \ PARSE_TOKEN ( a u -- arem urem atok utok )
> \ PARSE_LINE ( a u -- a1 u1 a2 u2 ... an un n )
> \ PARSE_ARGS ( a u -- n ) ( F: r1 ... rn )
> \ PARSE_CSV ( a u -- n ) ( F: r1 ... rn )
>
> PARSE_TOKEN skips leading spaces and parses the next blank(s)-delimited
> substring, atok utok, and also returns the remaining portion of the string.
>
> PARSE_LINE applies PARSE_TOKEN repeatedly to place the token strings and
> token count on the stack.
>
> PARSE_ARGS parses a string with blank delimited numbers, converting each
> substring to a floating point number, returning the n floating point
> numbers and the count.
>
> PARSE_CSV is a hack which replaces each comma in the string with a space
> and performs the same function as PARSE_ARGS.

In ntf/lxf I have

GET-WORD ( addr len -- addr" len" addr' len' )
GET-LINE ( addr len -- addr" len" addr' len' )
SPLIT-BUFFER ( addr len xchar -- addr" len" addr' len' )

GET-WORD is like your PARSE_TOKEN
GET-LINE will return a line but not split in tokens like your PARSE_LINE
SPLIT-BUFFER is a lower level word to construct other parsing words from.

Why do you need a word that potentially fills up the stack?

> These words need clearer names and an upgrade (particularly PARSE_CSV).
> My proposed replacements are
>
>
> \ NEXT-BS-TOKEN ( a u -- atok utok arem urem )
> \ NEXT-CS-TOKEN ( a u -- atok utok arem urem )
> \ PARSED-BSV ( a u -- a1 u1 a2 u2 ... an un n )
> \ PARSED-CSV ( a u -- a1 u1 a2 u2 ... an un n )
> \ ITH-PARSED ( a1 u1 ... an un n i -- a1 u1 ... an un n ai ui )
> \ DROP-PARSED ( a1 u1 ... an un n -- )
> \ PARSED>FLOATS ( a1 u1 ... an un n -- n ) ( F: r1 ... rn )
>
> NEXT-BS-TOKEN is a slightly more efficient version of PARSE_TOKEN, but
> leaves the token and remaining strings in a swapped order. The name
> indicates that it is parsing the next "blank(s)-separated" token in the
> string.

Why is it better to have them swapped?
In all my use cases I parse for a word, do something with it, check if there is remaining text
and repeat if there is.

BR
Peter Fälth

> NEXT-CS-TOKEN parses the next "comma-separated" token in the string and
> returns the token and remaining substrings.
>
> PARSED-BSV starts with an input string and repeatedly applies
> NEXT-BS-TOKEN until the remaining string is null and returns all of the
> substrings and the token count on the stack.
>
> PARSED-CSV starts with an input string and repeatedly applies
> NEXT-CS-TOKEN until the remaining string is null. It returns all of the
> substrings and the token count on the stack. Unlike PARSED-BSV, each
> comma delimiter will mark a separte token string, e.g. if the input
> string is ",," there will be three token strings, each of length zero.
>
> ITH-PARSED provides a way to PICK the ith substring returned by
> PARSED-BSV and PARSED-CSV.
>
> DROP-PARSED may be used to discard the n token substrings returned by
> PARSED-BSV and PARSED-CSV.
>
> PARSED>FLOATS converts each of the token substrings returned by
> PARSED-BSV or PARSED-CSV to n floating point numbers. If the string to
> float conversion fails, the floating point value NAN will be returned.
> If a substring has zero length, the corresponding fp value will also be NAN.
>
> Comments or suggestions?
>
> --
> Krishna Myneni
>
>
>
>
> The previous set of words will be redefined using the new words (for
> running existing source code); however, they will be considered deprecated.

On Sunday, March 13, 2022 at 1:12:12 AM UTC+1, Krishna Myneni wrote:
[..]
> Comments or suggestions?
[..]
It looks too specific to me, I mean, it probably serves your current
interests, but it wouldn't suffice for mine.

What sticks out is the lack of a separator character (space, comma, ..)
or string (cr+lf, eol, arbitrary word..).

It is interesting that you have words to break larger strings down in
sub-entities, however, returning all the addresses and lengths on the
stack is unacceptable. You may want to supply the name of a string
array as input to this parser, possibly with the count of tokens returned
on the stack. Such a word is the latest addition to my own string word
set. In my case I limit the number of tokens to a maximum, which
allows to have a fixed size offset/length buffer which can remain
unnamed.

Lately, I've encountered the need for a splitter where the start delimiter
is different from the end one ( like "(" and ")" ) but I didn't generalize it
yet.

Wil Baden's Toolbelt words are quite good (with some additions for upper
and lower case).

-marcel

Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>Comments or suggestions?

In one of my papers [ertl13-strings], I suggest for splitting/parsing
to do it one result at a time. We have SEARCH (and SCAN/SKIP) that
can help in working that way, but require some additional work around them. In the paper I suggest:

|search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
| |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
|c-a1 u3 is the substring before the first match, c-a4 u4 is the first
|match, and c-a5 u5 is the rest of the string, and the TOS is true;
|otherwise return false.

You would then call it again for c-a5 u5 to get the next match.

I also suggest to use a PARSE-like interface (in combination with
EXECUTE-PARSING) to avoid the stack load of keeping c-a1 u1 and c-a5
u5 around. The parsing version of the word above is:

|parse-regexp ( c-a2 u2 -- c-a1 u3 c-a4 u4 true | false )
| |Search for the regexp c-a2 u2 in the parse area. If a match is found,
|c-a4 u4 is the address of the match, and c-a1 u3 is the string that
|was skipped before the match was found. The next parse starts right
|behind the matching string.

SEARCH-REGEXP would be a factor of PARSE-REGEXP.

I have not implemented these words.

@InProceedings{ertl13-strings,
author = {M. Anton Ertl},
title = {Standardize Strings Now!},
crossref = {euroforth13},
pages = {39--43},
url = {http://www.complang.tuwien.ac.at/anton/euroforth/ef13/papers/ertl-strings.pdf},
OPTnote = {not refereed},
abstract = {This paper looks at the issues in string words: what
operations may be required, various design options,
and why this has lead to the current state of
standardization of string operations that is
insufficient in the eyes of many.}
}

@Proceedings{euroforth13,
title = {29th EuroForth Conference},
booktitle = {29th EuroForth Conference},
year = {2013},
key = {EuroForth'13},
url = {http://www.complang.tuwien.ac.at/anton/euroforth/ef13/papers/proceedings.pdf}
}

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

On 3/13/22 4:59 AM, Marcel Hendrix wrote:

> What sticks out is the lack of a separator character (space, comma, ..)

> You may want to supply the name of a string
> array as input to this parser, ...

Agreed. I do both of the above in my string library code and it has
proven very useful. I have a word :split that is supplied the char and
the string to split. The output is a dynamically resizable array of
string objects. The output array size is only limited by heap memory.

\ find substring(s) delimited by:
\ 1) start of string and char
\ 2) and char and char
\ 3) and char and end of string
\ return all of them as an array of string objects allocated in the heap
: :split ( char str-obj -- 1-arry-obj ) ...

-Doug

On Sunday, March 13, 2022 at 10:52:01 AM UTC+1, Anton Ertl wrote:
> Krishna Myneni <krishna...@ccreweb.org> writes:
> >Comments or suggestions?
>
> In one of my papers [ertl13-strings], I suggest for splitting/parsing
> to do it one result at a time. We have SEARCH (and SCAN/SKIP) that
> can help in working that way, but require some additional work around them. In the paper I suggest:
>
> |search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
> |
> |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
> |c-a1 u3 is the substring before the first match, c-a4 u4 is the first
> |match, and c-a5 u5 is the rest of the string, and the TOS is true;
> |otherwise return false.

One of the first BASIS documents had (LEX) :

FORTH> help (lex)
(lex) IFORTH
( c-addr1 u1 del -- c-addr3 u3 c-addr1 u4 del true ) or
( c-addr1 u1 del -- false )
Break a string in three pieces: before the delimiter del, the delimiter
itself, and after the delimiter. c-addr3 points to the remaining string,
c-addr1 points to the string in front of the delimiter. If false is
returned, the input string does not contain the delimiter character.

Coupling this to a full reg-exp package is tempting. However, what I've
seen with Python users is that they use regexp's all the time, without
any understanding of how it works. They rely on an online tool that shows
them how to hack it until the next gotcha. A middle ground would be nice.

-marcel

On Sunday, March 13, 2022 at 7:47:17 AM UTC-5, Marcel Hendrix wrote:
> Coupling this to a full reg-exp package is tempting. However, what I've
> seen with Python users is that they use regexp's all the time, without
> any understanding of how it works. They rely on an online tool that shows
> them how to hack it until the next gotcha. A middle ground would be nice.

Regex is standard fare in many programs. Used often enough
competent people don't require aids. Provide for it and your
Forth won't stick out like a sore thumb and seem lacking to the
many that use and expect Regex.

--
me

On 3/13/22 07:47, Marcel Hendrix wrote:
> On Sunday, March 13, 2022 at 10:52:01 AM UTC+1, Anton Ertl wrote:
>> Krishna Myneni <krishna...@ccreweb.org> writes:
>>> Comments or suggestions?
>>
>> In one of my papers [ertl13-strings], I suggest for splitting/parsing
>> to do it one result at a time. We have SEARCH (and SCAN/SKIP) that
>> can help in working that way, but require some additional work around them. In the paper I suggest:
>>
>> |search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
>> |
>> |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
>> |c-a1 u3 is the substring before the first match, c-a4 u4 is the first
>> |match, and c-a5 u5 is the rest of the string, and the TOS is true;
>> |otherwise return false.
>
> One of the first BASIS documents had (LEX) :
>
> FORTH> help (lex)
> (lex) IFORTH
> ( c-addr1 u1 del -- c-addr3 u3 c-addr1 u4 del true ) or
> ( c-addr1 u1 del -- false )
> Break a string in three pieces: before the delimiter del, the delimiter
> itself, and after the delimiter. c-addr3 points to the remaining string,
> c-addr1 points to the string in front of the delimiter. If false is
> returned, the input string does not contain the delimiter character.
>
> Coupling this to a full reg-exp package is tempting. However, what I've
> seen with Python users is that they use regexp's all the time, without
> any understanding of how it works. They rely on an online tool that shows
> them how to hack it until the next gotcha. A middle ground would be nice.
>

A middle ground to using a regexp-based parser could be a word such as
PARSE-STRING, described by

PARSE-STRING ( caddr1 u1 xt-nexttoken xt-processtoken -- ntokens )

where xt-nexttoken provides the execution semantics for obtaining the
next token from the input string,

caddr1 u1 -- caddr-rem urem caddr-token utoken

and xt-processtoken provides the execution semantics for processing the
token,

caddr-token utoken --

The number of tokens parsed and processed is returned by PARSE-STRING.

This allows one to write and use simple token parsers, e.g. not just
simple character delimiter token parsers, but whitespace token parsers
which can skip leading whitespace. It also allows flexible processing of
the tokens as they are parsed, e.g. conversion to floating point numbers
on the fp stack or copying the token strings into a buffer.

Nevertheless, more than 99.0% of my use cases are for blank(s)-delimited
tokens, whitespace-delimited tokens, or comma-delimited tokens. Simple
and fast specialized string parsers for these cases are still useful to
have in a library.

--
Krishna

On 3/13/22 09:49, Krishna Myneni wrote:
> On 3/13/22 07:47, Marcel Hendrix wrote:

>> ... A middle ground would be nice.

> A middle ground to using a regexp-based parser could be a word such as
> PARSE-STRING, described by
>
>   PARSE-STRING ( caddr1 u1 xt-nexttoken xt-processtoken -- ntokens )
>
> where xt-nexttoken provides the execution semantics for obtaining the
> next token from the input string,
>
>   caddr1 u1 -- caddr-rem urem caddr-token utoken
> ...

xt-nexttoken should also return a flag,

caddr u -- caddr-rem urem caddr-token utoken true | caddr-rem urem false

The order of args to PARSE-STRING should also be changed

PARSE-STRING ( xt-nexttoken xt-processtoken caddr u -- ntokens )

The implementation of PARSE-STRING should become trivial, except how to
handle the final case of urem not being equal to zero.

: PARSE-STRING ( xt-nexttoken xt-prcesstoken caddr u -- ntokens )
0 >R
BEGIN
2OVER DROP EXECUTE
WHILE
4 PICK EXECUTE
1 RP@ +!
REPEAT
2drop 2drop \ or handle urem <> 0
R> ;

Some clarity on what to do after REPEAT is needed.

--
Krishna

S Jack schrieb am Sonntag, 13. März 2022 um 14:28:32 UTC+1:
> On Sunday, March 13, 2022 at 7:47:17 AM UTC-5, Marcel Hendrix wrote:
> > Coupling this to a full reg-exp package is tempting. However, what I've
> > seen with Python users is that they use regexp's all the time, without
> > any understanding of how it works. They rely on an online tool that shows
> > them how to hack it until the next gotcha. A middle ground would be nice.
> Regex is standard fare in many programs. Used often enough
> competent people don't require aids. Provide for it and your
> Forth won't stick out like a sore thumb and seem lacking to the
> many that use and expect Regex.

Nobody with a minimum of engineering background will bloat his
embedded application programs with regex when there are smaller
and faster other options.

On Sunday, March 13, 2022 at 2:28:32 PM UTC+1, S Jack wrote:
> Provide for it and your
> Forth won't stick out like a sore thumb and seem lacking to the
> many that use and expect Regex.
Do you REALLY think that people who NEED REGEX will actually consider
using a language like Forth?!

I do have a tiny one - but I very rarely use it. Lots of time you don't provide
an option string on a command line, but need an expression fixed in a program.
For that I use a lib like "chmatch.4th" which can considered to be precompiled
regex code - but using Forth as the interpreter. It's much lighter - and it's Forth.

s" 0123456789" sconstant 'digits'
s" +-" sconstant 'sign'
char . constant 'point'

'sign' char-match drop
'point' char-equal >r
'digits' char-match r> swap >r >r
'digits' skip-while r> 0=
if
'point' char-equal
if
'digits' char-match r> and >r
'digits' skip-while
then
then r>

Flag returned means "match- or-no-match". Easy.

I guess those people also are incapable of doing a binary search, a decent sort and
desperately need hash tables and associative arrays to get any work done, huh?

Hans Bezemer

On 3/13/22 03:26, P Falth wrote:
> On Sunday, 13 March 2022 at 01:12:12 UTC+1, Krishna Myneni wrote:
>> I am overhauling the string parsing words in the kForth String Words
>> library (strings.4th). The existing words are not named as clearly as
>> they should be, and have inefficient implementations:
>>
>> \ PARSE_TOKEN ( a u -- arem urem atok utok )
>> \ PARSE_LINE ( a u -- a1 u1 a2 u2 ... an un n )
>> \ PARSE_ARGS ( a u -- n ) ( F: r1 ... rn )
>> \ PARSE_CSV ( a u -- n ) ( F: r1 ... rn )
>>
>> PARSE_TOKEN skips leading spaces and parses the next blank(s)-delimited
>> substring, atok utok, and also returns the remaining portion of the string.
>>
>> PARSE_LINE applies PARSE_TOKEN repeatedly to place the token strings and
>> token count on the stack.
>>
>> PARSE_ARGS parses a string with blank delimited numbers, converting each
>> substring to a floating point number, returning the n floating point
>> numbers and the count.
>>
>> PARSE_CSV is a hack which replaces each comma in the string with a space
>> and performs the same function as PARSE_ARGS.
>
> In ntf/lxf I have
>
> GET-WORD ( addr len -- addr" len" addr' len' )
> GET-LINE ( addr len -- addr" len" addr' len' )
> SPLIT-BUFFER ( addr len xchar -- addr" len" addr' len' )
>
> GET-WORD is like your PARSE_TOKEN
> GET-LINE will return a line but not split in tokens like your PARSE_LINE
> SPLIT-BUFFER is a lower level word to construct other parsing words from.
>
> Why do you need a word that potentially fills up the stack?
>

In practice, I mostly need to parse strings with fewer than 8 tokens
(columns/fields of data). Most Forth systems for desktop use provide a
deep enough data stack for handling parsing for much more than even 64
fields.

But your point is certainly valid. These words were not designed to
handle, say, a 256x256 matrix written to a file in text format, with
each row or column on a single line. Some scientific computing packages
such as R will do that. For reading such files, the stack approach could
be a problem.

In my revised approach with PARSE-STRING, mentioned elsewhere in this
thread, the token recognizer/parser will be accompanied by a token
processor for immediately handling the parsed string.

>> These words need clearer names and an upgrade (particularly PARSE_CSV).
>> My proposed replacements are
>>
>>
>> \ NEXT-BS-TOKEN ( a u -- atok utok arem urem )
>> \ NEXT-CS-TOKEN ( a u -- atok utok arem urem )
>> \ PARSED-BSV ( a u -- a1 u1 a2 u2 ... an un n )
>> \ PARSED-CSV ( a u -- a1 u1 a2 u2 ... an un n )
>> \ ITH-PARSED ( a1 u1 ... an un n i -- a1 u1 ... an un n ai ui )
>> \ DROP-PARSED ( a1 u1 ... an un n -- )
>> \ PARSED>FLOATS ( a1 u1 ... an un n -- n ) ( F: r1 ... rn )
>>
>> NEXT-BS-TOKEN is a slightly more efficient version of PARSE_TOKEN, but
>> leaves the token and remaining strings in a swapped order. The name
>> indicates that it is parsing the next "blank(s)-separated" token in the
>> string.
>
> Why is it better to have them swapped?
> In all my use cases I parse for a word, do something with it, check if there is remaining text
> and repeat if there is.
>

For the case where the parsed substrings are placed on the stack, it
allows the NEXT-xx-TOKEN to be used repeatedly. However, if you are
immediately processing the substring and removing it, then the exchanged
ordering ( arem urem atok utok ) is better.

--
Krishna

On 3/13/22 04:37, Anton Ertl wrote:
> Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>> Comments or suggestions?
>
> In one of my papers [ertl13-strings], I suggest for splitting/parsing
> to do it one result at a time. We have SEARCH (and SCAN/SKIP) that
> can help in working that way, but require some additional work around them. In the paper I suggest:
>
> |search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
> |
> |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
> |c-a1 u3 is the substring before the first match, c-a4 u4 is the first
> |match, and c-a5 u5 is the rest of the string, and the TOS is true;
> |otherwise return false.
>
> You would then call it again for c-a5 u5 to get the next match.
>
....

In my generalized approach with PARSE-STRING, the xt-nexttoken
represents a generalized token recognizer/parser, which could use regexp
but can also be a simpler algorithm or a more complex algorithm. For
example, it allows for the possibility of including a state machine for
fields validation.

One example of a multicolumn field comes the github repo, H2SPEC, linked
below. The fortran program in this repo, h2spec.f, generates a
multicolumn file, h2lines.dat, with each line providing info about a
spectral line of the hydrogen molecule (in the "vacuum ultraviolet"
region). An example line in this file, which contains a mixture of
integer, floating point, and text data fields, is

3226 1589.199 62924.797 0.1125E+08 X 14 3 O B 7 4 O

The first column is an index number, col 2 is the wavelength of the
spectral line in Angstroms, col 3 is the frequency of the line in units
of cm^-1, col 4 is the transition rate in units of s^-1, cols 5--8 give
the quantum numbers/labels for the electronic, vibration, rotation, and
parity (Ortho/Para hydrogen) describing the lower level of transition,
and cols 9--12 are the corresponding information for the upper level.

The xt-nexttoken can validate the field type using a simple state
machine as each token is parsed from the string.

--
Krishna

H2SPEC
https://github.com/mynenik/H2SPEC

>
> I have not implemented these words.
>
> @InProceedings{ertl13-strings,
> author = {M. Anton Ertl},
> title = {Standardize Strings Now!},
> crossref = {euroforth13},
> pages = {39--43},
> url = {http://www.complang.tuwien.ac.at/anton/euroforth/ef13/papers/ertl-strings.pdf},
> OPTnote = {not refereed},
> abstract = {This paper looks at the issues in string words: what
> operations may be required, various design options,
> and why this has lead to the current state of
> standardization of string operations that is
> insufficient in the eyes of many.}
> }
>
> @Proceedings{euroforth13,
> title = {29th EuroForth Conference},
> booktitle = {29th EuroForth Conference},
> year = {2013},
> key = {EuroForth'13},
> url = {http://www.complang.tuwien.ac.at/anton/euroforth/ef13/papers/proceedings.pdf}
> }
>
> - anton

On Sunday, 13 March 2022 at 00:12:12 UTC, Krishna Myneni wrote:
> I am overhauling the string parsing words in the kForth String Words
> library (strings.4th). The existing words are not named as clearly as
> they should be, and have inefficient implementations:
>
> \ PARSE_TOKEN ( a u -- arem urem atok utok )
> \ PARSE_LINE ( a u -- a1 u1 a2 u2 ... an un n )
> \ PARSE_ARGS ( a u -- n ) ( F: r1 ... rn )
> \ PARSE_CSV ( a u -- n ) ( F: r1 ... rn )
>
> PARSE_TOKEN skips leading spaces and parses the next blank(s)-delimited
> substring, atok utok, and also returns the remaining portion of the string.
>
> PARSE_LINE applies PARSE_TOKEN repeatedly to place the token strings and
> token count on the stack.
>
> PARSE_ARGS parses a string with blank delimited numbers, converting each
> substring to a floating point number, returning the n floating point
> numbers and the count.
>
> PARSE_CSV is a hack which replaces each comma in the string with a space
> and performs the same function as PARSE_ARGS.
>
> These words need clearer names and an upgrade (particularly PARSE_CSV).
> My proposed replacements are
>
>
> \ NEXT-BS-TOKEN ( a u -- atok utok arem urem )
> \ NEXT-CS-TOKEN ( a u -- atok utok arem urem )
> \ PARSED-BSV ( a u -- a1 u1 a2 u2 ... an un n )
> \ PARSED-CSV ( a u -- a1 u1 a2 u2 ... an un n )
> \ ITH-PARSED ( a1 u1 ... an un n i -- a1 u1 ... an un n ai ui )
> \ DROP-PARSED ( a1 u1 ... an un n -- )
> \ PARSED>FLOATS ( a1 u1 ... an un n -- n ) ( F: r1 ... rn )
>
> NEXT-BS-TOKEN is a slightly more efficient version of PARSE_TOKEN, but
> leaves the token and remaining strings in a swapped order. The name
> indicates that it is parsing the next "blank(s)-separated" token in the
> string.
>
> NEXT-CS-TOKEN parses the next "comma-separated" token in the string and
> returns the token and remaining substrings.
>
> PARSED-BSV starts with an input string and repeatedly applies
> NEXT-BS-TOKEN until the remaining string is null and returns all of the
> substrings and the token count on the stack.
>
> PARSED-CSV starts with an input string and repeatedly applies
> NEXT-CS-TOKEN until the remaining string is null. It returns all of the
> substrings and the token count on the stack. Unlike PARSED-BSV, each
> comma delimiter will mark a separte token string, e.g. if the input
> string is ",," there will be three token strings, each of length zero.
>
> ITH-PARSED provides a way to PICK the ith substring returned by
> PARSED-BSV and PARSED-CSV.
>
> DROP-PARSED may be used to discard the n token substrings returned by
> PARSED-BSV and PARSED-CSV.
>
> PARSED>FLOATS converts each of the token substrings returned by
> PARSED-BSV or PARSED-CSV to n floating point numbers. If the string to
> float conversion fails, the floating point value NAN will be returned.
> If a substring has zero length, the corresponding fp value will also be NAN.
>
> Comments or suggestions?
>
> --
> Krishna Myneni
>
>
>
>
> The previous set of words will be redefined using the new words (for
> running existing source code); however, they will be considered deprecated.

Comment :

For my situation, I ended up writing two sets of words because I couldnt decide
which solution was better.
(1) one that took strings as arguments and returned string ( a u ) pairs,
(2) one that assumed the string was pointed to by mysrc and myin> and returned
string ( a u ) pairs.

I ended up using solution (2) where I pointed mysrc to the string addr/len and myin> and
parsed the string using forth-like words

Additionally also found it simpler to maintain a vector and accumulate the result
there rather than the stack.

I would be interested to discover if it was something you considered at some point of time.

On 14/03/2022 11:49, NN wrote:
> ...
> For my situation, I ended up writing two sets of words because I couldnt decide
> which solution was better.

I agree. It's near impossible to come up with a set of words that works for
every situation. Other than a few parsing primitives in the kernel which
I use to implement >FLOAT and the like, I leave such things to libraries or
the application itself. The parsers in the kernel are mostly dumb as the
words that use them can check for things such as empty strings.

On 3/13/22 19:49, NN wrote:
> On Sunday, 13 March 2022 at 00:12:12 UTC, Krishna Myneni wrote:
>> I am overhauling the string parsing words in the kForth String Words
>> library (strings.4th). The existing words are not named as clearly as
>> they should be, and have inefficient implementations:
>>
>> \ PARSE_TOKEN ( a u -- arem urem atok utok )
>> \ PARSE_LINE ( a u -- a1 u1 a2 u2 ... an un n )
>> \ PARSE_ARGS ( a u -- n ) ( F: r1 ... rn )
>> \ PARSE_CSV ( a u -- n ) ( F: r1 ... rn )
>>
>> PARSE_TOKEN skips leading spaces and parses the next blank(s)-delimited
>> substring, atok utok, and also returns the remaining portion of the string.
>>
>> PARSE_LINE applies PARSE_TOKEN repeatedly to place the token strings and
>> token count on the stack.
>>
>> PARSE_ARGS parses a string with blank delimited numbers, converting each
>> substring to a floating point number, returning the n floating point
>> numbers and the count.
>>
>> PARSE_CSV is a hack which replaces each comma in the string with a space
>> and performs the same function as PARSE_ARGS.
>>
>> These words need clearer names and an upgrade (particularly PARSE_CSV).
>> My proposed replacements are
>>
>>
>> \ NEXT-BS-TOKEN ( a u -- atok utok arem urem )
>> \ NEXT-CS-TOKEN ( a u -- atok utok arem urem )
>> \ PARSED-BSV ( a u -- a1 u1 a2 u2 ... an un n )
>> \ PARSED-CSV ( a u -- a1 u1 a2 u2 ... an un n )
>> \ ITH-PARSED ( a1 u1 ... an un n i -- a1 u1 ... an un n ai ui )
>> \ DROP-PARSED ( a1 u1 ... an un n -- )
>> \ PARSED>FLOATS ( a1 u1 ... an un n -- n ) ( F: r1 ... rn )
>>
>> NEXT-BS-TOKEN is a slightly more efficient version of PARSE_TOKEN, but
>> leaves the token and remaining strings in a swapped order. The name
>> indicates that it is parsing the next "blank(s)-separated" token in the
>> string.
>>
>> NEXT-CS-TOKEN parses the next "comma-separated" token in the string and
>> returns the token and remaining substrings.
>>
>> PARSED-BSV starts with an input string and repeatedly applies
>> NEXT-BS-TOKEN until the remaining string is null and returns all of the
>> substrings and the token count on the stack.
>>
>> PARSED-CSV starts with an input string and repeatedly applies
>> NEXT-CS-TOKEN until the remaining string is null. It returns all of the
>> substrings and the token count on the stack. Unlike PARSED-BSV, each
>> comma delimiter will mark a separte token string, e.g. if the input
>> string is ",," there will be three token strings, each of length zero.
>>
>> ITH-PARSED provides a way to PICK the ith substring returned by
>> PARSED-BSV and PARSED-CSV.
>>
>> DROP-PARSED may be used to discard the n token substrings returned by
>> PARSED-BSV and PARSED-CSV.
>>
>> PARSED>FLOATS converts each of the token substrings returned by
>> PARSED-BSV or PARSED-CSV to n floating point numbers. If the string to
>> float conversion fails, the floating point value NAN will be returned.
>> If a substring has zero length, the corresponding fp value will also be NAN.
>>
>> Comments or suggestions?
>>
>> --
>> Krishna Myneni
>>
>>
>>
>>
>> The previous set of words will be redefined using the new words (for
>> running existing source code); however, they will be considered deprecated.
>
> Comment :
>
> For my situation, I ended up writing two sets of words because I couldnt decide
> which solution was better.
>
> (1) one that took strings as arguments and returned string ( a u ) pairs,
> (2) one that assumed the string was pointed to by mysrc and myin> and returned
> string ( a u ) pairs.
>
> I ended up using solution (2) where I pointed mysrc to the string addr/len and myin> and
> parsed the string using forth-like words
>
> Additionally also found it simpler to maintain a vector and accumulate the result
> there rather than the stack.
>
>
>
> I would be interested to discover if it was something you considered at some point of time.
>
>

Do you mean that you used standard words like WORD PARSE PARSE-NAME to
parse strings from the input source? That's an alternative. But I think
it doesn't eliminate the need for string parsing words in general.

--
Krishna

On Sunday, March 13, 2022 at 4:36:06 PM UTC-5, the.bee...@gmail.com wrote:
>
> s" 0123456789" sconstant 'digits'
> s" +-" sconstant 'sign'
> char . constant 'point'
>
> 'sign' char-match drop
> 'point' char-equal >r
> 'digits' char-match r> swap >r >r
> 'digits' skip-while r> 0=
> if
> 'point' char-equal
> if
> 'digits' char-match r> and >r
> 'digits' skip-while
> then
> then r>
>
> Flag returned means "match- or-no-match". Easy.
>

frogd
go

: sign "[+-]" ;
: digits "[[:digit:]]" ;
: point "\." ;
: any "*" ;
: some "\{1\}" ;

xx sign any digits any point digits some PATTERN
s -123.456 expr ==> -1
s +123.456 expr ==> -1
s 123.456 expr ==> -1
s .666 expr ==> -1
s -.42 expr ==> -1
s +1 expr ==> 0
s . expr ==> 0
s -. expr ==> 0

-fin-
ok
--
me

On Tuesday, March 15, 2022 at 2:39:50 AM UTC-5, S Jack wrote:

:) frogd
ELF32X86_64 Frog Version 1.0d

"regex" demo

Demo Regular Expression (REGEX)

PATTERN ( sN..s1 -- )

PATTERN concatenates regex sub-strings into one regex pattern
assigned to value SPATTERN

: 'sign' "[+-]" ;
: 'digits' "[[:digit:]]" ;
: 'point' "\." ;
: 'any' "*" ;
: 'some' "\{1\}" ;

xx 'sign' 'any' 'digits' 'any' 'point' 'digits' 'some' PATTERN

EXPR ( s1 -- PASS | FAIL | s2 )

EXPR applies SPATTERN to input string s1 and returns "PASS"
if the s1 matches SPATTERN, "FAIL" otherwise OR extracts string

s2 if s1 matches SPATTERN and SPATTERN contains a sub-pattern enclosed
in parenthesis where s2 is the matched content of the enclosed
sub-pattern.

bl !to s
s -123.456 EXPR ==> PASS
s +123.456 EXPR ==> PASS
s 123.456 EXPR ==> PASS
s .666 EXPR ==> PASS
s -.42 EXPR ==> PASS
s +1 EXPR ==> FAIL
s . EXPR ==> FAIL
s -. EXPR ==> FAIL

xx ".*\(X.*Y\).*" PATTERN
asc ; !to s
s Good magic XyzzY found here; EXPR ==> XyzzY
cr

-fin-
ok
( Not promoting; just showing off and having fun)
( What's not seen in the above pass/fail cases is output is lined up, PASS is white on
green background and FAIL is white on red background.)
--
me

Marcel Hendrix <mhx@iae.nl> writes:
>On Sunday, March 13, 2022 at 10:52:01 AM UTC+1, Anton Ertl wrote:
>> Krishna Myneni <krishna...@ccreweb.org> writes:
>> >Comments or suggestions?
>>
>> In one of my papers [ertl13-strings], I suggest for splitting/parsing
>> to do it one result at a time. We have SEARCH (and SCAN/SKIP) that
>> can help in working that way, but require some additional work around them. In the paper I suggest:
>>
>> |search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
>> |
>> |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
>> |c-a1 u3 is the substring before the first match, c-a4 u4 is the first
>> |match, and c-a5 u5 is the rest of the string, and the TOS is true;
>> |otherwise return false.
>
>One of the first BASIS documents had (LEX) :
>
>FORTH> help (lex)
>(lex) IFORTH
> ( c-addr1 u1 del -- c-addr3 u3 c-addr1 u4 del true ) or
> ( c-addr1 u1 del -- false )
> Break a string in three pieces: before the delimiter del, the delimiter
> itself, and after the delimiter. c-addr3 points to the remaining string,
> c-addr1 points to the string in front of the delimiter. If false is
> returned, the input string does not contain the delimiter character.

Interesting difference of result order compared to what I suggested.

>Coupling this to a full reg-exp package is tempting. However, what I've
>seen with Python users is that they use regexp's all the time, without
>any understanding of how it works. They rely on an online tool that shows
>them how to hack it until the next gotcha.

For the splitting task the benefit of having a regexp is that you can
e.g., use a sequence of chars, or one character from a set as the
in-between things. I expect that it would be very rare to use complex
regexps for that task.

In general, trying to do some sophisticated things with regexp
(especially things they are not very good at, such as complements of
patterns) leads to longish regexps that are hard to debug. And yet,
all the other options tend to be even more cumbersome, so we stick
with regexps even in cases where they obviously reach the limits of
their usability.

The whole problem is not helped by the fact that especially the more
sophisticated features have different syntax in different regexp
implementations.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>On 3/13/22 07:47, Marcel Hendrix wrote:
>> On Sunday, March 13, 2022 at 10:52:01 AM UTC+1, Anton Ertl wrote:
>>> |search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
>>> |
>>> |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
>>> |c-a1 u3 is the substring before the first match, c-a4 u4 is the first
>>> |match, and c-a5 u5 is the rest of the string, and the TOS is true;
>>> |otherwise return false.
....
>A middle ground to using a regexp-based parser could be a word such as
>PARSE-STRING, described by
>
> PARSE-STRING ( caddr1 u1 xt-nexttoken xt-processtoken -- ntokens )
>
>where xt-nexttoken provides the execution semantics for obtaining the
>next token from the input string,
>
> caddr1 u1 -- caddr-rem urem caddr-token utoken
>
>and xt-processtoken provides the execution semantics for processing the
>token,
>
> caddr-token utoken --
>
>The number of tokens parsed and processed is returned by PARSE-STRING.

While there are some who use "token" to mean "lexeme", I make a
difference: A token represents the lexeme and is typically implemented
an integer or address, i.e., in Forth something cell-sized. Therefore
I find the descriptions above confusing.

Anyway, yes, using an xt provides a way to match arbitrary strings,
although not convenient as regexps. However, your xt-nexttoken puts
the additional burden on the xt to walk through the string; this can
be helpful for performance, but makes the xt even more burdensome to
write; interestingly, with regexps one can optimize the
string-scanning (not just matching) case without having to write
additional code (except to fend off unwanted matches).

Xt-processtoken means that PARSE-STRING is a wrapper while
SEARCH-REGEXP is designed such that one would have to write a loop
around it for the same effect. However, if not all parts have to be
treated the same (e.g., the fields of a line in a csv file), the
uniform treatment of the lexemes in the wrapper approach is not
helpful.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>One example of a multicolumn field comes the github repo, H2SPEC, linked
>below. The fortran program in this repo, h2spec.f, generates a
>multicolumn file, h2lines.dat, with each line providing info about a
>spectral line of the hydrogen molecule (in the "vacuum ultraviolet"
>region). An example line in this file, which contains a mixture of
>integer, floating point, and text data fields, is
>
>3226 1589.199 62924.797 0.1125E+08 X 14 3 O B 7 4 O
>
>The first column is an index number, col 2 is the wavelength of the
>spectral line in Angstroms, col 3 is the frequency of the line in units
>of cm^-1, col 4 is the transition rate in units of s^-1, cols 5--8 give
>the quantum numbers/labels for the electronic, vibration, rotation, and
>parity (Ortho/Para hydrogen) describing the lower level of transition,
>and cols 9--12 are the corresponding information for the upper level.
>
>The xt-nexttoken can validate the field type using a simple state
>machine as each token is parsed from the string.

But how would xt-processtoken look for this example?

Anyway, it seems to me that execute-parsing together with parse-name
would work well in this case:

\ incomplete sketch
: parse-level ( -- level )
parse-name str>... \ not sure what to do with that
parse-name str>int
parse-name str>int
parse-name str>int
make-level ;

: process-line ( -- nindex rwavelength rfrequency rtransitionrate lower upper )
parse-name str>int
parse-name str>float
parse-name str>float
parse-name str>float
parse-level
parse-level ;

s" 3226 1589.199 62924.797 0.1125E+08 X 14 3 O B 7 4 O"
' process-line execute-parsing

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2021: https://euro.theforth.net/2021

On 3/18/22 03:29, Anton Ertl wrote:
> Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>> On 3/13/22 07:47, Marcel Hendrix wrote:
>>> On Sunday, March 13, 2022 at 10:52:01 AM UTC+1, Anton Ertl wrote:
>>>> |search-regexp ( c-a1 u1 c-a2 u2 -- c-a1 u3 c-a4 u4 c-a5 u5 true | false )
>>>> |
>>>> |Search for regexp c-a2 u2 in string c-a1 u1; if the regexp is found,
>>>> |c-a1 u3 is the substring before the first match, c-a4 u4 is the first
>>>> |match, and c-a5 u5 is the rest of the string, and the TOS is true;
>>>> |otherwise return false.
> ...
>> A middle ground to using a regexp-based parser could be a word such as
>> PARSE-STRING, described by
>>
>> PARSE-STRING ( caddr1 u1 xt-nexttoken xt-processtoken -- ntokens )
>>
>> where xt-nexttoken provides the execution semantics for obtaining the
>> next token from the input string,
>>
>> caddr1 u1 -- caddr-rem urem caddr-token utoken
>>
>> and xt-processtoken provides the execution semantics for processing the
>> token,
>>
>> caddr-token utoken --
>>
>> The number of tokens parsed and processed is returned by PARSE-STRING.
>
> While there are some who use "token" to mean "lexeme", I make a
> difference: A token represents the lexeme and is typically implemented
> an integer or address, i.e., in Forth something cell-sized. Therefore
> I find the descriptions above confusing.
>

"Token" is used in C programming, e.g. the description of strtok() is,

"The strtok() function breaks a string into a sequence of zero or
more nonempty tokens... Each call to strtok() returns a pointer to a
null-terminated string containing the next token."

Perhaps the more precise term is "lexeme", but it's clear that
interpretation of "token" should be based on the context.

> Anyway, yes, using an xt provides a way to match arbitrary strings,
> although not convenient as regexps. However, your xt-nexttoken puts
> the additional burden on the xt to walk through the string;

Yes.

> this can
> be helpful for performance, but makes the xt even more burdensome to
> write;

This was not the intent, but as I have not yet gotten around to writing
some examples to illustrate the PARSE-STRING approach, using
xt-nexttoken and xt-processtoken, you may be right. The idea was to
allow different approaches to walking through the string, including regexps.

> interestingly, with regexps one can optimize the
> string-scanning (not just matching) case without having to write
> additional code (except to fend off unwanted matches).
>
> Xt-processtoken means that PARSE-STRING is a wrapper while
> SEARCH-REGEXP is designed such that one would have to write a loop
> around it for the same effect. However, if not all parts have to be
> treated the same (e.g., the fields of a line in a csv file), the
> uniform treatment of the lexemes in the wrapper approach is not
> helpful.
>

Yes, if the fields are not uniform, separated by the same delimiter(s),
the burden is shifted to xt-nexttoken and xt-processtoken.

--
Krishna

On 3/18/22 03:58, Anton Ertl wrote:
> Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>> One example of a multicolumn field comes the github repo, H2SPEC, linked
>> below. The fortran program in this repo, h2spec.f, generates a
>> multicolumn file, h2lines.dat, with each line providing info about a
>> spectral line of the hydrogen molecule (in the "vacuum ultraviolet"
>> region). An example line in this file, which contains a mixture of
>> integer, floating point, and text data fields, is
>>
>> 3226 1589.199 62924.797 0.1125E+08 X 14 3 O B 7 4 O
>>
>> The first column is an index number, col 2 is the wavelength of the
>> spectral line in Angstroms, col 3 is the frequency of the line in units
>> of cm^-1, col 4 is the transition rate in units of s^-1, cols 5--8 give
>> the quantum numbers/labels for the electronic, vibration, rotation, and
>> parity (Ortho/Para hydrogen) describing the lower level of transition,
>> and cols 9--12 are the corresponding information for the upper level.
>>
>> The xt-nexttoken can validate the field type using a simple state
>> machine as each token is parsed from the string.
>
> But how would xt-processtoken look for this example?
>

I'll work up an example using xt-nexttoken and xt-processtoken for this
case. Since the fields are limited by blank characters, code for
xt-nexttoken (without fields validation) should be fairly simple. For
this case I would make a column counter, used by both xt-nexttoken and
xt-processtoken. The column counter indexes an array which specifies the
field type and size, allowing xt-nexttoken to perform a check on the
parsed string (if desired) and informs xt-processtoken about how to
convert the string to the proper data format and possibly provides a
structure field offset for storing the data.

A concrete implementation of the above is needed for comparison with
other approaches. I will be interested to see how regexps might be
useful for field validation.

> Anyway, it seems to me that execute-parsing together with parse-name
> would work well in this case:
>
> \ incomplete sketch
> : parse-level ( -- level )
> parse-name str>... \ not sure what to do with that

Return the first non-blank char.

> parse-name str>int
> parse-name str>int
> parse-name str>int
> make-level ;
>
> : process-line ( -- nindex rwavelength rfrequency rtransitionrate lower upper )
> parse-name str>int
> parse-name str>float
> parse-name str>float
> parse-name str>float
> parse-level
> parse-level ;
>
> s" 3226 1589.199 62924.797 0.1125E+08 X 14 3 O B 7 4 O"
> ' process-line execute-parsing
>

Yes, with EXECUTE-PARSING the input string can be processed as you
describe above by PROCESS-LINE. The converted fields stored on the data
and fp stacks will have to be stored into an array of data structures.
With the PARSE-STRING approach, the parsing, string to field conversion,
and storage of the data is handled on a token by token basis by
xt-nexttoken and xt-processtoken, with the latter two tasks being
handled by xt-processtoken.

I have been preoccupied with the complementary problem of creating
output strings, from numeric fields, for output to a file. This was
prompted by your discussion of optimizing "#". The string parsing
problem is useful for reading text data files. Both data file output and
input are, of course, practical necessities for storing and distributing
data. Optimization in both of these areas is worthwhile.

--
Krishna

On 3/18/22 09:44, Krishna Myneni wrote:
> On 3/18/22 03:58, Anton Ertl wrote:
>> Krishna Myneni <krishna.myneni@ccreweb.org> writes:
>>> One example of a multicolumn field comes the github repo, H2SPEC, linked
>>> below. The fortran program in this repo, h2spec.f, generates a
>>> multicolumn file, h2lines.dat, with each line providing info about a
>>> spectral line of the hydrogen molecule (in the "vacuum ultraviolet"
>>> region). An example line in this file, which contains a mixture of
>>> integer, floating point, and text data fields, is
>>>
>>> 3226  1589.199  62924.797 0.1125E+08   X 14  3 O   B  7  4 O
>>>
>>> The first column is an index number, col 2 is the wavelength of the
>>> spectral line in Angstroms, col 3 is the frequency of the line in units
>>> of cm^-1, col 4 is the transition rate in units of s^-1, cols 5--8 give
>>> the quantum numbers/labels for the electronic, vibration, rotation, and
>>> parity (Ortho/Para hydrogen) describing the lower level of transition,
>>> and cols 9--12 are the corresponding information for the upper level.
>>>
>>> The xt-nexttoken can validate the field type using a simple state
>>> machine as each token is parsed from the string.
>>
>> But how would xt-processtoken look for this example?
>>
>
> I'll work up an example using xt-nexttoken and xt-processtoken for this
> case. Since the fields are limited by blank characters, code for
> xt-nexttoken (without fields validation) should be fairly simple. For
> this case I would make a column counter, used by both xt-nexttoken and
> xt-processtoken. The column counter indexes an array which specifies the
> field type and size, allowing xt-nexttoken to perform a check on the
> parsed string (if desired) and informs xt-processtoken about how to
> convert the string to the proper data format and possibly provides a
> structure field offset for storing the data.
>
> A concrete implementation of the above is needed for comparison with
> other approaches. I will be interested to see how regexps might be
> useful for field validation.
>

I have now added an example of my proposed string parsing approach using
PARSE-STRING and xt-nexttoken and xt-processtoken to read, parse, and
store the entire file h2lines.dat. To keep it simple, the example does
not currently do field validation. The program, parse-h2lines.4th, may
be viewed or downloaded at the following link:

https://github.com/mynenik/kForth-64/blob/master/forth-src/parse-h2lines.4th

It runs under kForth and, with a few changes to the include files, shown
below, also runs under Gforth. Gforth processes the entire file,
consisting of 3261 spectral lines, very quickly (in about 4 ms under
gforth-fast). kforth64 is quite sluggish in comparison, about 150 ms,
due, I think, to inefficient implementations of SCAN and SKIP.

The following output is generated when the program is loaded:

Under kforth64
---
Elapsed time = 148 ms
Number of lines = 3261

Selected Lines
1 1108.127 90242.352 3.060e+06 X 0 0 P B 0 1 P
10 991.382 100869.258 5.200e+07 X 0 0 P B 9 1 P
140 1085.145 92153.586 2.575e+07 X 0 4 P B 2 5 P
3261 1449.615 68983.812 1.727e+07 X 14 4 P B 13 5 P
ok
---

Under gforth-fast
---
Elapsed time = 4 ms
Number of lines = 3261

Selected Lines
1 1108.127 90242.352 3.060E6 X 0 0 P B 0 1 P
10 991.382 100869.258 5.200E7 X 0 0 P B 9 1 P
140 1085.145 92153.586 2.575E7 X 0 4 P B 2 5 P
3261 1449.615 68983.812 1.727E7 X 14 4 P B 13 5 P
ok
---

The line list file, h2lines.dat, must be present in the directory. If
you wish to try the example program under your Forth (2012-compatible)
system, you will need a copy of the line list file. The program is meant
to illustrate a general purpose approach to parsing strings into data
fields, with a real, non-trivial example. It may also be used to check
the efficiency of your Forth system's string parsing words.

I am interested in comparing this method with other approaches which
provide the same functionality as this example, and which would be
applicable for general purpose string-to-data field parsing.

The word PARSE-STRING is different from what I proposed originally. In
particular xt-processtoken has the following stack diagram,

xt-processtoken ( caddr u ufield -- )

with ufield being the current field number ( 0, 1, 2, ... ).
PARSE-STRING is now defined as,

: parse-string ( xt-nexttoken xt-processtoken caddr u -- ntokens )
0 >r
BEGIN
2over drop execute
WHILE
r@ 5 pick execute
1 rp@ +!
REPEAT
2drop 2drop 2drop
r> ;

This example uses an xt-nexttoken for obtaining the next
blank(s)-separated substring:

: next-line-field ( caddr1 u1 -- caddr2 u2 caddr3 u3 flag )
next-bs-token dup 0= invert ;

with NEXT-BS-TOKEN defined by

: next-bs-token ( caddr u -- arem urem atok utok )
bl skip 2dup bl scan 2>r r@ - 2r> 2swap ;

Note the order of the remaining and token strings returned on the stack.
The xt-nexttoken only adds the execution semantics of returning a flag
as well, for use by PARSE-STRING.

The xt-processtoken makes use of a table of field processors, which both
convert the token substring and store the converted field into a data
structure array,

: process-line-field ( caddr u ufield -- )
CELLS FieldProcessors + a@ execute ;

where FieldProcessors is the address of the table of individual field
processing xt's, each of which is trivially defined. Writing a basic
field processor is simple but can become complex if one wants to add
field validation as well. There is no requirement to do so, however.

The main loop for reading each line of text from the file and passing it
to PARSE-STRING is pretty simple. It contains

inpLine swap \ caddr u
['] next-line-field ['] process-line-field
2swap parse-string

--
Krishna

For use of parse-h2lines.4th under Gforth, comment out the INCLUDE
statements at the top and add

INCLUDE kforth-compat.fs

\ include ans-words
\ include struct-200x
\ include strings
\ include files
include kforth-compat.fs

>
>> Anyway, it seems to me that execute-parsing together with parse-name
>> would work well in this case:
>>
>> \ incomplete sketch
>> : parse-level ( -- level )
>>    parse-name str>... \ not sure what to do with that
>
> Return the first non-blank char.
>
>>    parse-name str>int
>>    parse-name str>int
>>    parse-name str>int
>>    make-level ;
>>
>> : process-line ( -- nindex rwavelength rfrequency rtransitionrate
>> lower upper )
>>    parse-name str>int
>>    parse-name str>float
>>    parse-name str>float
>>    parse-name str>float
>>    parse-level
>>    parse-level ;
>>
>> s" 3226  1589.199  62924.797 0.1125E+08   X 14  3 O   B  7  4 O"
>> ' process-line execute-parsing
>>

On 3/19/22 09:56, Krishna Myneni wrote:
> ...
> I have now added an example of my proposed string parsing approach using
> PARSE-STRING and xt-nexttoken and xt-processtoken to read, parse, and
> store the entire file h2lines.dat. To keep it simple, the example does
> not currently do field validation. The program, parse-h2lines.4th, may
> be viewed or downloaded at the following link:
>
> https://github.com/mynenik/kForth-64/blob/master/forth-src/parse-h2lines.4th
>
....
> The line list file, h2lines.dat, must be present in the directory. If
> you wish to try the example program under your Forth (2012-compatible)
> system, you will need a copy of the line list file. ...

If you download the H2SPEC files, you may build h2spec with the
following command, under Linux:

$ gfortran -o h2spec h2spec.f

Then, run h2spec to generate the line list, using input parameters shown
below:
---
$ ./h2spec

Compute the H2 spectrum with these conditions:

Rotational temperature (K) >> 300
Lower wavenumber (cm-1) >> 60000
Upper wavenumber (cm-1) >> 90000
FWHM (cm-1) >> 10
$ ---
This will generate two files: the line list, h2lines.dat, and a spectrum
file, h2vuv.dat. Although the parameters entered into the program are
only relevant for the spectrum file, the design of the program does not
allow to only generate the line list.

---
$ ls -l h2lines.dat
-rw-rw-r--. 1 krishna krishna 202182 Mar 19 12:52 h2lines.dat
$ md5sum h2lines.dat
dbb81b16961d0a71e28fece40129def2 h2lines.dat
---

--
Krishna

--
Krishna