I was wondering if there is any problem with adding
a slurp() function to C90 which would do:

fopen()
fseek(SEEK_END)
malloc() of file size
fread of entire file
(not sure about fclose, probably don't do that)

And return a pointer to the malloc() region.

It would be unspecified what happens when two processes
do a slurp of the same file, with the intention that on some
implementations it would mean mmap() MAP_SHARED.

And no ability to extend the file or truncate the file.

And an unslurp to reverse the process, writing the memory
to disk, and again, in an mmap() implementation, just do
an munmap().

I guess slurp() should take "r+b" if you wish to unslurp,
and file is updated in-place rather than being rewritten.
A slurp() with "rb" would not cause a write to be done
on unslurp().

It would be trivial to write a conforming implementation,
while then providing shared memory for systems that have
the required virtual memory smarts.

Perhaps this already exists?

BFN. Paul.

On 2021-05-04, muta...@gmail.com <mutazilah@gmail.com> wrote:
> I was wondering if there is any problem with adding
> a slurp() function to C90 which would do:

Yes, namely that it is 2021. CX for any X <= 2021
is done and dusted.

> fopen()
> fseek(SEEK_END)
> malloc() of file size
> fread of entire file
> (not sure about fclose, probably don't do that)
>
> And return a pointer to the malloc() region.

The best way to ensure the existence of such a function so
that your program can rely on it today is to specify it in
code and add it to your program.
>
> It would be unspecified what happens when two processes
> do a slurp of the same file, with the intention that on some
> implementations it would mean mmap() MAP_SHARED.

It is dishonest to name such a view of the file "slurp", since it isn't
making a copy of the file.

> And no ability to extend the file or truncate the file.

This severely reduces the utility of slurp. The conents of the buffer
should have an independent duration from those of the file so that the
program can rewrite the file, replacing it with a datum which is
calculated from the contents of the slurp buffer.

> And an unslurp to reverse the process, writing the memory
> to disk, and again, in an mmap() implementation, just do
> an munmap().
>
> I guess slurp() should take "r+b" if you wish to unslurp,
> and file is updated in-place rather than being rewritten.
> A slurp() with "rb" would not cause a write to be done
> on unslurp().

This encourages implementing the file transformation by mutating the
buffer in place, while discouraging functional approaches such a
calculating a new buffer from the existing one. It places unreasonable
restrictions on the transformation, not permitting it to extend or
shrink the file. It places unreasonable restrictions on the source and
destination of the update, requiring them to be the same object. What if
the application needs to tweak the data and put it into a diffrent file?

There are considerations of integrity: operations on a writable map are
automatically flushed to the file. If the operation does not complete
by reaching unslurp, the file can be left in a partially updated state,
with no backup. Generally you want to redirect the transformation to a
temporary file and then atomically (if possible) replace the original
file with the successfully produced temp.

The unslurp function can instead take a name argument, and simply open
or create the file for writing, truncate it, if necessary, and then dump
the buffer.

> It would be trivial to write a conforming implementation,

Only to waste effort working around its bad requirements whenever
it is used.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On Tuesday, May 4, 2021 at 12:22:24 PM UTC-3, muta...@gmail.com wrote:
> I was wondering if there is any problem with adding
> a slurp() function to C90 which would do:
>
> fopen()
> fseek(SEEK_END)
> malloc() of file size
> fread of entire file

I am using this..
http://thradams.com/readfile.htm

It uses stat to get the size and it skips BOM if present.

"muta...@gmail.com" <mutazilah@gmail.com> writes:
> I was wondering if there is any problem with adding
> a slurp() function to C90 which would do:
>
> fopen()
> fseek(SEEK_END)
> malloc() of file size
> fread of entire file
> (not sure about fclose, probably don't do that)
>
> And return a pointer to the malloc() region.

The biggest problem is that C90 is obsolete, replaced by newer editions
of the C standard. It will not be updated.

If you want to define your own specification based on C90 plus your
slurp() function, nobody will stop you. (There might be some copyright
issues; I won't get into that.)

But it's such a simple function, you can just write it and include it in
any program that needs it, or you can implement it in a library. I see
no point in adding it to the language standard, especially with so many
unresolved issues.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */

On Wednesday, May 5, 2021 at 5:20:35 AM UTC+10, Keith Thompson wrote:

> If you want to define your own specification based on C90 plus your
> slurp() function, nobody will stop you. (There might be some copyright
> issues; I won't get into that.)

It would be interesting to see ISO take me to court.
I know companies take other companies to court,
but I've never heard of a standards organization
taking an individual (or another standards
organization) to court.

They will have a bit of a problem showing lost
revenue when they refuse to sell C90. They would
at least have to show that they lost revenue from
C21 or whatever because of C90+.

But C90+ would just use C90 as a reference, ie delete
gets() and add '\e', "\e" and slurp/unslurp.

> But it's such a simple function, you can just write it and include it in
> any program that needs it, or you can implement it in a library. I see
> no point in adding it to the language standard,

Well what do you think the criteria should be for
inclusion of a new facility in an ISO language
standard? How do you suggest people read files
into memory so that they can use pointers instead
of file operations?

If they do it manually (malloc/fread) as I have done
myself for 30+ years, it prevents the file from being
made shareable, independent of the application
program itself (it still needs support from the CRT).

> especially with so many unresolved issues.

Can you list them please?

Thanks. Paul.

On Wednesday, May 5, 2021 at 2:31:09 AM UTC+10, Kaz Kylheku wrote:

> > I was wondering if there is any problem with adding
> > a slurp() function to C90 which would do:

> Yes, namely that it is 2021. CX for any X <= 2021
> is done and dusted.

That's what GNU will tell you about GCC 3.2.3 too.
And Binutils 2.14a.

But I have picked both of the above up, and have been
banging them into C90 compliance and fixing bugs
and making enhancements. I forked Hercules 3.07 too,
circa 2011.

Instead of chasing the latest greatest version of things,
I'm trying to get something that actually works to my
satisfaction.

> > fopen()
> > fseek(SEEK_END)
> > malloc() of file size
> > fread of entire file
> > (not sure about fclose, probably don't do that)
> >
> > And return a pointer to the malloc() region.

> The best way to ensure the existence of such a function so
> that your program can rely on it today is to specify it in
> code and add it to your program.

Well, I have the option of adding it to PDPCLIB.
Currently PDPCLIB is pure C90 and I have refused
to diverge one iota from it. One person complained
bitterly about me treating C90 as a sacred text,
willing to throw everything else in the computer
world under the bus.

> > It would be unspecified what happens when two processes
> > do a slurp of the same file, with the intention that on some
> > implementations it would mean mmap() MAP_SHARED.

> It is dishonest to name such a view of the file "slurp", since it isn't
> making a copy of the file.

I'm not sure I agree with that, either part, but I am happy
to rename the function.

> > And no ability to extend the file or truncate the file.

> This severely reduces the utility of slurp. The conents of the buffer
> should have an independent duration from those of the file so that the
> program can rewrite the file, replacing it with a datum which is
> calculated from the contents of the slurp buffer.

I think I agree with this, and actually we need another
two functions to do what you want.

But what I described above is, with the restrictions, was
deliberately designed to allow file sharing among
processes.

> > And an unslurp to reverse the process, writing the memory
> > to disk, and again, in an mmap() implementation, just do
> > an munmap().
> >
> > I guess slurp() should take "r+b" if you wish to unslurp,
> > and file is updated in-place rather than being rewritten.
> > A slurp() with "rb" would not cause a write to be done
> > on unslurp().

> This encourages implementing the file transformation by mutating the
> buffer in place, while discouraging functional approaches such a
> calculating a new buffer from the existing one. It places unreasonable
> restrictions on the transformation, not permitting it to extend or
> shrink the file. It places unreasonable restrictions on the source and
> destination of the update, requiring them to be the same object. What if
> the application needs to tweak the data and put it into a diffrent file?

Yes, see above. You have a non-sharing, and possibly
more common, API requirement.

> There are considerations of integrity: operations on a writable map are
> automatically flushed to the file. If the operation does not complete
> by reaching unslurp, the file can be left in a partially updated state,
> with no backup.

This is up to the implementation. The OS could be
designed to flush the data on program termination.
It is the OS that owns this data at all times anyway
if it is mmap() based.

And the file may be read-only with no integrity issues.

> Generally you want to redirect the transformation to a
> temporary file and then atomically (if possible) replace the original
> file with the successfully produced temp.

Maybe this is something you can negotiate with a
C90+ author or an OS author, for higher integrity.
I don't see a problem with simply saying
"implementation-defined" with regard to that.

> The unslurp function can instead take a name argument, and simply open
> or create the file for writing, truncate it, if necessary, and then dump
> the buffer.

Yes, that's a good idea for the second set of operations.

> > It would be trivial to write a conforming implementation,

> Only to waste effort working around its bad requirements whenever
> it is used.

Sure. That's why slurp/unslurp exist in comp.lang.c not
Sourceforge/PDPCLIB.

Also note that I'm here because a real-world requirement
popped up that forced me to finally investigate memory
mapped files. And it is Windows, so the underlying API
seems to be MapViewOfFile(). But I don't want to code
that directly, even though Sqlite has. I want to move it
out of Sqlite and into PDPCLIB. Move mmap out of
Sqlite too.

BFN. Paul.

"muta...@gmail.com" <mutazilah@gmail.com> writes:
> On Wednesday, May 5, 2021 at 5:20:35 AM UTC+10, Keith Thompson wrote:
>> If you want to define your own specification based on C90 plus your
>> slurp() function, nobody will stop you. (There might be some copyright
>> issues; I won't get into that.)
>
> It would be interesting to see ISO take me to court.

Not to me.

[...]

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */

On Tuesday, 4 May 2021 at 20:20:35 UTC+1, Keith Thompson wrote:
> "muta...@gmail.com" <muta...@gmail.com> writes:
> > I was wondering if there is any problem with adding
> > a slurp() function to C90 which would do:
> >
> > fopen()
> > fseek(SEEK_END)
> > malloc() of file size
> > fread of entire file
> > (not sure about fclose, probably don't do that)
> >
> > And return a pointer to the malloc() region.
> The biggest problem is that C90 is obsolete, replaced by newer editions
> of the C standard. It will not be updated.
>
> If you want to define your own specification based on C90 plus your
> slurp() function, nobody will stop you. (There might be some copyright
> issues; I won't get into that.)
>
> But it's such a simple function, you can just write it and include it in
> any program that needs it, or you can implement it in a library. I see
> no point in adding it to the language standard, especially with so many
> unresolved issues.
>
There is a point. To implement fslurp() portably, you cannot fseek() to the file
end position to get the size, fseek() back, then do an fread(), which would
be the most efficient way of getting the input.
Instead my fslurp() uses an fgetc() loop and an expanding realloc() buffer.
Since efficiency isn't a major concern for most uses of fslurp(), this doesn't
matter too much. But by putting it into the standard, implementations could
use the optimal method.

On 5/5/21 5:11 AM, Malcolm McLean wrote:
> On Tuesday, 4 May 2021 at 20:20:35 UTC+1, Keith Thompson wrote:
>> "muta...@gmail.com" <muta...@gmail.com> writes:
>>> I was wondering if there is any problem with adding
>>> a slurp() function to C90 which would do:
>>>
>>> fopen()
>>> fseek(SEEK_END)
>>> malloc() of file size
>>> fread of entire file
>>> (not sure about fclose, probably don't do that)
>>>
>>> And return a pointer to the malloc() region.
>> The biggest problem is that C90 is obsolete, replaced by newer editions
>> of the C standard. It will not be updated.
>>
>> If you want to define your own specification based on C90 plus your
>> slurp() function, nobody will stop you. (There might be some copyright
>> issues; I won't get into that.)
>>
>> But it's such a simple function, you can just write it and include it in
>> any program that needs it, or you can implement it in a library. I see
>> no point in adding it to the language standard, especially with so many
>> unresolved issues.
>>
> There is a point. To implement fslurp() portably, you cannot fseek() to the file
> end position to get the size, fseek() back, then do an fread(), which would
> be the most efficient way of getting the input.
> Instead my fslurp() uses an fgetc() loop and an expanding realloc() buffer.
> Since efficiency isn't a major concern for most uses of fslurp(), this doesn't
> matter too much. But by putting it into the standard, implementations could
> use the optimal method.
>

That says it wants to be in A Standard and implemented by the
implementation, it doesn't say it needs to be implemented by the C
Standard. It could be in something like POSIX which defines additional
capabilities for the system.

Particularly since you want it back-ported to older Standards, that is
way to do it.

On 05/05/2021 10:11, Malcolm McLean wrote:
> On Tuesday, 4 May 2021 at 20:20:35 UTC+1, Keith Thompson wrote:
>> "muta...@gmail.com" <muta...@gmail.com> writes:
>>> I was wondering if there is any problem with adding
>>> a slurp() function to C90 which would do:
>>>
>>> fopen()
>>> fseek(SEEK_END)
>>> malloc() of file size
>>> fread of entire file
>>> (not sure about fclose, probably don't do that)
>>>
>>> And return a pointer to the malloc() region.
>> The biggest problem is that C90 is obsolete, replaced by newer editions
>> of the C standard. It will not be updated.
>>
>> If you want to define your own specification based on C90 plus your
>> slurp() function, nobody will stop you. (There might be some copyright
>> issues; I won't get into that.)
>>
>> But it's such a simple function, you can just write it and include it in
>> any program that needs it, or you can implement it in a library. I see
>> no point in adding it to the language standard, especially with so many
>> unresolved issues.
>>
> There is a point. To implement fslurp() portably, you cannot fseek() to the file
> end position to get the size, fseek() back, then do an fread(), which would
> be the most efficient way of getting the input.
> Instead my fslurp() uses an fgetc() loop and an expanding realloc() buffer.
> Since efficiency isn't a major concern for most uses of fslurp(), this doesn't
> matter too much. But by putting it into the standard, implementations could
> use the optimal method.
>

You always want it to be as efficient as possible. The program below,
when run on Windows (so relies on msvcrt.dll), takes 450ms to read the
test file mentioned.

That is about the same time as it takes to me compile the whole thing
(sqlite3.c to sqlite3.obj).

Using my normal method of first determining the size then doing a single
fread, it takes about 8ms, over 50 times faster.

In both cases, the filedata will have been cached by the OS.

----------------------------
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define filename "/cx/big/sqlite3.c"

char* pfiledata;
int pcapacity=1024;
int psize;

void newfilebuffer(void){
pfiledata=malloc(pcapacity);
psize=0;
}

char storechar(int c) {
int newfiledata;

++psize;
if (psize>=pcapacity) {
pcapacity*=2;
pfiledata=realloc(pfiledata,pcapacity);
}
pfiledata[psize-1]=c;
}

int main(void) {
FILE* f=fopen(filename,"rb");
int c;

if (f==NULL) exit(0);

newfilebuffer();

while (1) {
c=fgetc(f);
if (c==EOF) break;
storechar(c);
}
storechar(0);
fclose(f);

printf("P=%p size=%d\n", pfiledata, psize);
}

Bart <bc@freeuk.com> writes:

> On 05/05/2021 10:11, Malcolm McLean wrote:
>> On Tuesday, 4 May 2021 at 20:20:35 UTC+1, Keith Thompson wrote:
>>> "muta...@gmail.com" <muta...@gmail.com> writes:
>>>> I was wondering if there is any problem with adding
>>>> a slurp() function to C90 which would do:
>>>>
>>>> fopen()
>>>> fseek(SEEK_END)
>>>> malloc() of file size
>>>> fread of entire file
>>>> (not sure about fclose, probably don't do that)
>>>>
>>>> And return a pointer to the malloc() region.
>>> The biggest problem is that C90 is obsolete, replaced by newer editions
>>> of the C standard. It will not be updated.
>>>
>>> If you want to define your own specification based on C90 plus your
>>> slurp() function, nobody will stop you. (There might be some copyright
>>> issues; I won't get into that.)
>>>
>>> But it's such a simple function, you can just write it and include it in
>>> any program that needs it, or you can implement it in a library. I see
>>> no point in adding it to the language standard, especially with so many
>>> unresolved issues.
>>>
>> ... To implement fslurp() portably, you cannot fseek()
>> to the file end position to get the size, fseek() back, then do an
>> fread(), which would be the most efficient way of getting the input.
>> Instead my fslurp() uses an fgetc() loop and an expanding realloc()
>> buffer.

The question I pose to Bart applies to this too.

>> Since efficiency isn't a major concern for most uses of
>> fslurp(), this doesn't matter too much. But by putting it into the
>> standard, implementations could use the optimal method.
>
> You always want it to be as efficient as possible.

What are you suggesting to make it efficient? I think you posted the
code below is as an example of what you consider inefficient.

> while (1) {
> c=fgetc(f);
> if (c==EOF) break;
> storechar(c);
> }

Why not use fread and expand the size to read as the buffer grows? That
may not help much on systems with very good fgetc implementations, but
it can't hurt.

BTW, Bart writes non-canonical C. I think most C programmers would
write

while ((c = fgetc(f)) != EOF) storechar(c);

(with layout etc. adjusted for preference).

--
Ben.

On 05/05/2021 15:19, Ben Bacarisse wrote:
> Bart <bc@freeuk.com> writes:
>
>> On 05/05/2021 10:11, Malcolm McLean wrote:

>>> Instead my fslurp() uses an fgetc() loop and an expanding realloc()
>>> buffer.

>> You always want it to be as efficient as possible.
>
> What are you suggesting to make it efficient? I think you posted the
> code below is as an example of what you consider inefficient.

It uses Malcolm's suggestion of an fgetc loop.

>> while (1) {
>> c=fgetc(f);
>> if (c==EOF) break;
>> storechar(c);
>> }
>
> Why not use fread and expand the size to read as the buffer grows?

You mean, tentatively read as much as the buffer allows, and monitor the
number of bytes actually read?

I was going to try it, but realised it's going to be rather fiddly.
(However a quick test reading my test file as 7756 1KB blocks with
fread, into a fixed 8MB buffer, was nearly as fast as my single fread of
the whole file.)

The fast is, nearly every file I'm going to read in using my whole-file
method is going to be small. My test file that took 8ms here is over 100
times bigger than a typical input file, and still only takes up 0.1% of
my machine's memory.

It's a non-issue. If I was to encounter a file that is too big to fit
into memory, then it will be too big whichever method I use.

On 2021-05-04, muta...@gmail.com <mutazilah@gmail.com> wrote:
> Sure. That's why slurp/unslurp exist in comp.lang.c not
> Sourceforge/PDPCLIB.
>
> Also note that I'm here because a real-world requirement
> popped up that forced me to finally investigate memory
> mapped files. And it is Windows, so the underlying API
> seems to be MapViewOfFile(). But I don't want to code
> that directly, even though Sqlite has. I want to move it
> out of Sqlite and into PDPCLIB. Move mmap out of
> Sqlite too.

I think you're neglecting to consider the viewpoint that a group of
people already took the C standard and made a more extensive standard
which includes features like memory mapping of files. That standard is
POSIX.

There was POSIX in 1990, which extended C90 with useful functions.

To be fair, the ISO C people have themselves begun to neglect this view
and started reinventing POSIX wheels, such as adding threads to C that
are similar to POSIX threads, with different naming.

What should have happened is that C and POSIX people should have
negotiated to migrate parts of POSIX threads into C, with exactly the
same syntax. The same <pthread.h> header and pthread_mutex_lock
and whatever. All the more simple, easily portable aspects could be
encoded in C, so that POSIX then just describes extensions, like real
time signals or whatever is not put into C.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

On Wednesday, 5 May 2021 at 15:20:01 UTC+1, Ben Bacarisse wrote:
> Bart <b...@freeuk.com> writes:
>
> > On 05/05/2021 10:11, Malcolm McLean wrote:
> >> On Tuesday, 4 May 2021 at 20:20:35 UTC+1, Keith Thompson wrote:
> >>> "muta...@gmail.com" <muta...@gmail.com> writes:
> >>>> I was wondering if there is any problem with adding
> >>>> a slurp() function to C90 which would do:
> >>>>
> >>>> fopen()
> >>>> fseek(SEEK_END)
> >>>> malloc() of file size
> >>>> fread of entire file
> >>>> (not sure about fclose, probably don't do that)
> >>>>
> >>>> And return a pointer to the malloc() region.
> >>> The biggest problem is that C90 is obsolete, replaced by newer editions
> >>> of the C standard. It will not be updated.
> >>>
> >>> If you want to define your own specification based on C90 plus your
> >>> slurp() function, nobody will stop you. (There might be some copyright
> >>> issues; I won't get into that.)
> >>>
> >>> But it's such a simple function, you can just write it and include it in
> >>> any program that needs it, or you can implement it in a library. I see
> >>> no point in adding it to the language standard, especially with so many
> >>> unresolved issues.
> >>>
> >> ... To implement fslurp() portably, you cannot fseek()
> >> to the file end position to get the size, fseek() back, then do an
> >> fread(), which would be the most efficient way of getting the input.
> >> Instead my fslurp() uses an fgetc() loop and an expanding realloc()
> >> buffer.
> The question I pose to Bart applies to this too.
> >> Since efficiency isn't a major concern for most uses of
> >> fslurp(), this doesn't matter too much. But by putting it into the
> >> standard, implementations could use the optimal method.
> >
> > You always want it to be as efficient as possible.
> What are you suggesting to make it efficient? I think you posted the
> code below is as an example of what you consider inefficient.
> > while (1) {
> > c=fgetc(f);
> > if (c==EOF) break;
> > storechar(c);
> > }
> Why not use fread and expand the size to read as the buffer grows? That
> may not help much on systems with very good fgetc implementations, but
> it can't hurt.
>
There's a case for an optimal version which isn't portable, and there's a
case for a version which isn't optimised, without being catastrophic, because
the run time will be insignificant in relation to everything else. There's a
weaker case for a semi-optimised version.
>
> BTW, Bart writes non-canonical C. I think most C programmers would
> write
>
> while ((c = fgetc(f)) != EOF) storechar(c);
>
> (with layout etc. adjusted for preference).
>
You said it. People are used to the pattern of reading a stream
a character at a time, then processing it character by character
within the read loop.

On Thursday, May 6, 2021 at 1:41:34 AM UTC+10, Kaz Kylheku wrote:

> > Also note that I'm here because a real-world requirement
> > popped up that forced me to finally investigate memory
> > mapped files. And it is Windows, so the underlying API
> > seems to be MapViewOfFile(). But I don't want to code
> > that directly, even though Sqlite has. I want to move it
> > out of Sqlite and into PDPCLIB. Move mmap out of
> > Sqlite too.

> I think you're neglecting to consider the viewpoint that a group of
> people already took the C standard and made a more extensive standard
> which includes features like memory mapping of files. That standard is
> POSIX.
>
> There was POSIX in 1990, which extended C90 with useful functions.
>
> To be fair, the ISO C people have themselves begun to neglect this view
> and started reinventing POSIX wheels, such as adding threads to C that
> are similar to POSIX threads, with different naming.
>
> What should have happened is that C and POSIX people should have
> negotiated to migrate parts of POSIX threads into C, with exactly the
> same syntax. The same <pthread.h> header and pthread_mutex_lock
> and whatever. All the more simple, easily portable aspects could be
> encoded in C, so that POSIX then just describes extensions, like real
> time signals or whatever is not put into C.

Ok, that's a good point - why not just copy aspects
of POSIX. You can also ask why Windows didn't do
that too. Or since it's too late now, do the reverse -
why doesn't C copy Windows?

I've just taken another look at mmap, to see if it can
be used unchanged, no need for slurp().

https://man7.org/linux/man-pages/man2/mmap.2.html

#include <sys/mman.h>

Directories don't exist in MVS.

int fd

Files are not numbered in MVS. They do have an
associated DCB though, so if POSIX had said:

DCB *dcb

instead, MVS users may not have complained.

If addr is NULL

This is complicating the implementation. C90 seems
to be simple.

(page-aligned)

No such thing as pages in MSDOS.

offset must be a multiple of the page size as returned by sysconf(_SC_PAGE_SIZE).

Yikes! That just made my brain freeze.

After the mmap() call has returned, the file descriptor, fd, can
be closed immediately without invalidating the mapping.

Sounds like a recipe for disaster to me. You need to
fopen() the file again to write it out, and hope it still
exists. What's the benefit of closing a file before you
have finished using it? After all the effort of opening
it.

The prot argument describes the desired memory protection of the mapping

No such thing as memory protection on 8086. This
would be the first thing added to C90 that introduced
the concept.

PROT_EXEC
Pages may be executed.

PROT_READ
Pages may be read.

PROT_WRITE
Pages may be written.

PROT_NONE
Pages may not be accessed.

Not only is the concept being introduced - it's coming
in by storm!

The flags argument
The flags argument determines whether updates to the mapping are
visible to other processes mapping the same region,

No such thing as processes in MSDOS. Another new
concept for C90.

MAP_SHARED_VALIDATE

Made my brain freeze about how anyone could even
conceive of something so complicated.

EOPNOTSUPP

A new errno not seen in C90.

MAP_PRIVATE
Create a private copy-on-write mapping.

You expect an MSDOS programmer trying to slurp a
file into memory, a conceptually simple thing, to come
to grips with whether they want a copy-on-write
mapping if they ever end up on a Unix system?

MAP_SHARED_VALIDATE is a Linux extension

There's no reason to stop there. You can incorporate
all the BSD extensions into C90+ too. And Windows.

MAP_32BIT (since Linux 2.4.20, 2.6)
Put the mapping into the first 2 Gigabytes of the process

C90 doesn't have much concept of 32-bit. There is
only one place - the minimum size of a long.

That's enough for now. If people want to create a
sophisticated, largely Unix-specific API, that's fine.
Windows has one too, and so does MVS. And
MSDOS does too. MSDOS even has an API that
allows you to mark a file as "hidden". I'm not sure
POSIX even has that concept.

But this is not suitable for C90 or the proposed C90+.

slurp(), or as someone else has renamed it (and I think
it is a good change), fslurp(), is the conceptually simple
task of reading a file into memory in a single, simple
operation that is easy to implement.

If it does a complicated mmap() behind the scenes,
on just Unix-like platforms, that's fine, and in fact,
that's the exact intention. But it is the nature of C90
to be discreet about why it has carefully worded
things. They may expose their thinking in the
Rationale though, and maybe they explain why they
wrote that you can rely on numbers being contiguous
code points, but not letters (because of EBCDIC).

I'm not sure how much of POSIX can actually be
incorporated into C, and especially not starting
from C90. POSIX seems to have been designed to
standardize Unix. That would be like if there were
5 competitors for Windows and they were trying
to standardize what they actually had. I've never
heard of anyone trying to put the Windows API
into C, even picking and choosing.

Maybe I can pose a different question - if your boss
told you you needed to code in C90 with 5 extensions,
what would those 5 extensions be?

I guess what I'm trying to do is take C90 - noting that
C90 isn't even universally implemented yet - and then
negotiate one change at a time, based on reasonable,
real-world (especially in 1990, with MSDOS all the
rage) need. But with the benefit of hindsight. So I'd
like to negotiate whether we need far pointers or not.
Maybe we don't need that, but what we instead need
is a keyword abs_addr. And you can set an absolute
address from an integer type, ie:

abs_addr x;

set_abs_addr(&x, 0xb8000UL);

But forget about that for now. That's just an example.

Do you agree that C90, even in 1990, or maybe 1991,
could be extended with an fslurp() so that one day
in the future, when a 4 GB machine is running 10
processes that all use the same 1.9 GB file, read-only
file, they can use pointers instead of file operations?
Sharing the exact same real memory, thanks to
virtual memory being available?

Is the concept of reading a file into memory for
efficient processing something that would have
been required in 1990? Maybe the scheme needs
to be extended, with some sort of API that wraps
the pointer access and falls back to file operations
if the file is too big to fit into memory.

But it seems to me that this concept of data being
in either memory or a file is pretty fundamental.
They are the only choices in C90. And there should
be some consideration given to switching between
the two for storage/processing. Yes, fread() exists,
but is that where things should end? Or is an fslurp()
logical?

And for that matter, why is a file size being given as
"long"? There was a size_t created for memory
buffers. Why not a corresponding type for files?
Who created size_t anyway? And in C90 they didn't
provide a way of printing out a size_t. So how are
you supposed to print out:

printf("your string is %xxxx bytes long\n", strlen(p));

What did the ISO committee expect people to do?

But nevermind about ISO. I'd rather negotiate with
comp.lang.c about changes, probably minimal, or
at least, one at a time, to C90. Additions and
subtractions. With fslurp() on the agenda at the
moment.

I have a C90 library already (PDPCLIB), so I can add
fslurp() to it as soon as the comp.lang.c committee
agrees on an extension to C90. Based on their
knowledge in 2021 (which I am largely lacking - I'm
still largely in 1990 - I deliberately held myself back -
especially spending a lot of time on MVS 3.8J from
the 1980s).

I also have GCC 3.2.3 as a C90-compliant compiler,
so I can make language changes such as adding
'\e' for ESC, but that is more difficult than changing
PDPCLIB. Maybe it's better to add ESC_CHAR and
ESC_STRING to stddef.h. If a platform implements
them as '\e' and "\e", that should be allowed, as
that escape sequence can be "reserved for future
use as an ESC character".

I'm here to negotiate. Please. Someone from 2021
negotiate. Bridge the time gap. Help me prepare
for time travel in case I am forced to leave 1990.
I'll tell you what the situation is in 1990 in case
you've forgotten, or never knew. I've got a public
domain replacement for MSDOS, and the divide
by zero interrupt (INT 0) has just been implemented,
and prints MVS-style diagnostics when that occurs,
ie the registers, load point of the program etc. Not
sure why MSDOS itself didn't do that, when it has
MVS as a reference. I'm mulling the possibility of
an 8086-like environment/processor that offers
rudimentary memory protection so that the MSDOS
replacement (PDOS/86) can protect itself from
applications. Then you will get diagnostics not
just from divide by zero errors but also memory
access violations. And of course, the segment shift
will be flexible, set to either 4 bits or 16 bits.

On 05/05/2021 23:40, muta...@gmail.com wrote:
> On Thursday, May 6, 2021 at 1:41:34 AM UTC+10, Kaz Kylheku wrote:
>
>>> Also note that I'm here because a real-world requirement
>>> popped up that forced me to finally investigate memory
>>> mapped files. And it is Windows, so the underlying API
>>> seems to be MapViewOfFile(). But I don't want to code
>>> that directly, even though Sqlite has. I want to move it
>>> out of Sqlite and into PDPCLIB. Move mmap out of
>>> Sqlite too.
>
>> I think you're neglecting to consider the viewpoint that a group of
>> people already took the C standard and made a more extensive standard
>> which includes features like memory mapping of files. That standard is
>> POSIX.
>>
>> There was POSIX in 1990, which extended C90 with useful functions.
>>
>> To be fair, the ISO C people have themselves begun to neglect this view
>> and started reinventing POSIX wheels, such as adding threads to C that
>> are similar to POSIX threads, with different naming.
>>
>> What should have happened is that C and POSIX people should have
>> negotiated to migrate parts of POSIX threads into C, with exactly the
>> same syntax. The same <pthread.h> header and pthread_mutex_lock
>> and whatever. All the more simple, easily portable aspects could be
>> encoded in C, so that POSIX then just describes extensions, like real
>> time signals or whatever is not put into C.
>
> Ok, that's a good point - why not just copy aspects
> of POSIX. You can also ask why Windows didn't do
> that too.

Why would Windows do that? It's got its own API will which work on every
version of Windows.

But there were a million different versions of Unix, and POSIX as I
understand it was to tie them all together. That's what the "IX" of
POSIX was for.

> Or since it's too late now, do the reverse -
> why doesn't C copy Windows?

I'm glad it didn't because the C standard API is a lot easier to
Windows' API.

(That's actually when I started using the standard C library, because it
was just a much simpler library to use than WinAPI. I didn't even
realise it was meant to be used from C; it was just another library that
came with Windows.)

POSIX itself I think is about 80 different headers; I wouldn't want to
get involved with that either. It's so annoying when you want to build
something on Windows, and C programs use POSIX headers instead of
standard C headers

On 5/5/2021 8:41 AM, Kaz Kylheku wrote:
> On 2021-05-04, muta...@gmail.com <mutazilah@gmail.com> wrote:
>> Sure. That's why slurp/unslurp exist in comp.lang.c not
>> Sourceforge/PDPCLIB.
>>
>> Also note that I'm here because a real-world requirement
>> popped up that forced me to finally investigate memory
>> mapped files. And it is Windows, so the underlying API
>> seems to be MapViewOfFile(). But I don't want to code
>> that directly, even though Sqlite has. I want to move it
>> out of Sqlite and into PDPCLIB. Move mmap out of
>> Sqlite too.
>
> I think you're neglecting to consider the viewpoint that a group of
> people already took the C standard and made a more extensive standard
> which includes features like memory mapping of files. That standard is
> POSIX.
>
> There was POSIX in 1990, which extended C90 with useful functions.
>
> To be fair, the ISO C people have themselves begun to neglect this view
> and started reinventing POSIX wheels, such as adding threads to C that
> are similar to POSIX threads, with different naming.
>
> What should have happened is that C and POSIX people should have
> negotiated to migrate parts of POSIX threads into C, with exactly the
> same syntax. The same <pthread.h> header and pthread_mutex_lock
> and whatever. All the more simple, easily portable aspects could be
> encoded in C, so that POSIX then just describes extensions, like real
> time signals or whatever is not put into C.
>

Last time I checked, POSIX has no means for portable fine grain memory
barriers and atomic operations. I loved it when C/C++ finally supported
them. Still do. :^)

On 5/5/2021 9:19 AM, Ben Bacarisse wrote:
> Bart <bc@freeuk.com> writes:
>
>> On 05/05/2021 10:11, Malcolm McLean wrote:
>>> On Tuesday, 4 May 2021 at 20:20:35 UTC+1, Keith Thompson wrote:
>>>> "muta...@gmail.com" <muta...@gmail.com> writes:
>>>>> I was wondering if there is any problem with adding
>>>>> a slurp() function to C90 which would do:
>>>>>
>>>>> fopen()
>>>>> fseek(SEEK_END)
>>>>> malloc() of file size
>>>>> fread of entire file
>>>>> (not sure about fclose, probably don't do that)
>>>>>
>>>>> And return a pointer to the malloc() region.
>>>> The biggest problem is that C90 is obsolete, replaced by newer editions
>>>> of the C standard. It will not be updated.
>>>>
>>>> If you want to define your own specification based on C90 plus your
>>>> slurp() function, nobody will stop you. (There might be some copyright
>>>> issues; I won't get into that.)
>>>>
>>>> But it's such a simple function, you can just write it and include it in
>>>> any program that needs it, or you can implement it in a library. I see
>>>> no point in adding it to the language standard, especially with so many
>>>> unresolved issues.
>>>>
>>> ... To implement fslurp() portably, you cannot fseek()
>>> to the file end position to get the size, fseek() back, then do an
>>> fread(), which would be the most efficient way of getting the input.
>>> Instead my fslurp() uses an fgetc() loop and an expanding realloc()
>>> buffer.
>
> The question I pose to Bart applies to this too.
>
>>> Since efficiency isn't a major concern for most uses of
>>> fslurp(), this doesn't matter too much. But by putting it into the
>>> standard, implementations could use the optimal method.
>>
>> You always want it to be as efficient as possible.
>
> What are you suggesting to make it efficient? I think you posted the
> code below is as an example of what you consider inefficient.
>
>> while (1) {
>> c=fgetc(f);
>> if (c==EOF) break;
>> storechar(c);
>> }
>
> Why not use fread and expand the size to read as the buffer grows? That
> may not help much on systems with very good fgetc implementations, but
> it can't hurt.
>
> BTW, Bart writes non-canonical C. I think most C programmers would
> write
>
> while ((c = fgetc(f)) != EOF) storechar(c);
>
> (with layout etc. adjusted for preference).
>

technical competence.

--
Copyright 2021 Pete Olcott

"Great spirits have always encountered violent opposition from mediocre
minds." Einstein