Hi Kent,

>What is your suggested algorithm for auto-detecting the sector order of
>.dsk images?

Maybe there's yet another misunderstanding because I wasn't precise
enough: In my POV, the only ProDOS ordered .DSK files acceptable are
ones actually containing ProDOS.

For a ProDOS disk in ProDOS order, the complete 512 bytes ProDOS boot
block is located at offset $000 - in contrast to a ProDOS disk in
DOS33 order having the second 256 bytes of the ProDOS boot block at
offset $E00.

At least up to ProDOS 2.4.x, the ProDOS boot block of all ProDOS
versions has the string "PRODOS " in the second 256 bytes. So
if that string is found at $100-$1FF, the .DSK file contains ProDOS in
ProDOS order.

Every .DSK file that doesn't match this criteria, is to be considered
in DOS33 order.

Regards,
Oliver

PS: If the ProDOS 2.5 boot block doesn't contain that string anymore,
than ProDOS 2.5 disk images MUST be named .PO.

On 10/30/22 5:43 PM, fadden wrote:
> On Sunday, October 30, 2022 at 11:07:53 AM UTC-7, Kent Dickey wrote:
>> Can you describe CiderPress's algorithm for determining sector order of
>> a .dsk file?
>
> "Complicated".

I've mirrored the same algorithm in AppleCommander. It works in all
cases where a person has a reasonable expectation of there being
reasonable data laid down by a reasonable operating system.

Of note - ADTPro (and ADT before it) always has written 140k Disk II
images in DOS order no matter what. Every other disk type writes images
in ProDOS order no matter what.

Kent Dickey wrote:
>
> Can you share some of this information? Do you know roughly what is the
> percentage of .dsk images that are in ProDOS order?

Not off hand, but I can get that if you are really curious.

>
> I think what you're saying is, if someone wants to throw together an Apple
> II
> emulator (say, written in whatever new language is hot this year), before
> releasing it they need to:
>
> - Write .dsk auto-detect code. As far as I know, my description of
> AppleWin's
> algorithm is the first public documentation of .dsk sector order
> detection. Otherwise, they would need to do a bunch of research to
> decide what to do. Note: other disk formats other than DOS 3.3 and
> ProDOS are not covered, such as Pascal.

Or CP/M. I wrote detectors for these formats too, and I can produce a
catalog of their contents programmatically.

>
> - Download every Apple II .dsk image available on the web.
> How should I go about doing this? I am unable to see how to download
> all Apple II files from macgui.com, can you provide some help?

I started at Asimov. I would recommend that as a source for an Apple II
emulator author who wants a big stack of disk images to test against.

>
> - Test their auto-detect code and make sure they don't make any mistakes.
>
> This seems like a pretty high bar, and I'm not sure any existing emulator
> meets this requirement.

Indeed, which is why I've often found using the real thing to be much more
satisfying.

--
]DF$
The New Apple II User's Guide:
https://macgui.com/newa2guide/

> What is your suggested algorithm for auto-detecting the sector order of
> .dsk images?

I imagined this as an auto-detector.

I like the way Sweet16 loads disk images. It makes a best guess as to what order the disk image is and if unable to decide it loads it as a RAW file. And due to this RAW file format, one can load Commodore 64 disk images which are 174,848 bytes compared to Apple II disk image of 143,360 bytes. Which was then easy to convert C64 T/S's to DOS T/S's.

I would assume the best way to auto-detect an image is to load all disk images as RAW data which can be used by an OS that reads/writes blocks. It is just simple math that converts to Track/Sectors.

A Prodos image is identified as having these standard unchanged bytes that would be extremely rare to have duplicated on an image that is just data.

Block #2 - has byte #2 = $03 and byte #4 = $Fx
Block #3 - has byte #2 = $04
Block #4 - has byte #2 = $05

A DOS image has this:

Block #88 (which calculates to Track 17 Sector 0) - bytes #1 & #2 show the Track/Sector start of the first Sector of the directory. Normally shows $11 0F, but modified disks can be anything. I have on occasion changed this to $11 01 and then have Track $11 Sector $01 - byte #1 changed to a $00 to indicate the end of the directory. This allows only 7 file names but frees up the other 14 sectors of the directory for data on Track 17.

But technically, byte #1 of Track $11 Sector 0 should match byte #1 of the directory sector it points to. for example, if byte #1 and #2 of Track $11 Sector 0 is $11 0F, then byte #1 of Track $11 Sector $0F should also be $11, but can also be a zero for end of directory. This is very easy to find if the first directory sector is in the same track as the VTOC track number and since Sector $0F is in the second half of Block #88. From an emulators perspective should be very easy to get and compare byte #1 of the VTOC and byte #1 of the first sector of the directory to determine the disk image is a DOS disk image. But byte #1 of the next T/S pointer would still have to be read in to determine if this disk is DOS order or Prodos order.

PCDOS and HFS are also very easy to determine using this RAW block method, as well as I imagine CP/M would be. I think an emulator should read in the entire track of $0-F and $11-1F into memory, then it should be able to easily identify any disk image that gets loaded.

With values in hex...

If I may, block 88 can be changed to getting the value of T1/SB/01. It contains the track of the VTOC. Using it allows to handle VTOC that are not on T11.

Antoine

> With values in hex...
>
> If I may, block 88 can be changed to getting the value of T1/SB/01. It contains the track of the VTOC. Using it allows to handle VTOC that are not on T11.
>
> Antoine

Is this part of DOS and is it the same for all versions of DOS? Including DOS3.2 and Third Party DOS's?

On Wednesday, November 2, 2022 at 3:16:19 PM UTC-7, gid...@sasktel.net wrote:
> > If I may, block 88 can be changed to getting the value of T1/SB/01. It contains the track of the VTOC. Using it allows to handle VTOC that are not on T11.
> >
> > Antoine
> Is this part of DOS and is it the same for all versions of DOS? Including DOS3.2 and Third Party DOS's?

DOS: track 17 sector 0 holds the VTOC. Everything else is potentially variable.

ProDOS: block 2 holds the start of the volume directory. Everything else is potentially variable.

Most disks follow the standard pattern: track 17 is a full catalog track, ProDOS has 4 volume directory blocks followed by the bitmap. Detecting "most disks" isn't the interesting part of the problem though. :-)

> > > If I may, block 88 can be changed to getting the value of T1/SB/01. It contains the track of the VTOC. Using it allows to handle VTOC that are not on T11.
> > >
> > > Antoine
> > Is this part of DOS and is it the same for all versions of DOS? Including DOS3.2 and Third Party DOS's?
> DOS: track 17 sector 0 holds the VTOC. Everything else is potentially variable.
>
> ProDOS: block 2 holds the start of the volume directory. Everything else is potentially variable.
>
> Most disks follow the standard pattern: track 17 is a full catalog track, ProDOS has 4 volume directory blocks followed by the bitmap. Detecting "most disks" isn't the interesting part of the problem though. :-)

I was taking Antoine's comment of T1/SB/01 as the location of the VTOC within DOS. Unless that was a typing error and was supposed to be T11/SB/01, in which case it has a different meaning.

The value of Track 17 Sector 0 for the VTOC is stored somewhere in Tracks 0-2 which is usually DOS, unless the disk is DOSless. The question I was asking Antoine was, is the VTOC Track/Sector number in the same location for all versions of DOS including third party DOSes that one can use a standard routine to find the VTOC without assuming it is always Track 17.

On Wednesday, November 2, 2022 at 7:09:31 PM UTC-7, gid...@sasktel.net wrote:
> I was taking Antoine's comment of T1/SB/01 as the location of the VTOC within DOS. Unless that was a typing error and was supposed to be T11/SB/01, in which case it has a different meaning.

T1/S11 holds the code loaded into $ac00-acff. It starts with the tail end of a function that initializes some file manager stuff, including the VTOC location (the byte at +$01 in holds the track number). The value would not be useful if the disk weren't bootable. There are likely some disks with a relocated VTOC that would benefit from looking there, though you'd want to test the code around that location to see if it looked like DOS before putting too much faith in what you find.

For DOS 3.2, it would be T2/S1. Custom DOSs often just made a few in-place edits, so standard routines tended to stay in the standard places.

If you wanted to include this in the strategy, it might be best applied as a fallback if T17/S0 doesn't look like a VTOC, and you were about to write the disk off as "unrecognized".

D Finnigan wrote:
> Kent Dickey wrote:
>>
>> Can you share some of this information? Do you know roughly what is the
>> percentage of .dsk images that are in ProDOS order?
>
> Not off hand, but I can get that if you are really curious.
>

Counting ProDOS format disks:

mysql> SELECT COUNT(*) FROM files WHERE file_format = 11;
+----------+
| COUNT(*) |
+----------+
| 2700 |
+----------+
1 row in set (0.10 sec)

Counting ProDOS format disks with file extension .dsk:

SELECT COUNT(*) FROM files WHERE file_format = 11
-> AND file_url LIKE '%.dsk.%';
+----------+
| COUNT(*) |
+----------+
| 2341 |
+----------+
1 row in set (0.10 sec)

Hi David,

> SELECT COUNT(*) FROM files WHERE file_format = 11;
> +----------+
> | 2700 |
> +----------+
> SELECT COUNT(*) FROM files WHERE file_format = 11
> -> AND file_url LIKE '%.dsk.%';
> +----------+
> | 2341 |
> +----------+

Thanks for sharing :-)

The result seems even (much) more clear than I expected: .DSK files with
ProDOS order are simply the de facto standard. Period.

Regards,
Oliver

On Monday, November 7, 2022 at 3:38:07 PM UTC-8, Oliver Schmidt wrote:
> The result seems even (much) more clear than I expected: .DSK files with
> ProDOS order are simply the de facto standard. Period.

The results indicate that 86% of ProDOS disk images are called ".dsk". It does not indicate what percentage of ".dsk" are ProDOS. I expect that's just as easy to pull out... but it doesn't matter. The goal should be to eliminate ambiguity, not clarify its bounds so we can cope with it using best-effort heuristics.

Is this discussion moving in circles at high speed? :-)

In article <dog_cow-1667828188@macgui.com>,
D Finnigan <dog_cow@macgui.com> wrote:
>D Finnigan wrote:
>> Kent Dickey wrote:
>>>
>>> Can you share some of this information? Do you know roughly what is the
>>> percentage of .dsk images that are in ProDOS order?
>>
>> Not off hand, but I can get that if you are really curious.
>>
>
>Counting ProDOS format disks:
>
>mysql> SELECT COUNT(*) FROM files WHERE file_format = 11;
>+----------+
>| COUNT(*) |
>+----------+
>| 2700 |
>+----------+
>1 row in set (0.10 sec)
>
>
>
>Counting ProDOS format disks with file extension .dsk:
>
>
>SELECT COUNT(*) FROM files WHERE file_format = 11
> -> AND file_url LIKE '%.dsk.%';
>+----------+
>| COUNT(*) |
>+----------+
>| 2341 |
>+----------+
>1 row in set (0.10 sec)

Can you provide some background on what your total images are, and
what it means to be ProDOS format? Using the output from Andy's file list
can only show which disks have a ProDOS filesystem on it, not which have
sectors in ProDOS order.

Kent

Hi Kent,

> Using the output from Andy's file list
> can only show which disks have a ProDOS filesystem on it, not which have
> sectors in ProDOS order.

That's of course the relevant question. Sorry for jumping to conclusions in
my last post.

Regards,
Oliver

D Finnigan wrote:
> D Finnigan wrote:
>> Kent Dickey wrote:
>>>
>>> Can you share some of this information? Do you know roughly what is the
>>> percentage of .dsk images that are in ProDOS order?
>>
>> Not off hand, but I can get that if you are really curious.
>>
>
> Counting ProDOS format disks:
>
> mysql> SELECT COUNT(*) FROM files WHERE file_format = 11;
> +----------+
> | COUNT(*) |
> +----------+
> | 2700 |
> +----------+
> 1 row in set (0.10 sec)
>
>
>
> Counting ProDOS format disks with file extension .dsk:
>
>
> SELECT COUNT(*) FROM files WHERE file_format = 11
> -> AND file_url LIKE '%.dsk.%';
> +----------+
> | COUNT(*) |
> +----------+
> | 2341 |
> +----------+
> 1 row in set (0.10 sec)
>

I made a mistake. :-( These queries don't answer the question of how many
disk images are in ProDOS order. Instead, these queries count disk images
which use the ProDOS file system.

I can still answer the desired question, but I'll need to write a little bit
of code to record whether a disk image was in DOS or ProDOS order. Right
now, my code discovers what order is used, but after the disk catalog is
generated, the order is not recorded. (I'm talking about my code that
generates disk catalogs for the Downloads section of the Mac GUI web site.)

--
]DF$
The New Apple II User's Guide:
https://macgui.com/newa2guide/

Statistics on disk images with ProDOS file system:

2,639 total disk images with ProDOS file system

2,197 in DOS sector order (83.25%), and 442 (16.75%) in ProDOS sector order

Disk image filename extensions:

2,161 .dsk in DOS order
144 .dsk in ProDOS order

0 .po in DOS order
257 .po in ProDOS order

34 .do in DOS order
0 .do in ProDOS order

43 files have no file extension, or have some other file extension

NOTE: 2MG, DC42, SDK, and all other disk image wrappers are NOT COUNTED in
these statistics.

Hi David,

> 2,161 .dsk in DOS order
> 144 .dsk in ProDOS order

So only 6.5% of the .DSK files are in ProDOS order.

> 257 .po in ProDOS order

So the vast majority of ProDOS order images are .PO files.

Looking at those numbers, it seems to me that Kents desire to get rid of
..DSK files in ProDOS order (with all the trouble they cause) is totally
reasonable.

Regards,
Oliver

On 11/8/22 7:26 PM, D Finnigan wrote:
> Statistics on disk images with ProDOS file system:
>
> 2,639 total disk images with ProDOS file system
>
> 2,197 in DOS sector order (83.25%), and 442 (16.75%) in ProDOS sector order
>
>
> Disk image filename extensions:
>
> 2,161 .dsk in DOS order
> 144 .dsk in ProDOS order

That's in keeping with my experience too - "just about everything" 140k
with a .dsk extension is in DOS order, regardless of the filesystem inside.

David Schmidt wrote:
> On 11/8/22 7:26 PM, D Finnigan wrote:
>> Statistics on disk images with ProDOS file system:
>>
>> 2,639 total disk images with ProDOS file system
>>
>> 2,197 in DOS sector order (83.25%), and 442 (16.75%) in ProDOS sector
>> order
>>
>>
>> Disk image filename extensions:
>>
>> 2,161 .dsk in DOS order
>> 144 .dsk in ProDOS order
>
> That's in keeping with my experience too - "just about everything" 140k
> with a .dsk extension is in DOS order, regardless of the filesystem
> inside.
>

And the other observation is that when it comes to disk images with ProDOS
file system, the .do and .po extensions are reliable indicators of sector
order.

Oliver Schmidt wrote:
> Hi David,
>
>> 2,161 .dsk in DOS order
>> 144 .dsk in ProDOS order
>
> So only 6.5% of the .DSK files are in ProDOS order.
>
>> 257 .po in ProDOS order
>
> So the vast majority of ProDOS order images are .PO files.
>
> Looking at those numbers, it seems to me that Kents desire to get rid of
> .DSK files in ProDOS order (with all the trouble they cause) is totally
> reasonable.
>

Agreed. And one might even go so far as to write a script to change the file
extension of ProDOS-ordered .dsk files to .po.

Next point of investigation for me is to see a list of these 144 .dsk images
that are in ProDOS order. What application made these images? Are
ProDOS-sector ordered .dsk images still being made by some application
today?

--
]DF$
The New Apple II User's Guide:
https://macgui.com/newa2guide/

> Next point of investigation for me is to see a list of these 144 .dsk images
> that are in ProDOS order. What application made these images? Are
> ProDOS-sector ordered .dsk images still being made by some application
> today?

Probably the most popular application would have been Dsk2File5.8. It has been around the longest and can handle both Dos order and Prodos order disks as well as create the disk image that can be stored on a hard drive, and from there transferred to a modern computer for upload.

I think Disk.Maker.8 would have got some usage when it came out. And the earlier version of Copy2Plus could make disk images as well, but unsure if it created Prodos disks in Dos order.

I believe one of the questions is still lingering unanswered is, what algorithm did you use to differentiate a Dos order disk from a Prodos order one?

Some emulators use the .po extension to indicate it is an 800kb Prodos disk and doesn't work if a 144 kb disk is used with .po instead. And .dsk was still used to indicate an 800kb AMDos disk as well.

I am Rob wrote:
>
> I believe one of the questions is still lingering unanswered is, what
> algorithm did you use to differentiate a Dos order disk from a Prodos
> order
> one?
>

I look for the catalog sectors.

Hi David,

> And one might even go so far as to write a script to change the file
> extension of ProDOS-ordered .dsk files to .po.

That was - of course - my first thought. But as I'm not the one to perform
such a change, I didn't feel entitled to bring up the topic.

Regards,
Oliver

On Wednesday, November 9, 2022 at 11:54:59 AM UTC-8, gid...@sasktel.net wrote:
> I think Disk.Maker.8 would have got some usage when it came out. And the earlier version of Copy2Plus could make disk images as well, but unsure if it created Prodos disks in Dos order.

IIRC, there's a version that could create disk images in *physical* sector order, but didn't become popular. I'm not sure what version did that; I found copies of 7.4, 8.2, and 9.0 but didn't see the feature in the menus.

On Wednesday, November 9, 2022 at 3:50:41 PM UTC-8, fadden wrote:
> > I think Disk.Maker.8 would have got some usage when it came out. And the earlier version of Copy2Plus could make disk images as well, but unsure if it created Prodos disks in Dos order.
> IIRC, there's a version that could create disk images in *physical* sector order, but didn't become popular. I'm not sure what version did that; I found copies of 7.4, 8.2, and 9.0 but didn't see the feature in the menus.

Found it. With Copy ][+ 7.x, use Copy > Disk, and try to copy a 5.25" disk onto a larger ProDOS volume. It will ask for a filename. The result is a physical-ordered image with type $F7 and auxtype $0118 (==280, indicating the number of blocks).

Attempting to do this with Copy ][+ v6.0 yields the error "both drives must be the same size", while v8.4 reports "device size mismatch". So this was a very short-lived feature.

The procedure works for 800K disks as well, creating an unadorned ProDOS block-order image.

I'm guessing that the feature's obscurity and lack of compression led to its demise. Further, I discovered that if you run out of space on the target volume, it marks the blocks as in-use but doesn't create the necessary index entries, so you have to run Mr. Fixit on the volume afterward.

The preferred extension appears to be ".img".