Hi,

I’m a hobbyist with an AlphaServer DS10 running 8.4. I have a certain PCI card that I’m writing a driver for as a project.

The card uses 16MB of linear address space in the PCI segment and no I/O space. I’ve used IOC$MAP_IO to map it into virtual memory and can access the card’s memory space from the kernel context no problem. I have read and write QIO commands hooked up as well which a user process can call.

What I would like to do, though, is make the PCI memory buffer directly accessible from a user process that’s opened a channel to the device and made a request for it. In Linux this would be mmap() with a shared buffer, in NT this would be IOCTL_MAPMEM_USER_PHYSICAL_MEMORY but I can’t find the equivalent in the VMS documentation.

Any ideas?

Katherine

On Sunday, May 9, 2021 at 1:41:08 AM UTC-7, lui...@gmail.com wrote:

> What I would like to do, though, is make the PCI memory buffer directly
> accessible from a user process that’s opened a channel to the device
> and made a request for it. In Linux this would be mmap() with a shared
> buffer, in NT this would be IOCTL_MAPMEM_USER_PHYSICAL_MEMORY
> but I can’t find the equivalent in the VMS documentation.

the term you're looking for is pfn-mapping. in this situation, i've arranged an io$_sensemode function that returns the information the user process needs to create a pfn mapping that covers the section. there are, however, many landmines in this area.

we have several custom devices in our labs and i am responsible for the device drivers for those devices. many years ago, i got tired of maintaining several device drivers that were very similar, so i created a generic pci device driver that can handle pretty much any device. it provides the information a user process needs to create a pfn-mapped section that covers the register, collects interrupts for deliver to the user program, and can pin buffers down and describe their mapping so that a user program can configure the device to access the buffer.

it's been a while since i've had to go into the driver and i don't have access to the documentation (and no idea where hp has moved them this week) and driver sources, so my memory is a bit fuzzy around the details.

back in the day, global sections and pfn-mapped sections were accessed using the same system services, $crmpsc and $mgblsc. there are sections that will be either pfn-mapped or mapped as a global section, depending on the details of the lab configuration. my strategy to deal with this was to try one type of mapping and, if that failed, retry it as the other type. eventually, the global section and pfn-mapped section services were split into independent system services. at that point, there was one combination (i.e., either trying to map a pfn-mapped section as a global section or trying to map a global section as a pfn-mapped section) that would cause the system to crash. naturally, my mapping routine was making the attempts in the order that caused the system to crash.

interrupts are handled by placing the irp on a queue, then enabling interrupts from the device. when an interrupt happens, the driver disables interrupts from the device, then completes all of the irps sitting on the queue. some versions of alpha/vms do not properly handle disabling interrupts from a device; it's as if the bus support author never entertained the possibility that someone might want to disable interrupts. in addition to doing the actual work involved in disabling interrupts, the bus support code maintains a bitmap tracking which interrupts are disabled. when an interrupt is enabled, the bus support code does the work and sets a bit in the map to note that it has enabled the interrupt. however, when an interrupt is disabled, the bus support code does the work but *does not* clear the bit in the map. consequently, the next time you enable an interrupt, the bus support code does nothing because it thinks the interrupt is already enabled. the device driver works around this on those situations by clearing the bit in the bus support code's bitmap when it disables an interrupt.

the most recent issue i had with the driver involved shared interrupts. if you attach a driver that does not handle shared interrupts to a device that is connected to a shared interrupt line, the operating system refuses to initialize the driver; it does not call *any* of the driver's initialization functions. however, there are still situations in which the operating system calls the driver's $cancel entry point, even though the driver has not been initialized. i didn't expect this, and my driver would crash when the $cancel entry was called without any of the driver's data structures having been initialized.

good luck!
--
roger ivie
roger.ivie@gmail.com

Okay, cool, so I need to find the PFN corresponding to the buffer and set up a mapping so that the process holding the channel can access that region. I'll need to figure out the full 64-bit PCI address space mapping for the DS10 then, since my documentation is conspicuously missing that and I can't seem to find the physical address with SHOW ADDR/PHY in SDA.

Fortunately it's a pretty simple PCI card from the host system's perspective - there's no interrupts or I/O space required, it's all memory-mapped I/O..

On Sunday, May 9, 2021 at 10:24:20 AM UTC-5, Roger Ivie wrote:
> On Sunday, May 9, 2021 at 1:41:08 AM UTC-7, lui...@gmail.com wrote:
>
> > What I would like to do, though, is make the PCI memory buffer directly
> > accessible from a user process that’s opened a channel to the device
> > and made a request for it. In Linux this would be mmap() with a shared
> > buffer, in NT this would be IOCTL_MAPMEM_USER_PHYSICAL_MEMORY
> > but I can’t find the equivalent in the VMS documentation.
> the term you're looking for is pfn-mapping. in this situation, i've arranged an io$_sensemode function that returns the information the user process needs to create a pfn mapping that covers the section. there are, however, many landmines in this area.
>
> we have several custom devices in our labs and i am responsible for the device drivers for those devices. many years ago, i got tired of maintaining several device drivers that were very similar, so i created a generic pci device driver that can handle pretty much any device. it provides the information a user process needs to create a pfn-mapped section that covers the register, collects interrupts for deliver to the user program, and can pin buffers down and describe their mapping so that a user program can configure the device to access the buffer.
>
> it's been a while since i've had to go into the driver and i don't have access to the documentation (and no idea where hp has moved them this week) and driver sources, so my memory is a bit fuzzy around the details.
>
> back in the day, global sections and pfn-mapped sections were accessed using the same system services, $crmpsc and $mgblsc. there are sections that will be either pfn-mapped or mapped as a global section, depending on the details of the lab configuration. my strategy to deal with this was to try one type of mapping and, if that failed, retry it as the other type. eventually, the global section and pfn-mapped section services were split into independent system services. at that point, there was one combination (i.e., either trying to map a pfn-mapped section as a global section or trying to map a global section as a pfn-mapped section) that would cause the system to crash. naturally, my mapping routine was making the attempts in the order that caused the system to crash.
>
> interrupts are handled by placing the irp on a queue, then enabling interrupts from the device. when an interrupt happens, the driver disables interrupts from the device, then completes all of the irps sitting on the queue. some versions of alpha/vms do not properly handle disabling interrupts from a device; it's as if the bus support author never entertained the possibility that someone might want to disable interrupts. in addition to doing the actual work involved in disabling interrupts, the bus support code maintains a bitmap tracking which interrupts are disabled. when an interrupt is enabled, the bus support code does the work and sets a bit in the map to note that it has enabled the interrupt. however, when an interrupt is disabled, the bus support code does the work but *does not* clear the bit in the map. consequently, the next time you enable an interrupt, the bus support code does nothing because it thinks the interrupt is already enabled. the device driver works around this on those situations by clearing the bit in the bus support code's bitmap when it disables an interrupt.
>
> the most recent issue i had with the driver involved shared interrupts. if you attach a driver that does not handle shared interrupts to a device that is connected to a shared interrupt line, the operating system refuses to initialize the driver; it does not call *any* of the driver's initialization functions. however, there are still situations in which the operating system calls the driver's $cancel entry point, even though the driver has not been initialized. i didn't expect this, and my driver would crash when the $cancel entry was called without any of the driver's data structures having been initialized.
>
> good luck!
> --
> roger ivie
> roger...@gmail.com

Specifically, the problem is that IOC$MAP_IO returns a handle value which I can't figure out how to resolve to a virtual address, so I don't know where in system memory my PCI device is mapped. Unless the "magic number" it returns happens to be a pointer to an IOHANDLE or something.

On Sunday, May 9, 2021 at 8:21:30 PM UTC-5, Luigi Thirty wrote:
> Okay, cool, so I need to find the PFN corresponding to the buffer and set up a mapping so that the process holding the channel can access that region. I'll need to figure out the full 64-bit PCI address space mapping for the DS10 then, since my documentation is conspicuously missing that and I can't seem to find the physical address with SHOW ADDR/PHY in SDA.
>
> Fortunately it's a pretty simple PCI card from the host system's perspective - there's no interrupts or I/O space required, it's all memory-mapped I/O.
> On Sunday, May 9, 2021 at 10:24:20 AM UTC-5, Roger Ivie wrote:
> > On Sunday, May 9, 2021 at 1:41:08 AM UTC-7, lui...@gmail.com wrote:
> >
> > > What I would like to do, though, is make the PCI memory buffer directly
> > > accessible from a user process that’s opened a channel to the device
> > > and made a request for it. In Linux this would be mmap() with a shared
> > > buffer, in NT this would be IOCTL_MAPMEM_USER_PHYSICAL_MEMORY
> > > but I can’t find the equivalent in the VMS documentation.
> > the term you're looking for is pfn-mapping. in this situation, i've arranged an io$_sensemode function that returns the information the user process needs to create a pfn mapping that covers the section. there are, however, many landmines in this area.
> >
> > we have several custom devices in our labs and i am responsible for the device drivers for those devices. many years ago, i got tired of maintaining several device drivers that were very similar, so i created a generic pci device driver that can handle pretty much any device. it provides the information a user process needs to create a pfn-mapped section that covers the register, collects interrupts for deliver to the user program, and can pin buffers down and describe their mapping so that a user program can configure the device to access the buffer.
> >
> > it's been a while since i've had to go into the driver and i don't have access to the documentation (and no idea where hp has moved them this week) and driver sources, so my memory is a bit fuzzy around the details.
> >
> > back in the day, global sections and pfn-mapped sections were accessed using the same system services, $crmpsc and $mgblsc. there are sections that will be either pfn-mapped or mapped as a global section, depending on the details of the lab configuration. my strategy to deal with this was to try one type of mapping and, if that failed, retry it as the other type. eventually, the global section and pfn-mapped section services were split into independent system services. at that point, there was one combination (i.e., either trying to map a pfn-mapped section as a global section or trying to map a global section as a pfn-mapped section) that would cause the system to crash. naturally, my mapping routine was making the attempts in the order that caused the system to crash.
> >
> > interrupts are handled by placing the irp on a queue, then enabling interrupts from the device. when an interrupt happens, the driver disables interrupts from the device, then completes all of the irps sitting on the queue. some versions of alpha/vms do not properly handle disabling interrupts from a device; it's as if the bus support author never entertained the possibility that someone might want to disable interrupts. in addition to doing the actual work involved in disabling interrupts, the bus support code maintains a bitmap tracking which interrupts are disabled. when an interrupt is enabled, the bus support code does the work and sets a bit in the map to note that it has enabled the interrupt. however, when an interrupt is disabled, the bus support code does the work but *does not* clear the bit in the map. consequently, the next time you enable an interrupt, the bus support code does nothing because it thinks the interrupt is already enabled. the device driver works around this on those situations by clearing the bit in the bus support code's bitmap when it disables an interrupt.
> >
> > the most recent issue i had with the driver involved shared interrupts. if you attach a driver that does not handle shared interrupts to a device that is connected to a shared interrupt line, the operating system refuses to initialize the driver; it does not call *any* of the driver's initialization functions. however, there are still situations in which the operating system calls the driver's $cancel entry point, even though the driver has not been initialized. i didn't expect this, and my driver would crash when the $cancel entry was called without any of the driver's data structures having been initialized.
> >
> > good luck!
> > --
> > roger ivie
> > roger...@gmail.com

On Sunday, May 9, 2021 at 6:44:32 PM UTC-7, lui...@gmail.com wrote:
> Specifically, the problem is that IOC$MAP_IO returns a handle value which I can't figure
> out how to resolve to a virtual address, so I don't know where in system memory my PCI
> device is mapped. Unless the "magic number" it returns happens to be a pointer to an
> IOHANDLE or something.

the iohandle holds the *physical* address at which your device is mapped. you use that to create the page frame number you need to hand to the appropriate flavor of $crmpsc to create a pfn-mapped section. $crmpsc gives you the virtual address.
--
roger ivie
roger.ivie@gmail.com

Okay, cool. I'm close now, but I'm running into an issue passing the return values to sys$crmpsc_pfn64.

VOID_PQ mapped_addr_base;
uint64 mapped_addr_length;
r0_status = sys$crmpsc_pfn_64(
VA$C_P0,
buf/8192,
(1048576*16) / 8192,
PSL$C_USER,
SEC$M_EXPREG|SEC$M_WRT,
&mapped_addr_base,
&mapped_addr_length
);

This fails with an SS$_ACCVIO, meaning VMS can't write to mapped_addr_base or mapped_addr_length according to the system service docs. I'm not sure why that's the case, since I'm just passing it these two variables by reference. buf is the 64-bit physical address to the card. I saw someone else have this problem in another older post but the solution didn't end up on here.

On Tuesday, May 11, 2021 at 8:00:55 AM UTC-5, Roger Ivie wrote:
> On Sunday, May 9, 2021 at 6:44:32 PM UTC-7, lui...@gmail.com wrote:
> > Specifically, the problem is that IOC$MAP_IO returns a handle value which I can't figure
> > out how to resolve to a virtual address, so I don't know where in system memory my PCI
> > device is mapped. Unless the "magic number" it returns happens to be a pointer to an
> > IOHANDLE or something.
> the iohandle holds the *physical* address at which your device is mapped. you use that to create the page frame number you need to hand to the appropriate flavor of $crmpsc to create a pfn-mapped section. $crmpsc gives you the virtual address.
> --
> roger ivie
> roger...@gmail.com

On Wednesday, May 12, 2021 at 11:26:32 AM UTC-4, lui...@gmail.com wrote:
> Okay, cool. I'm close now, but I'm running into an issue passing the return values to sys$crmpsc_pfn64.
>
> VOID_PQ mapped_addr_base;
> uint64 mapped_addr_length;
> r0_status = sys$crmpsc_pfn_64(
> VA$C_P0,
> buf/8192,
> (1048576*16) / 8192,
> PSL$C_USER,
> SEC$M_EXPREG|SEC$M_WRT,
> &mapped_addr_base,
> &mapped_addr_length
> );
>
> This fails with an SS$_ACCVIO, meaning VMS can't write to mapped_addr_base or mapped_addr_length according to the system service docs.

The docs also say the first argument is passed by reference, so it should be &VA$C_P0. I've played with
sys$crmpsc_file_64(), and the first argument for it is the same.

On Wednesday, May 12, 2021 at 9:07:42 AM UTC-7, osuv...@gmail.com wrote:
> > VOID_PQ mapped_addr_base;
> > uint64 mapped_addr_length;
> > r0_status = sys$crmpsc_pfn_64(
> > VA$C_P0,
> > buf/8192,
> > (1048576*16) / 8192,
> > PSL$C_USER,
> > SEC$M_EXPREG|SEC$M_WRT,
> > &mapped_addr_base,
> > &mapped_addr_length
> > );
> >

peeked at my code while i was at work today (can't read news there, don't have sources here at home). the only things that stand out as obviously different between my codes and yours are 1) yes, va$c_p0 needs to be passed by reference. 2) i included sec$m_pfnmap in the flags; don't recall whether that's necessary, but wouldn't surprise me. 3) i included the
final parameter (which you've omitted here; it's supposed to be optional when sec$m_expreg is set); i passed a zero in that parameter.
--
roger ivie
roger.ivie@gmail.com

On Wednesday, May 12, 2021 at 8:27:29 PM UTC-5, Roger Ivie wrote:
> On Wednesday, May 12, 2021 at 9:07:42 AM UTC-7, osuv...@gmail.com wrote:
> > > VOID_PQ mapped_addr_base;
> > > uint64 mapped_addr_length;
> > > r0_status = sys$crmpsc_pfn_64(
> > > VA$C_P0,
> > > buf/8192,
> > > (1048576*16) / 8192,
> > > PSL$C_USER,
> > > SEC$M_EXPREG|SEC$M_WRT,
> > > &mapped_addr_base,
> > > &mapped_addr_length
> > > );
> > >
> peeked at my code while i was at work today (can't read news there, don't have sources here at home). the only things that stand out as obviously different between my codes and yours are 1) yes, va$c_p0 needs to be passed by reference. 2) i included sec$m_pfnmap in the flags; don't recall whether that's necessary, but wouldn't surprise me. 3) i included the
> final parameter (which you've omitted here; it's supposed to be optional when sec$m_expreg is set); i passed a zero in that parameter.
> --
> roger ivie
> roger...@gmail.com

Yeah, passing the first parameter by reference was the problem. I don't need the last parameter (I don't care where in my process' virtual address space it gets mapped as long as it's there). I got the card to map and work from a user process. Thanks!