What OS-level APIs would something communicating over serial be using? Sure, Windows abstracts the hardware, but that's been true since well before Windows XP, and modern versions of Windows still expose virtual COM ports for serial adapters connected through USB or PCIE, just as XP did.
In fact, even DOS abstracts away serial communications: your hardware would be set to a specific I/O port and IRQ, but DOS exposes those as I/O devices, so you would just read and write to COM1, COM2, etc.
My guess is that there's some other issue running that old Windows program on modern versions of Windows -- maybe some components are 16-bit, and since current 64-bit Windows doesn't include the VDM, they can't run. Or maybe it's an issue with current versions of certain DLLs.
In fact, even DOS abstracts away serial communications: your hardware would be set to a specific I/O port and IRQ, but DOS exposes those as I/O devices, so you would just read and write to COM1, COM2, etc.
My guess is that there's some other issue running that old Windows program on modern versions of Windows -- maybe some components are 16-bit, and since current 64-bit Windows doesn't include the VDM, they can't run. Or maybe it's an issue with current versions of certain DLLs.