Modem tranfser problem

Trying to transfer database files via modem. I am using custom software for the transfer which works fine on all machines apart from 1.
Machine in question is a laptop running xp.
When I try to connect I get an error message headed “16 BIT MS DOs SUBSYSTEM”
The Error is: “The NTVDM CPU has encountered an illegal instruction. CS:93c9 IP:dee5 OP:ff ff ff ff Choose close to terminate the application.
There is a close and an ignore button. If I click on ignore I get the same message back again.If I click on close the application continues and the modem connects and handshakes but no files get transferred.

Can anyone tell me what this error message means please.