You need to add a modem manually. After adding in the serial port to the virtual machine, you then add the modem into windows just as if you attached an old-fashioned serial port modem, the kind that's not plug-and-play.

For instance on my host machine, I have 2 serial ports, COM1 & COM2, and one winmodem set as COM3. I add a serial port to the virual machine, COM1. I configure the virual COM1 to point to the host's COM3. Then I add a modem to the virual machine. Since it's not a plug-and-play modem, it's detected as "unknown modem." I then manually change that to Standard 33600. If you know your modem is Flex, v90 or x2, then you can choose one of those. Otherwise Standard 33600 will always work for any winmodem. Done! The virtual machine now has a modem on COM1.

BTW, VMWare "takes over" the serial port, and thus the modem, while it's connected. So if you want to use the modem from the host, either disconenct the serial port the from VM menu or shut down the virtual machine.

or

Use an external modem that plugs into the serial port on your host.