In order of what to try:
1. Enable TCP/IP
Go to: SQL Server Configuration Manager on your SQL Server VM, under SQL Server Network Configuration choose your instance (should be Protocols for SQL2008), enable your preferred protocol (I'd guess TCP/IP) and give it another test.
As to why it's usually disabled:
From MSDN article:
For security reasons, networking protocols are disabled by default in
SQL Server Express. Individual protocols can be enabled manually or
during installation of SQL Server Express.
2. Start SQL Server Browser
Go to: SQL Server Configuration Manager on your SQL Server VM, under SQL Server Services make sure SQL Server Browser is running.
from MSDN article:
SQL Server Browser is used by some networking protocols to associate named instances with TCP/IP port numbers. However, if SQL Server Express is installed on the same computer as a pre-SP3 version of MSDE or SQL Server 2000, there might be conflicts. Also, the SQL Server Browser service does not start automatically after installation
3. Check your instance name
If you install SQL 2008 Express (or later versions), the Server instance will be named as SQLExpress, so to connect you will need to specify ServerName and InstanceName, ie:
[VMServer\SQLExpress]
4. Check Connection on SQL Server
In SSMS on Server right click Server, go to Connections and make sure that Allow remote connections to this server is checked
5. Basic network checks
- Create a folder on your server, share it and make sure you can see it from the other machine.
- Try to telnet to it
- netstat -ab
Security
You might run into authentication issues, but that should be a straight forward matter of adding your user onto the SQL Server instance.
Note: If none of the above work for you, and you find the solution, please add your solution to the list above (trying to help future users as well)