Installing Sybase within a virtual environment purely for development purposes is possible and I have used it that way for years with Windows, Linux and Solaris x86. I'm not sure about Vagrant, but with Virtual Box it works.
You must ensure your virtualised host has sufficient memory to run the database and have made the relevant additional settings.
You'll need to set the maximum amount of shared memory to a suitable value and there is probably a dependency on libaio1.
For example, add the following to /etc/sysctl.conf. If 1GB is too large you can use a smaller value. You'll need to consult the Sybase documentation for the version you're installing for a minimum value.
# Set the kernel shared memory to 1GB