I have a Windows server running MongoDB. My database has big indexes and all RAM is usually busy. As I know, it can be a cause of Windows swapping and performance issues. Does it make sense to limit Mongo service RAM (e.g. use only 80%)?
No. If it starts to swap, it means that MongoDB needs more RAM than the system has available. Limiting the RAM used would make the situation worse.
MongoDB uses only up to a certain limit (95% of physical RAM, iirc), which might lead to the wrong assumption that there is free RAM available and it must be a problem caused by MongoDB. Quite the contrary: MongoDB keeps the system usable for administrative tasks.
Solution: increase the physical RAM (scale up) or start sharding (scale out).