When Linux uses swap for virtual memory even once, it generally leaves it stored there, even if it also has the same memory in physical RAM. If the content of the RAM does not change, and the process needs to use swap for that memory later, at least it doesn't have to write to disk again.
This doesn't mean that the process is necessarily reading from swap and incurring the performance hit. It might be reading that data from RAM with high performance. But it still shows up as swap space "in use" in the report from free.
I'd use vmstat or top to monitor if there is swap reads or writes going on currently. That's what causes the performance hit.
If you are concerned, you might consider disabling swap space on your database server. That's what I've done in the past. Swapping is so bad for database performance that I'd rather just prevent swapping by having no swap partition.
This does create a risk that if any process allocates memory in excess of physical RAM, it triggers an out-of-memory (OOM) reaction, and the kernel will kill some process to free RAM. This often ends up being the database server process, because its big buffer pool allocation is usually the biggest user of RAM on the server (assuming the server is a dedicated database server). But if you monitor memory consumption and tune your allocation so there's some breathing room, this should be rare and the risk is minimal.
