A port number is not necessary to use DBMS.
Individual programs could write directly to the DBMS's files. But the rules of a DBMS have become very complex, and an immense amount of logic is needed to respect those rules and to respect the actions of other programs using the DB. I've known that situation with Oracle, you could compile your programs to access the DB directly.
To avoid having even the most simple program becoming a huge blob, the DB authors came up with the DB server. A single program had access to the DB, others wanting to use the DB, had to "talk" with that program, which executed their requests and sent back the results. That single program has become a complete library, you can't count the processes of a single Oracle DB on your fingers and toes.
Now, "talking" between programs is inter-process communication. One possibility is a "socket", no port needed, on the unixes it is presented as a file to which the program can write a request, the server reads it, executes, and writes the result, which the program can read. Another possibility is TCP/IP, for which you need to know the port number. The program sends its requests to that port, on which the DB server is listening, etc. TCP is much more secure than UDP, slower, but no packets can get lost. You can't afford that with a DB. Another big advantage: programs running on other systems can use the DB.