I had a table to store user votes for other users. So user1 can vote for user2 (vote up or vote down). One user should be able to vote for many users, but not repeatedly to the same user; user1 can only vote once for user2. I do not think my table is properly structured to reflect this. Currently each time a user gets a vote up/down an ew row is created for this with either a 1 in the vote_up/vote_down column. What changes should I make to properly structure this table?

vote_id              int
voter_id         int
voted_for_id         int
vote_down        int  //stores the up votes
vote_up              int  //stores the down votes

To clarify users 1-19 can all vote for user 20 once each and user 20 can vote for 1-19 once each. At the end of the day I want to get a tally of how many votes each user has for both up votes and down votes each.