Performance issues can stem from a wide variety of issues - the size of the table is only one factor. Things such as indexes and the design of the query are just as, if not more important than, how well something performs. For example, take a look at the advice on this Stack Exchange question:
Slow Performance Inserting Few Rows Into Huge Table
Once your performance starts to degrade, you will start to look at the reason why, and then fix that problem. Is your table actually "too large?" Then maybe partitioning it will solve the issue. Maybe the query is bringing over unused columns and you don't need them all - rewrite the query and all may be better. In your case, a covering index on the users that includes the product will probably keep it from ever degrading any noticeable amount. There are a host of reasons why performance will be degraded over time.
Don't plan for any certain size - instead keep good practice in database design and query writing. That way when something comes up, it will be much easier to find that issue and fix it.