I'll be the first to admit the documentation is not very clear on this. They state that you should be offloading your backups to the secondaries, but while most statements are made in a general sense, they really mean, specifically, log backups (and copy_only backups, if you have some need for those).
You will need to occasionally run full backups on the primary in order to keep your TLogs in check. The copy_only restriction is about full backups, not log backups, and only applies to the secondary. You should not be using the copy_only option on the primary (and you may not want to be running it on the secondaries either, if you don't have a specific need for them).
You will not be able to use shrinkfile against your current TLogs because they are full of activity that has technically not been backed up. Once you take a full (non-copy_only) backup on the primary, then let one log backup run, you should be able to shrink the log file manually. Currently your log backups are working because the database is set to full, but they are continuing to grow because (presumably) you've never run a proper full backup on the primary.
This should be a one-time operation, and you shouldn't shrink them too small; you need to set your primary to be backing up regularly, and you need to accommodate for the largest set of activity that will occur between full or log backups. I won't go on and on about how shrinking files only so they can grow again is a futile exercise and guarantees poor performance, but I could. :-)
