I agree with Jose that you are overcomplicating the problem. I think you can write a simple check constrainttrigger on the column you want the database to check its integrity. I mean something like
select ((select * from users where user_id = actor_id) or (select * from "groups" where group_id = actor_id))