It seems to be a weird error, I have tried everything from giving Control, Alter, and View Definition permission on the symmetric key but nothing seems to work out.
Tried everything from this article. https://learn.microsoft.com/en-us/sql/t-sql/statements/grant-symmetric-key-permissions-transact-sql?view=sql-server-ver16
It works perfectly fine when I grant sysadmin permission to the user, however not working with db_owner and db_datareader roles. A user with sysadmin privilege can access it without any issues.
Have tried running the following alter command on the DB as well but still no go.
alter master key add encryption by service master key