You could leave the agent active in both servers and you could add a first step to them in this way:
check the database status (active or restoring)
if the DB is not active the job stop itself
if the DB is active the job can contine on next steps
You can keep the Agent enabled but disable the individual jobs on ServerB and create a dedicated job on ServerB that continuously runs (or schedule it to run every X amount of time) and checks if the current server is the Primary. If it is, then enable the other jobs with sp_update_job.
You can also check which jobs are enabled with msdb.dbo.sysjobs so your ...
It seems there are many who has faced this issue in the past and they all have implemented one or the other work-around to get rid of it. Not documented at Microsoft site or presented by any known SME(Subject Matter Expert).
First of all, I would say database mirroring is deprecated and will not be supported by Microsoft in near future, so you should plan to ...