MongoDB replication is to provide automatic fail over during disaster recovery, minimize downtime during maintenance, to use the secondary for analytical use cases and so on. In your scenario, you don't want to avail these benefits during off-peak hours and as far as I know there is no perfect workaround to do these tasks without downtime. 

> Simply, shutting down the secondary + arbiter makes Mongo unable to
> elect a new primary and the app fails

Yes, election process will not happen in a replica set with single node. It needs at least two mongod nodes which might not be present in your case if you remove secondary and an arbiter.

> If the best/only solution is to remove the secondary and arbiter from
> the replica set

You can remove the secondary and arbiter from the replica set by following the steps in this link [mongo-replica-set-to-single-server][1]. 

> what are the consequences of doing when adding them
> back on the next peak hours cycle?

While adding back the replica sets, you will have to restart mongod instances, issue rs.reconfig() method and it can force the current primary to step down which causes an election. During this window time, your app will not read/write anything from/into database which can last up to 10-30 seconds (for re-configuring replicas), another 20-30 seconds (for restarting mongod instances) based on network bandwidth and replica set configuration. Hence, it is not recommended during peak hours.

  [1]: http://serverfault.com/questions/389876/mongo-replica-set-to-single-server