I think your best approach would be to start with just tracking the counters in sys.dm_db_index_usage_stats. Just store the results off to a table (and have that run on a schedule) for as long as you feel necessary. Then look at the counters to see if there is significant movement on them. There will always be a little, especially if you have maintenance going.
If you think they are done, then rename the table and see if anyone screams. You can quickly rename it back if you need to.
Finally, if you want an extended events session to track, the below might work for you. As you are aware, it has an impact, but it's not going to be as bad as you might think, especially for a table that you think is not being used. Still, when you first enable it, test it to make sure it catches what you want, as well as that it's impact isn't hindering your users. However, it's the only sure way I know to catch this type of thing.
The below is what I typically use when I want to catch usage of specific stored procedures or queries that contain certain text strings. I send it to a ring buffer because I typically only run it until I've got what I'm looking for. But you can alter the target to whatever you want. It creates the XE turned off, so you can go back in and edit it before starting it up.
CREATE EVENT SESSION [SpecificSQLWatch] ON SERVER
ADD EVENT sqlserver.rpc_completed(
WHERE ([sqlserver].[like_i_sql_unicode_string]([sqlserver].[sql_text],N'%<my string here>%')))
ADD TARGET package0.ring_buffer
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=OFF)