So the way I did this came from JNK's comments, so all props to him.
In order to use time formatting correctly, I return the following
(SELECT DATEDIFF(n, CAST('' AS Time), dbo.FactWorkDaySummary.StartTime) / 1440.0) AS StartTime
in my view.
In my cube I have a calculated measure to calculate the average appropriately, and set the Format_String property to "short time". This works, because dividing by 1440.0 converts the time into a float representing the fraction of the day that those minutes represent, which is the representation of the datetime. This can also, handily, be averaged easily.
Just make sure you don't display the date, because it's `30/12/1899', which isn't really applicable.