We have an application that queries a SQL database periodically throughout the day. There are periods of zero or only light activity, interspersed with individual requests for relatively large amounts of data. When those requests come in, the primary objective is to deliver the data quickly, and the secondary objective is to do that cost-effectively. Due to the nature of the application, it is fairly unlikely that the data/indexes will have been cached in RAM from the previous query (different users, working on different portions of the data).
For a system that experiences relatively steady usage, I have heard the rule of thumb to observe the disk queue length and keep that number relatively small. This will specifically run in AWS, where I have seen the rule of thumb that a disk queue length of 1 per 100 IOPS is reasonable.
How can I estimate the IO requirements for such a system? Is the disk queue length a reliable indicator when dealing with individual, bursty queries? Are there other metrics that I should consider?