My first encounter with "using MySQL as a queue" ended in disaster. The team was very wedded to tossing things into a table, then pulling it out to work on. They were limited to how many things the could achieve per hour.
After studying not only the queuing mechanism but the enqueue and dequeue API and the worker threads, I decided on:
"Don't queue it, just do it".
I estimated (for their case) that they could increase the throughput 10-fold by removing the queue.
Here are some random lessons:
- If the task to perform is fast enough, you may be spending more time on enqueuing/dequeuing than the task.
- If the tasks arrive in a steady stream, the "buffering" that queuing gives you is unnecessary. On the other hand, if the tasks are "bursty", queuing may be beneficial.
- Replication (HA was a requirement in the case) significantly complicates the queuing code.
AUTO_INCREMENT for queuing with replication adds a complication because the
ids don't always arrive in the Slave in order! This happens randomly (but not very frequently) with InnoDB.
- Deletion of queue items leads to fragmentation. This was especially bad with MyISAM.
- Note how switching Engines solves one problem but creates another?
- UUIDs are terrible for performance if the table gets bigger than can be cached.
- Transactional integrity gets messy -- the item in the queue is mostly independent of the task it represents. Issues to resolve: requeuing a task that fails; crashing in the middle of a task; etc.
- Sure, the queue will have very few rows at any normal moment. But there will come a time when something hiccups and there is a million tasks queued up. The system will croak for any of several reasons that you failed to plan for. You have a crisis on your hands and the looming job of analyzing and planning for such eventualities.
- Based on my previous comment, do you really want to leave off any index? No. The dequeuing mechanism will get slower and slower, further exasperating the crisis.
- If a queued task is assigned to a worker, but that worker crashes, then the item needs to stick around somewhere (in the queue or some other place) and some separate task needs to eventually discover an unfinished task after some timeout. More messy code to write.
An alternative (that may or may not apply to your situation): Leave information around and have a continuously running job (not a cron) that looks for the info and acts on it. This can be handy if the items arrive rapidly and they can be processed in batches.