I'm working on a cloud-agnostic service which must allow the ingestion of a lot of media files in a very limited timeframe.
- 3-4 hours total of workload: the service will be active during several distinct live events, with no traffic before and virtually no traffic after (I plan to tear-down the whole structure after each event)
- I have all the time I want for a warm up before the event
- it will be used to ingest a variable quantity of files, approx between 1k and 100k files
- each file will be approx between 10MB and 200MB
- the ingestion will be in waves, with up to approx 1k-5k files to be ingested in 5 minutes, then virtually no traffic for another 5-10 minutes
As for now I'm using:
- Kubernetes (cloud agnostic)
- MongoDB with GridFS sharded for scalability
The issue is that adding a new shard to MongoDB triggers the balancer which takes a lot of time, and during rebalancing the ingestion performance are even worse.
I wish to be able to:
- start with a minimal setup with 2 shards
- observe disk and CPU usage during each 5 minutes wave of peak load
- decide whether to scale up (no scale down required)
- add some shards (probably double them up)
- having the rebalancing completed in 5 minutes before a new wave arrives.
My current setup scales good if I leave enough time for the balancer to complete balancing before the new wave arrives, which most of times is not the case.
Somebody can help me achieving the desired behavior?
I'll eventually consider alternatives to MongoDB if they still are cloud-agnostic.