Our current setup:
Cluster of ~500 ec2 c1.xLarge worker nodes with 8 threads on each running scientific simulations
Results temporarily stored at runtime on each worker node's local MongoDB, using ephemeral data storage.
Once a worker node finishes it's simulation jobs, it will stream the results to a master node (m2.2xlarge) which has a MongoDB instance using EBS Raid-0 for storage. Writing on the fly directly to the master DB was causing socket errors.
Data across the cluster is not necessarily independent, so the incoming data is coming in via upserts and $push, not inserts.
The worker node is terminated once it's data has been received by the master.
A typical run will generate ~100GB of data on the master, ~200,000,000 records.
Our problem is that we encounter severe write latency, causing our worker nodes to sit idly waiting on the master node write locks and page faults. We have been looking into setting up a sharded environment, and are looking for suggestions on how best to configure this for our use case.
Increasing our available memory via sharding seems like a logical step, but leaving a cluster of ec2 instances up can be significantly more costly than a single machine. It would be nice if we could use a sharded cluster at runtime, then replicate the results to a persistent single instance for durability.
EDIT: For a given run, our end goal is to have all results in a single collection on the monolithic mongo instance.