125 reputation
3
bio website williamtpayne.blogspot.com
location New York, United States
age
visits member for 2 years, 7 months
seen Jul 4 '12 at 16:08

Data Engineer & Quantitative Software Developer.

I work with academics & domain experts to bring quantitative algorithms into production.

As a result, I have developed an interest in how software engineering best practices can be applied to multidisciplinary teams, as well as to the development of software whose behaviour is specified in statistical terms. (I.e. software that is both hard-to-specify & hard-to-verify).


Jul
4
comment How can I automate sharding in the presence of continuous reads
In the end, I went with option (1), as I was able to get it working very quickly. I am also intending to implement 2(b) also, to reduce the amount of tricksy stuff happening on the change of the hour.
Jul
4
accepted How can I automate sharding in the presence of continuous reads
Jul
2
comment How can I automate sharding in the presence of continuous reads
It takes only a few seconds for the new empty DB to be ready for use, but I do not know exactly how many seconds.
Jun
29
comment Does mongodb automatically move small non-partitoned databases to spread read load?
Thank you very much for your answer. I am now sharding the databases when they are created, so that the load balancer can spread them around as required. I am expecting the databases to get quite a bit bigger as the load on our system ramps up, so I think that this is the right approach.
Jun
28
asked How can I automate sharding in the presence of continuous reads
Jun
28
awarded  Scholar
Jun
28
accepted Does mongodb automatically move small non-partitoned databases to spread read load?
Jun
28
awarded  Student
Jun
27
awarded  Supporter
Jun
27
comment Does mongodb automatically move small non-partitoned databases to spread read load?
I could do that, but I was hoping I would not have to. We already have the 10 machine cluster available, and too limited time to reconfigure it dramatically. (It is also being used by another set of processes, which complicates things). The databases are automatically generated - one per hour, so I could switch to use 1 per day, which would make them big enough to shard (at the expense of some write performance). I would like to be able to explicitly specify which shard they should go on, but I suspect that movePrimary will not work for this.
Jun
27
asked Does mongodb automatically move small non-partitoned databases to spread read load?