Timeline for Scalability of Postgres for table with large number of indexed columns
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 28 at 3:34 | comment | added | J.D. | @LukeHutchison No doubt. I will preface by saying, I find FAANG devs are pretty apt for reinventing the wheel to try to be revolutionary when solving a problem. It's not uncommon I run into strong software devs who hardly understand how traditional database systems work or how to properly architect with one to build a performant system (not at their fault, rather due to a lack of training in academia, but that's a whole other tangent). "a knob that applications can turn to tune the consistency vs. availability tradeoff" - Generally this is transaction isolation levels as my answer discusses. | |
| Jan 28 at 3:18 | comment | added | Luke Hutchison | @J.D. Fair enough, I shouldn't have used Facebook as my main example, since I didn't realize they were still using MySOL -- Amazon and Google would be better examples, since both have implemented custom "planet-scale databases". I worked at Google for many years, and briefly during that time did some DB infrastructure work. All the major custom database frameworks there (there are several) have a knob that applications can turn to tune the consistency vs. availability tradeoff (e.g. selecting for eventual consistency or strong consistency). | |
| Jan 27 at 15:05 | comment | added | J.D. | @LukeHutchison "For apps at say Facebook scale, you can't afford to enforce strong consistency worldwide for every view of the data" - FWIW, it depends on the database. In their RDBMS (which I believe is MySQL still), they are doing such that. They have other database systems they use for other use cases, but a RDBMS is their database for a lot of their main use cases, and the data is consistent. The point is, when you architect your system properly, most times, you don't have to worry about trading consistency away to maximize performance. | |
| Jan 27 at 5:24 | comment | added | Luke Hutchison | Thanks for all this detail. What I mean by being OK with lack of consistency is that it's OK for a result to briefly be missed from a result set because an index is still being updated. Every application has some amount of consistency they are willing to live without, whether that amount is zero or greater than zero. For apps at say Facebook scale, you can't afford to enforce strong consistency worldwide for every view of the data. For my application, scalability is more important than consistency. | |
| Jan 26 at 17:35 | history | answered | jjanes | CC BY-SA 4.0 |