Forgive me if this is an inappropriate place for this architectural question but I'm not sure where it best fits in the StackExchange family.
I have been tasked with developing an online/live auction system that operates at multiple locations simultaneously as well as online. These auctions are high paced, selling thousands of items in 3-4 hours with roughly 3 "auction blocks" at each location. The point is, there are several people connecting to this at once. Normally, I'd begin designing this "in the cloud" but I've been told it MUST operate even if internet access is lost (and this internet access is of low quality to begin with).
Initially I thought it might be best to simply write two pieces of software, one running locally and one running in the cloud with an API bridge between them. The unfortunate part of this is the database would have to live locally and be served from the location and I don't particularly care for that especially considering the bandwidth limitations. Additionally, the scope requires that each location should be able to access another locations auction items.
Lately I've thought a replication method may work where the database is replicated across all locations and the local "client" operates with the cloud until latency hits a predefined threshold or internet completely dies and then switches to the local copy of the database (it's ok if the online auction gets shut off in a bad connections scenario – it would be the first casualty).
Does anyone have any best practices or experience in dealing with large amounts of data, changing rapidly with tons of connections, that must be available both on a local network and online (in the cloud)? What works best in these scenarios?
Would MySQL or MongoDB using replication, replicate fast enough to be used in this scenario? Perhaps with the primary database in the cloud and several "fail-over" (one at each location).