How do you approach the problem of resolving access permissions when matching tens of thousands of users with tens of millions of documents containing unstructured data? How do you even build and represent a filtered folder structure for navigation? In particular, I'm perplexed as to how you would provide functional ways to search only the content currently accessible by any given user.
Would you filter the result set from a full text query through some sort of ACL table? Before or after other filters or merging with other result sets? Build and maintain in memory bitmap- or hash-tables for active users? Stored procedures? Inverted indexing? De-normalization?
Is there an established way of doing this or is everyone brewing their own solutions?
Seeing how this must be core functionality in any ECM or just about any type of content management system really, I'm sure there must be numerous solutions and best practices available. What might be good sources of information?
I'm not sure if any large scale services like Evernote even include shared documents in full text searches, but if they do, I sure would like to know how.
If it comes down to a product specific level, I'm biased towards Oracle Database and Lucene XML Index.