I'm currently working on connecting a few large google sheets into postgreSQL. My current system is that the sheet gets extracted & loaded to postgreSQL, I'd then do the transformations afterwards.
This is a real-time job that gets ran every couple minutes or so. The default schema of the sheet is quite messy and I'd like to introduce relational structure to it; I'd also want to use Python for data cleaning as it comes in. Any suggestions for an efficient approach to tackle this?
