We have a requirement to move about 300GB/day of telemetry data from an on-premise collector to a Palantir Foundry Cloud system. Updates should be incremental and occur every 10 minutes. Additional requirements are: 1. Minimize latency from collection of telemetry to arrival in Palantir Data Lake (but not real time). 2. Minimize bespoke software development.

The hardware engineers who produce this data are definitely NOT Big Data Engineers or programmers, so they are proposing sending batches of CSV files every 10 minutes.

Our thinking is that since Palantir Foundry supports JDBC connections, it might be smarter to stage the data into a relation DB (e.g. PostGreSQL) and then use the Palantir Foundry JDBC connector to poll for incremental data refresh every 10 minutes.

Does anyone see anything stupid here, that mitigates against doing a pilot of this? Is there a completely different approach we should look at (Not considering the ASPERA FASB way right now since that would take a lot of engineering).

What we don't like about the CSV approach:

  1. Wasted time to convert binary telemetry to CSV
  2. CSV makes the data volume grow (ASCII) and adds to network traffic
  3. CSV has to be unpacked back into Parquet Data files
  4. JDBC connector supports CDC, for fine-grain incremental refresh. CSV does not


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.