Questions tagged [apache-spark]

Use this tag for questions related to the distributed computing framework published by Apache.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Writing large dataset from spark dataframe

We have a azure databricks job that retrieves some large dataset with pyspark. The dataframe has about 11 billion rows. We are currently writing this out to a postgresql DB (also in azure). Currently ...
Kyle Chamberlin's user avatar
0 votes
1 answer
68 views

spark-cassandra-connector read throughput unpredictable

A user reports that the range query throughput is far higher than expected when setting spark.cassandra.input.readsPerSec in the spark-cassandra-connector. Job dependencies. The Java driver version is ...
Paul's user avatar
  • 381
0 votes
0 answers
20 views

Creating a real time data pipeline with transactional DB

I am working in fintech industry, they recently needed a dash-boarding solution for executive , finance and other departments while they also want AI solution to keep track of valid disputes and fraud ...
Muhammad Affan's user avatar
0 votes
0 answers
45 views

CDC on postgreSQL to Redshift

I am trying to implement CDC on PostgreSQL to Redshift and looking for some approaches. I had a look at Debezium but it looks like it's not ideal for my requirement. I want to read all delta changes ...
Bab's user avatar
  • 1
1 vote
0 answers
85 views

Why does MySQL select offset not work in pyspark?

select distinct day from t1 order by 1 desc limit 1,1 should return the second latest date in t1, but this error is thrown: ParseException: mismatched input ',' expecting {, ';'}
uhmosdhsjxpbcrstis's user avatar
1 vote
1 answer
603 views

Optimal join for joining facts with scd-type-2 dimension for aggregation/reporting

I have a fact table and an scd-type-2 dimension table. I want to produce sales report by region and year. I have working solution with a query that joins them for reporting purposes. When I run the ...
Kashyap's user avatar
  • 149
1 vote
2 answers
137 views

When will the Spark Cassandra connector with support for Spark 3.3 be released?

Master branch has a merged pull request supporting Spark 3.3 how long before this gets built and published on maven repository?
Rubber Duck's user avatar
2 votes
1 answer
52 views

Can we use Cassandra in place of Hadoop with Spark?

Considering we have a backend written in NodeJS and uses MySQL and Cassandra as it's databases, if we want to add Spark to the system to do some data analyzing stuff like recommendation, can we do it ...
user20551429's user avatar
1 vote
2 answers
588 views

Cassandra-4-Update: Multiple Schema Versions

after upgrading our first node, it has a different schema version (according to node tool describe cluster). This caused Spark Jobs to hang, because of reoccurring "schema agreement not reached&...
Sven's user avatar
  • 11
2 votes
1 answer
487 views

Best practices for large JOINs - Warehouse or External Compute (e.g. Spark)

I am working on a problem that requires a very large join. The JOIN itself is pretty straightforward but the amount of data I am processing is very large. I am wondering for very large JOINs, is there ...
Arthur Putnam's user avatar
0 votes
0 answers
78 views

Is there an open source implementation of QGM (Query Graph Model)?

I am building a new system that needs to interact essentially as an SQL backend. We would like to import logical queries into it (e.g. from ApacheSPARQ or Postgres and related things) and want to ...
intel_chris's user avatar
1 vote
0 answers
114 views

MySQL is not showing the history queries after the ETL is complete [closed]

I am using docker container for MySQL and Spark. Both containers are on AWS Ec2 instance. Pyspark ETL connects to MySQL with JDBC and start extracting the data. When the ETL is running i can see the ...
nomad123's user avatar
2 votes
2 answers
474 views

Check the time that PostgreSQL is taking to automatically create existing indexes when you do bulk insert using copy command from Spark

I am doing a bulk insert from the spark to the postgres table. Amount of data that I am ingesting is huge. The number of records is around 120-130 million I am first saving the records as multiple csv ...
Nikunj Kakadiya's user avatar
0 votes
3 answers
91 views

Data storage for analytics [closed]

I have to store some amount of data for analytical purposes. The data source produces 2TB data per month. Data is collected on a monthly basis (not real-time). Data is fully structured. There are 100+...
Leeloo's user avatar
  • 111
0 votes
1 answer
686 views

Bulk SQL INSERT into Azure SQL Database using spark causes blocking/contention?

I am running the following code using microsoft's sql sparkconnector to write a 1-2 Billion dataframe into Azure SQL Database. df.write \ .format("com.microsoft.sqlserver.jdbc.spark") \ ....
user avatar
1 vote
0 answers
345 views

Pyspark error inserting into mysql database table that exists

I am trying to insert into an existing mysql table using Pyspark JDBC connection however I get the following error(picture attached) Can I get assistance on this error. The table exist in the MySql ...
IIShriyaII's user avatar
0 votes
1 answer
1k views

Auto increment field on update

I have a field called: update_count, which I need to reflect the number of times a row has been updated. I have been looking into the SERIAL type for the column which sounds promising, but it has one ...
7heViking's user avatar
  • 101
1 vote
0 answers
3k views

HIVE + understanding the hive-metastore logs

we have HDP cluster version - 2.6.4 , and we are runs spark streaming app and we are uses presto cluster in order to run Hive queries when we look on the hivemetastore logs ( under /var/log/hive ) ,...
King David's user avatar
0 votes
0 answers
32 views

Move from SSIS to Azure

I have an ETL solution which we have been doing in conventional SSIS. The job of SSIS is just to create star schema from the application database to dimension and fact tables. There is very little ...
imba22's user avatar
  • 141
0 votes
1 answer
3k views

How to use case and filter with a group by statement and having an aggregate calculation?

I would like to do the following SELECT PO.col1, PO.col2, CASE WHEN PO.col3 <> 8 THEN PO.col4 WHEN PO.col3 = 8 THEN CASE WHEN (ROUND(CAST(...
scientific_explorer's user avatar