Questions tagged [apache-spark]
Use this tag for questions related to the distributed computing framework published by Apache.
19
questions
0
votes
0
answers
11
views
CDC on postgreSQL to Redshift
I am trying to implement CDC on PostgreSQL to Redshift and looking for some approaches. I had a look at Debezium but it looks like it's not ideal for my requirement. I want to read all delta changes ...
- 1
1
vote
0
answers
30
views
Why does MySQL select offset not work in pyspark?
select distinct day
from t1
order by 1 desc limit 1,1
should return the second latest date in t1, but this error is thrown:
ParseException: mismatched input ',' expecting {, ';'}
1
vote
0
answers
18
views
Optimal join for joining facts with scd-type-2 dimension for aggregation/reporting
I have a fact table and an scd-type-2 dimension table. I want to produce sales report by region and year.
I have working solution with a query that joins them for reporting purposes. When I run the ...
- 139
0
votes
0
answers
69
views
pyspark.sql.utils.AnalysisException: Accessing outer query column is not allowed in CASE SELECT Subquery
I'm trying to re-write the below SQL query into PySpark dataframe.
query = '''
SELECT dense_rank() OVER(PARTITION BY t3.id, t3.uid, t3.datetime) ORDER BY t3.datetime DESC, t3.eid DESC) AS rank,
...
1
vote
2
answers
59
views
When will the Spark Cassandra connector with support for Spark 3.3 be released?
Master branch has a merged pull request supporting Spark 3.3 how long before this gets built and published on maven repository?
- 111
2
votes
1
answer
21
views
Can we use Cassandra in place of Hadoop with Spark?
Considering we have a backend written in NodeJS and uses MySQL and Cassandra as it's databases, if we want to add Spark to the system to do some data analyzing stuff like recommendation, can we do it ...
- 69
1
vote
2
answers
145
views
Cassandra-4-Update: Multiple Schema Versions
after upgrading our first node, it has a different schema version (according to node tool describe cluster). This caused Spark Jobs to hang, because of reoccurring "schema agreement not reached&...
- 11
0
votes
0
answers
67
views
how to fix SQL Server Bulk Copy SQLServerException Source and destination schemas do not match
I have a springboot - Spark application that will inject a dataFrame into an SQL Server DB, the app was working fine before we decide to enable Alaway Encrypted for a couple sensible columns. Just ...
- 101
2
votes
1
answer
206
views
Best practices for large JOINs - Warehouse or External Compute (e.g. Spark)
I am working on a problem that requires a very large join. The JOIN itself is pretty straightforward but the amount of data I am processing is very large. I am wondering for very large JOINs, is there ...
- 441
0
votes
0
answers
41
views
Is there an open source implementation of QGM (Query Graph Model)?
I am building a new system that needs to interact essentially as an SQL backend. We would like to import logical queries into it (e.g. from ApacheSPARQ or Postgres and related things) and want to ...
- 141
1
vote
0
answers
78
views
MySQL is not showing the history queries after the ETL is complete [closed]
I am using docker container for MySQL and Spark. Both containers are on AWS Ec2 instance.
Pyspark ETL connects to MySQL with JDBC and start extracting the data. When the ETL is running i can see the ...
- 11
1
vote
2
answers
199
views
Check the time that PostgreSQL is taking to automatically create existing indexes when you do bulk insert using copy command from Spark
I am doing a bulk insert from the spark to the postgres table. Amount of data that I am ingesting is huge. The number of records is around 120-130 million
I am first saving the records as multiple csv ...
- 113
0
votes
3
answers
69
views
Data storage for analytics [closed]
I have to store some amount of data for analytical purposes.
The data source produces 2TB data per month.
Data is collected on a monthly basis (not real-time).
Data is fully structured.
There are 100+...
- 111
0
votes
1
answer
441
views
Bulk SQL INSERT into Azure SQL Database using spark causes blocking/contention?
I am running the following code using microsoft's sql sparkconnector to write a 1-2 Billion dataframe into Azure SQL Database.
df.write \
.format("com.microsoft.sqlserver.jdbc.spark") \
....
Youssef Fares
1
vote
0
answers
259
views
Pyspark error inserting into mysql database table that exists
I am trying to insert into an existing mysql table using Pyspark JDBC connection however I get the following error(picture attached)
Can I get assistance on this error. The table exist in the MySql ...
- 11
0
votes
1
answer
689
views
Auto increment field on update
I have a field called: update_count, which I need to reflect the number of times a row has been updated.
I have been looking into the SERIAL type for the column which sounds promising, but it has one ...
- 101
1
vote
0
answers
2k
views
HIVE + understanding the hive-metastore logs
we have HDP cluster version - 2.6.4 , and we are runs spark streaming app
and we are uses presto cluster in order to run Hive queries
when we look on the hivemetastore logs ( under /var/log/hive ) ,...
- 111
0
votes
0
answers
28
views
Move from SSIS to Azure
I have an ETL solution which we have been doing in conventional SSIS. The job of SSIS is just to create star schema from the application database to dimension and fact tables. There is very little ...
- 141
0
votes
1
answer
3k
views
How to use case and filter with a group by statement and having an aggregate calculation?
I would like to do the following
SELECT
PO.col1,
PO.col2,
CASE
WHEN PO.col3 <> 8 THEN PO.col4
WHEN PO.col3 = 8
THEN CASE
WHEN (ROUND(CAST(...
