Questions tagged [apache-spark]

Use this tag for questions related to the distributed computing framework published by Apache.

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

MySQL is not showing the history queries after the ETL is complete

I am using docker container for MySQL and Spark. Both containers are on AWS Ec2 instance. Pyspark ETL connects to MySQL with JDBC and start extracting the data. When the ETL is running i can see the ...
0
votes
0answers
47 views

Would Star Schema be useful for a Lakehouse table accessed from PowerBI?

In a Lakehouse, the data is stored in a datalake (deltalake in to be precise in my case) instead of a data warehouse. The data there is stored in parquet format, which is a columnar format. I am ...
1
vote
2answers
52 views

Check the time that PostgreSQL is taking to automatically create existing indexes when you do bulk insert using copy command from Spark

I am doing a bulk insert from the spark to the postgres table. Amount of data that I am ingesting is huge. The number of records is around 120-130 million I am first saving the records as multiple csv ...
0
votes
3answers
54 views

Data storage for analytics [closed]

I have to store some amount of data for analytical purposes. The data source produces 2TB data per month. Data is collected on a monthly basis (not real-time). Data is fully structured. There are 100+...
0
votes
1answer
170 views

Bulk SQL INSERT into Azure SQL Database using spark causes blocking/contention?

I am running the following code using microsoft's sql sparkconnector to write a 1-2 Billion dataframe into Azure SQL Database. df.write \ .format("com.microsoft.sqlserver.jdbc.spark") \ ....
1
vote
0answers
35 views

Pyspark error inserting into mysql database table that exists

I am trying to insert into an existing mysql table using Pyspark JDBC connection however I get the following error(picture attached) Can I get assistance on this error. The table exist in the MySql ...
0
votes
1answer
18 views

Auto increment field on update

I have a field called: update_count, which I need to reflect the number of times a row has been updated. I have been looking into the SERIAL type for the column which sounds promising, but it has one ...
1
vote
0answers
1k views

HIVE + understanding the hive-metastore logs

we have HDP cluster version - 2.6.4 , and we are runs spark streaming app and we are uses presto cluster in order to run Hive queries when we look on the hivemetastore logs ( under /var/log/hive ) ,...
0
votes
0answers
25 views

Move from SSIS to Azure

I have an ETL solution which we have been doing in conventional SSIS. The job of SSIS is just to create star schema from the application database to dimension and fact tables. There is very little ...
0
votes
1answer
2k views

How to use case and filter with a group by statement and having an aggregate calculation?

I would like to do the following SELECT PO.col1, PO.col2, CASE WHEN PO.col3 <> 8 THEN PO.col4 WHEN PO.col3 = 8 THEN CASE WHEN (ROUND(CAST(...