I have been playing around with Apache Spark, firstly I learned PostgreSQL and I have a few queries that I needed to run on Spark. I managed to run them as SQL Strings in Spark SQL, but now I have to perform RDD operations in order to get the same results. I load my data from csv to map. Now I have to select specific columns in those maps, but I do not know how to join them. Second question I have is how to best perform RDD operations in order to get the same results from postgresql queries?

One of the queries:

SELECT Tournaments.TYear,Countries.Name,Max(Matches.MatchDate) - 
Min(Matches.MatchDate) AS LENGTH
FROM Tournaments,Countries,Hosts,Teams,Matches
WHERE Tournaments.TYear = Hosts.TYear AND Countries.Cid = Hosts.Cid 
AND (Teams.Tid = Matches.HomeTid OR Teams.Tid = Matches.VisitTid) AND 
date_part('year', Matches.MatchDate)::text LIKE (Tournaments.TYear || 
GROUP BY Tournaments.TYear,Countries.Name

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.