Questions tagged [databricks]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Writing large dataset from spark dataframe

We have a azure databricks job that retrieves some large dataset with pyspark. The dataframe has about 11 billion rows. We are currently writing this out to a postgresql DB (also in azure). Currently ...
Kyle Chamberlin's user avatar
-2 votes
0 answers
8 views

Data Validation in databricks

I have created a table(1st) and loaded data from another table(2nd) in databricks using merge statement. But after loading, some rows are missing in the 1st table which are there in 2nd table. So can ...
Dibyarup Dhar's user avatar
0 votes
1 answer
15 views

Next Business Date Column

I have a dataset that looks like this. Where business_day indicates whether the transaction_created_date is a business day or not. I'm trying to sum the line_amount so that values that occurred over ...
Lena Zheng's user avatar
0 votes
0 answers
45 views

Whitelist Azure Databricks SQL Warehouse access to selected tables

Is there an option to allow a service principal (SP) that accesses Azure Databricks SQL Warehouse using a personal access token (PAT) to restrict access only to selecting data from present tables and ...
Talos's user avatar
  • 101
-1 votes
1 answer
30 views

How to create "On this day in history" query

I'm using Databricks and I have a table with a list of event from various years. I want to return the event most recent to today's date from each year. For example, Today's date is 6th May and my ...
ben_al's user avatar
  • 1
1 vote
1 answer
311 views

DESCRIBE TABLE in databricks piped into dataframe

Does anyone know of a method to pipe the "DESCRIBE TABLE" output in databricks into dataframe? (or other usable format which could be used for further analysis/computation)?
Doc's user avatar
  • 121
1 vote
1 answer
2k views

How to call python file in repo in databricks from data factory outside DBFS?

In Azure Databricks I have I have a repo cloned which contains python files, not notebooks. In Azure Data Factory I want to configure a step to run a Databricks Python file. However when I enter the /...
Brendan Hill's user avatar
2 votes
0 answers
584 views

Troubleshoting slow running queries/jobs in Azure Databricks

I have Azure Databricks workspace with cluster configured to run Standard 6.4 runtime (Apache Spark 2.4.5, Scala 2.11). Cluster uses shared metastore (Azure MySQL). I'm trying to figure out possible ...
Mike's user avatar
  • 727