Databricks Integration Setup Guide

Overview

This guide walks through how to authenticate Databricks with BoostUp and configure access so we can securely read data from your Databricks tables.

1. Authentication (OAuth M2M via Service Principal)

BoostUp authenticates to Databricks using an OAuth machine-to-machine (M2M) flow with a Databricks Service Principal.

Steps:
1. Log into your Databricks workspace.
2. Navigate to User menu → Settings → Identity and Access → Service Principals.
3. Click Add Service Principal and name it (e.g., boostup-databricks-sp).
4. Open the service principal and create a secret under Manage → Secrets.

BoostUp will use the Service Principal Client ID and OAuth Client Secret for authentication.

2. Storage Integration (External Objects)

Create a storage integration pointing to BoostUp’s S3 bucket.

Steps:
1. Navigate to Catalog → Manage → Credentials.
2. Click Create Credential.
3. Select Storage Credential and choose AWS IAM Role.
4. Use the following IAM Role ARN (substitute <email_domain> by your company’s email domain):
arn:aws:iam::258989535243:role/databricks-etl-external-objects_<email_domain>
5. Assign a name and create the credential.

Please provide BoostUp with the External ID and generated AWS IAM policy.

3. Create an External Location

Run the following SQL command:

CREATE EXTERNAL LOCATION ext_loc
URL 's3://boostup-production-databricks-etl-external-objects/<email-domain>/'
WITH (CREDENTIAL <name_of_storage_credential>);

4. Grant Permissions

Grant file permissions:
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION ext_loc TO <service_principal_app_id>;

Grant table permissions:
- USE CATALOG
- USE SCHEMA
- SELECT on target tables.

5. Provide Table Metadata

Please send BoostUp the following for each table:
- Catalog name
- Schema name
- Table name
- Column names and data types
- Unique ID column
- Optional: datetime column for incremental sync

6. Create the Databricks Connection in BoostUp

Please send BostUp the following databricks information so we can connect you up:

•     server_hostname = fields.String(required=True)
•     http_path = fields.String(required=True)
•     client_id = fields.String(required=True)
•     client_secret = fields.String(required=True)
•     catalog = fields.String(required=True)
•     schema = fields.String(required=True)
  
  The connection will only save if authentication is successful.

7. Final Validation

BoostUp will validate access, confirm table visibility, and enable syncs.

Note: You can find the http path by:

1. Log in to your Databricks workspace.
2. In the sidebar, click SQL Warehouses (sometimes labeled simply as SQL).
3. In the list of available warehouses, click the name of your target warehouse.
4. Go to the Connection Details tab.
5. Search HTTP path value. It will look something like /sql/1.0/warehouses/a1b234c567d8e9fa.