Databricks Data Sources

In Honeycomb Maps, connections define how the application can connect to and fetch data from data systems.

The Honeycomb Maps Databricks App comes with three different connections pre-installed:

• Lakehouse
• Lakebase
• File Upload

Connections define 'where' and 'how' to pull data. Data sources define 'what' data Honeycomb should pull - for example, a data table, or a query. Data sources are defined by users in Honeycomb. Multiple data sources can be added to a map in Honeycomb, and each data source can be used in different map layers and components

Lakehouse (Databricks Warehouse)

Data stored in the Databricks Lakehouse can be accessed in Honeycomb. All queries utilize On-Behalf-Of (OBO) authentication, meaning that the end user's identity is associated with each query that Honeycomb Maps runs. This is the default behavior and cannot be changed.

Users can define data sources in Honeycomb, which include a table name or SQL query. Honeycomb will use this query to fetch data when a map is loaded. This SQL query is run on Databricks, which means that all of Databricks spatial data features are available natively.

Features available include 80+ Spatial SQL functions (ST_Distance, ST_Transform, ST_Buffer, etc.) and the GEOGRAPHY type. These functions can also be used with SQL Parameters to build powerful interactive experiences.

Lakebase

Honeycomb also supports querying data from Lakebase instances. By default, Honeycomb Maps connects to the Lakebase instance that is configured during setup.

By default, Honeycomb uses the applications Service Principal identity to authenticate to Lakebase. OBO authentication is a planned feature.

The ability to connect to additional Lakebase instances will be available soon.

File Upload

Users can upload files directly within the Honeycomb app. Honeycomb supports CSV, GeoJSON, KML, GPKG, and Parquet files, up to 100MB.

This feature is designed to allow users to upload small datasets and add them to maps created with Honeycomb, alongside data from Databricks tables. It is meant to be used in cases where it wouldn't be feasible to maintain Databricks tables for the data, for example neighborhood boundaries for a city that is part of an ad-hoc analysis.

When users upload files, they are stored in a Unity Catalog volume by the application. By default, files are only visible to the user that uploaded them.