What's new in Databricks - September 2024

NextGenLakehouse

Oct 08, 2024

September 2024 Release Highlights

Publish to Power BI is GA
Meta Llama 3.2 3B models are now supported in Model Serving
Mosaic AI Gateway has several new features including usage tracking, payload logging, and guardrails.

GenAI/ML

No time to read? Check out this 10-minute recap video of all the announcements listed below for September 2024 👇

Meta Llama 3.2 is available on Databricks: 1B-Instruct and 3B-Instruct are purpose built for low-latency and low-cost enterprise use cases. 11B-Vision-Instruct and 90B-Vision-Instruct can be used for visual understanding tasks, like document parsing and product description generation. → Blog Post
Mosaic AI Gateway allows you to manage AI traffic for a wide range of models, including OpenAI, Anthropic, and Meta Llama models. New capabilities were introduced, including usage tracking, payload logging, and guardrails. → Blog post and Demo:

Hybrid search is generally available in Mosaic AI Vector Search. Pre-trained embedding models are trained using external data and don't have explicit knowledge of your data. By using hybrid search, you can add a learned keyword search index on top of your vector search index. → Blog post and Demo
Mosaic AI Agent Evaluation includes built-in LLM judges can reason about the semantic correctness of a generated answer with respect to a reference answer. Significant improvements to the built-in LLM judges, with an improved answer-correctness judge in Agent Evaluation. → Blog post and Agent Evaluation Demo:
You can now give your AI agent tools in the form of Unity Catalog functions and interact with the agent directly in AI Playground. You can export the AI agent to notebooks to iterate further, evaluate quality, and deploy it. → Documentation

Governance

System tables are now GA

The Databricks system tables platform is now generally available. This launch also includes the GA release of the billing.usage and billing.list_price tables

The billing.usage table now includes a usage_metadata.central_clean_room_id value, allowing you to monitor costs incurred by clean room usage.

Get started with System tables

Extended AI generated Comments support

AI-generated comments are supported now for Catalogs,Schemas, Tables, Table columns, Functions,Models and Volumes. Don’t forget to use this feature to make your data assets discoverable

Control external access to data in Unity Catalog using the new external use Schema privilege

The new privilege enables you to restrict access to data in Unity Catalog when external processing engines like Iceberg clients or Microsoft Fabric use Unity Catalog open APIs or Iceberg APIs to access that data. Learn more

Data Engineering

Row Filters and Column Masks for materialized views and streaming tables are in Public Preview. You can now add row filters and column masks to a materialized view or streaming table and you can define materialized views or streaming tables from source tables that include row filters and column masks.

Data Warehousing

You can now use named parameter marker syntax in the SQL editor. Named parameter marker syntax can be used across the SQL editor, notebooks, and AI/BI dashboards. See Work with query parameters.
Enable UniForm Iceberg using ALTER TABLE
You can now enable UniForm Iceberg on existing tables without rewriting data files. See Enable by altering an existing table.
Optionally allow the optimizer to rely on unenforced foreign key constraints
To improve query performance, you can now specify the RELY keyword on FOREIGN KEY constraints when you CREATE or ALTER a table.
Improved performance for change data feed with selective overwrites
Selective overwrites using replaceWhere on tables with change data feed no longer write separate change data files for inserted data. These operations use a hidden _change_type column present in the underlying Parquet data files to record changes without write amplification.
Improved query latency for the COPY INTO command
This release includes a change that improves the query latency for the COPY INTO command. This improvement is implemented by making the loading of state by the RocksDB state store asynchronous. With this change, you should see an improvement in start times for queries with large states, such as queries with a large number of already ingested files.

AI/BI

Genie

Users can now request a review from a space editor if they’re unsure about a Genie response. See Provide response feedback.
Genie space editors can now clone a Genie space to create a new space for testing or reusing instructions and other context settings. See Clone a Genie space.
Genie space editors can now define and run benchmark questions to measure a Genie space’s overall accuracy. See Use benchmarks in a Genie space.

Dashboards

Draft dashboards now automatically come with a draft Genie space based on the dashboard’s datasets and visualizations. When you publish your dashboard, you can also publish the associated Genie space so that viewers can use it to ask their own questions. See Enable a Genie space from your dashboard.
Cross filtering is now available in AI/BI dashboards to allow users to build highly interactive dashboards.

You can now embed AI/BI dashboards as an iframe in applications outside of Databricks. See Embed a dashboard.
Static widget parameters now allows you to create multiple visualisations highlighting different aspects of the data.
Filter widgets are now automatically named according to the selected filter fields.
Quick filter settings for the Date picker filter widgets now support 3, 6, and 9 months as options. Click the calendar icon on the right side of the date selector to view all quick filter options.
You can now copy and share direct links to dashboard widgets using the widget kebab menu.

DevXp

Databricks extension for Visual Studio code is GA

The extension allows you to connect to your remote Databricks workspaces from Visual Studio Code and then easily define, deploy, and run Databricks Asset Bundles, debug notebooks and run them as jobs, run files on clusters and as jobs, and synchronize local code to your workspace, all from the VSCode IDE.

Databricks Assistant Quick Fix debugs code inline

Assistant Quick Fix recommends single-line fixes for running code when it returns an error. Accept the fix and continue to run the code.