What's new in Databricks - October 2024
October 2024 Release Highlights
AI functions is now available in EU regions
Databricks Apps is now public preview
GenAI/ML
You can now use a customizable dashboard template to monitor the quality of your Agents, surface quality issues and do root cause analysis on individual requests (public preview) → Documentation & Demo
Mosaic AI Model Serving now supports batch LLM inference using the SQL function ai_query (public preview) → Documentation & Demo
OpenAI-compatible structured outputs are now supported on Mosaic AI Model Serving, allowing you to specify a JSON schema to format responses generated from your chat models (public preview) → Documentation & Demo
AI Functions powered by Foundation Model APIs are now available in EU regions eu-west-1 and eu-central-1 → Documentation
The Llama 2 70B Chat model is now retired → Documentation
Governance
Fine grained access control on single user compute is GA
Enable a metastore to be assigned automatically to new workspaces
Data Engineering
Salesforce ingestion connector simplification
The Salesforce ingestion connector now automates source setup. Learn more
Force reclustering on tables with Liquid clustering (DBR 16+)
You can use the optimize full syntax to force the reclustering of all records in a table with liquid clustering enabled. Learn more
Create liquid clustered tables during streaming writes(DBR 16)
You can now use clusterBy to enable liquid clustering when creating new tables with Structured Streaming writes. Learn more
Support for the optimize full clause(DBR 16)
This clause optimizes all records in a table that uses liquid clustering, including data that might have previously been clustered.
Platform
Databricks Apps
You can use popular Python frameworks to create applications that run in the Databricks platform and use the resources and features of the Databricks platform, including Unity Catalog for governance, Databricks SQL to query data, AI features such as model serving, and the already configured security rules in your workspaces. Learn more
Notebooks improvements
Notebooks results tables now persist filters when you reopen a notebook.
Improvements to the UI for commenting in notebooks make comments easier to add and find.
Improve the efficiency of SQL queries with syntax warnings and the
/optimizeslash command. Optimizations pop up in real time, helping you identify inefficiencies before execution.
Databricks Assistant Autocomplete is now GA
Assistant Autocomplete provides intelligent inline code suggestions as you type to help you write code more quickly and efficiently →Blogpost
The billable usage system tables now attributes networking costs
The billable usage system table now includes fields to help monitor costs incurred from connecting serverless compute to your resources. Learn more
Attribute serverless usage with budget policies
To assist with serverless billing attribution, workspace admins can now create and assign budget policies to users, groups, or service principals. Budget policies enforce custom tags on all serverless usage incurred by the policy assignee. This allows for granular billing attribution of serverless usage in notebooks, jobs, and pipelines. Learn more.
Enable cost attribution of fine grained access control on single user compute
This release includes a change to assist with the cost attribution of fine-grained access control on single user compute. With this change, when the single user compute passes a query to serverless compute to run data filtering, any custom tags assigned to the single user compute are propagated and recorded in the system.billing.usage table. Learn more
Data Warehousing
Data Warehousing update summary
Streaming tables and materialised views are now GA on Databricks SQL.
Queries and visualisations from SQL cells in notebooks can now be added to AI/BI dashboards
New columns query_source, executed_as, and executed_as_user_id have been added to the query history table for improved traceability
Notification destinations are now GA. Users can create and configure notification destinations that workspace users can add to certain workflows such as alerts, Databricks jobs, and AI/BI dashboard schedules to send emails or webhooks based on an event trigger
AI/BI Dashboards updates with more visuals
AI/BI dashboards are now updated with box charts to show spread of of numerical data, and any skews through quartiles. Cross filtering on dashboard now supports multi-select on bar, scatter, heatmap and pie charts. There are also additional functionalities for pivot tables which now includes subtotals, sorting by measures and conditional formatting of data
Symbol map visualisation is now also available, allowing you to plot latitude and longitude on maps.
AI/BI Genie
Genie can now intelligently filter columns, descriptions and example SQL queries to help avoiding token limits when generating a response.