What's new in Databricks for November and December 2023
Get started with Gen AI on Databricks
Foundation model APIs
You can now query state-of-the-art open source models from an endpoint and easily build applications that leverage a high-quality gen AI model without the overhead of maintenance.
For more information: Foundation models
Mistral 8x7b is now available as serverless LLM
On top of existing models (llama2, MPT…), Databricks just released the latest Mistral 8x7B model, providing state of the art inferences.
External models
Databricks Model Serving now offers support for External models, enabling you to seamlessly access, govern, and manage third-party hosted models known as external models. This functionality empowers you to create endpoints for interacting with models hosted outside the Databricks environment, including but not limited to Azure OpenAI GPT models, Anthropic Claude Models, or AWS Bedrock Models. Once configured, teams and applications can be granted access to these models, allowing them to query through a standardized interface without the need to expose credentials.
Databricks Vector Search fully managed is now available for RAG Application on Databricks
Databricks Vector Search is designed for high performance at a lower cost, with up to 5 times lower latency than other options. It simplifies the developer experience by enabling easy synchronization of any Delta Table into a vector index with a single click. Select your table, the field to index, and your embedding models, the platform does the rest. This eliminates the need for complex custom data pipelines.
Vector Search ensures unified governance by using the same security and data governance tools as the Data Intelligence Platform, eliminating the need for separate policies for unstructured data.
Databricks Assistant is becoming smarter
Databricks Assistant is a context-aware AI assistant, available natively in Databricks notebooks, SQL editor and file editor. Databricks Assistant lets you query data through a conversational interface, making you more productive inside Databricks.
Databricks Feature Store
Online tables for real-time features serving public preview is starting!
Online tables are a fully serverless, read-only copy of a Delta table optimized for low latencies, online access. You can use it as a realtime Key-Value database to retrieve rows in milliseconds given a specific ID!
This is the beginning of new game-changer capabilities, moving toward low latency use cases for your feature-store retrieval!
For more information: Online tables
Databricks Feature and function serving
Not only Databricks can serve pre-feature, but also package your transformations to prepare the features for your model. This makes it easy to always use the same transformation for training and inference, batch and low latency!
Function serving scale to accommodate dynamic real-time traffic and deliver a highly available low latency service, regardless of scale.
For more information: Function and feature serving
Governance and Devx on Databricks
Data modeling made easy with PK/FK support
Entity Relation are now available in the Catalog Explorer. You can view primary key and foreign key relationships as a graph in the Entity Relationship Diagram (ERD).
This makes it easy to explore and discover your database relationship, allowing the AI Assistant to provide more accurate answer!
Big Query available in Lakehouse Federation
Lakehouse Federation addresses critical pain points that make it simple for organizations to expose, query and govern siloed data systems as an extension of their lakehouse. To enrich the user experience, you can now configure Lakehouse Federation to run queries with the BigQuery Connector Type.
Unity catalog improvements
Unity Catalog will be enabled automatically for all the new workspaces!
Improvements in Unity Catalog:
You can Delete/ Download files in the catalog explorer for files in Volume
You can now install libraries stores as a workspace files to compute configured with no-isolation shared access mode in DBR 14.1 +
You can now grant user permission to create a storage credential in a Unity Catalog metastore using the new privilege.
Shallow clone for Unity Catalog external tables ( DBR 14.1+)
Get started with the Marketplace on Databricks
The Databricks marketplace is a comprehensive platform for data analytics and AI, offering a wide range of opportunities to drive innovation and progress in your analytics and AI projects. It now features solution accelerators, complete with fully functional notebooks, best practices, and sample data. These accelerators facilitate quick setup for popular use cases, enhancing efficiency and effectiveness.
You can get the data product from the marketplace with Delta Sharing connectors to access various platforms like Power BI, Excel, Apache Spark and Databricks Workspace without Unity Catalog.
In order to simplify the user experience you can now find data sets, AI Models and Solution Accelerators in the marketplace using the global search Bar.
Get started with the Delta Sharing capabilities on Databricks
Share Volumes
You can now use Delta Sharing to share volumes between Databricks workspaces on different Unity Catalog metastores including workspaces on different Databricks accounts and different clouds.
Share dynamic views
You can now use Delta Sharing to share dynamic views that restrict access to certain table data based on recipient properties.
For more information:
Add dynamic views to a share to filter rows and columns
In a nutshell
Databricks JDBC driver 2.6.36 is now available. For more information
You can now select from a variety of themes in the notebook editor
You can now select a new cell table rendering that allows results, copying subsets of the results to the clipboard and other features to make it easier to work with tabular outputs
Databricks Runtime 14.2 is GA. For more information: Link
Authentication using OAuth is GA
Deprecation of Workspace Libraries
Access Controls lists can no longer be disabled
IAM Policies for storage credentials now require an external ID