Aiven for OpenSearch^® updated to version 2.11.1

As 2023 draws to a close, we're pleased to announce that Aiven for Opensearch® is now running on OpenSearch® 2.11.1. Since our previous release was running version 2.8.0, there's way more new under the hood than could be reasonably condensed into a single blog post. With that in mind, I've captured some of the most notable highlights below, but be sure to check out the OpenSearch release notes for a full accounting of what's been added in the last few releases.

Machine learning, neural search, and vectors. Oh my!

There are a ton of machine learning and vector search updates made available in this release. For starters, the machine learning (ML) framework is now generally available, with a variety of pre-trained models available. Models hosted on third-party platforms can also be integrated via connectors. OpenSearch provides a variety of connector blueprints for several popular AI platforms, or you can create a blueprint of your own to integrate a different platform.

When it comes to putting those platforms to work, neural search processes ingested documents by passing them through a user-defined ML model to generate vector embeddings which are indexed alongside the source text. Once an ML model has been set up, there are a variety of search methodologies that can be applied for neural search:

• Semantic search
• Hybrid search
• Multimodal search
• Sparse search
• Conversational search (experimental)

Search pipelines are generally available

Search pipelines allow you to customize your searches by chaining together a series of modular processors to modify a query, its results, or both. A search pipeline can perform tasks like personalizing search results or automatically applying a predefined set of query filters, which would otherwise need to be handled within your application code. Previously available only as an experimental feature, pipelines are now generally available and ready to simplify your apps.

Integrated alerts and anomaly detection in OpenSearch Dashboards

The Anomaly Detection and Alerting plugins are invaluable tools for getting insight into potential issues impacting your users. However, historically it's been difficult to correlate the data they provide with existing dashboards and visualizations.

In this release, new monitors and detectors can be created directly from OpenSearch Dashboards, and can additionally overlay triggered alerts and anomalies over their associated visualizations. Existing monitors and detectors can be associated with visualizations in similar fashion. The end result is a single view that combines your visualizations with related alerts and anomalies, making it easier than ever to diagnose issues and contextualize them with relevant metric visualizations.

Side-by-side search query comparison

The relevance of a given search result can depend on the context in which a particular query is evaluated. With varying field weights, document parsing strategies, or ML models, ostensibly similar queries can produce significantly different results.

This release brings the search comparison tool to general availability, which provides a side by side comparison of two different search queries in OpenSearch Dashboards. This makes it easier than ever to compare the results, rankings, and relevance of different query strategies without having to change contexts or juggle multiple windows.

Upgrade your clusters today

To reiterate, this is all just the tip of the iceberg – for a full list of changes and updates, be sure to check out the OpenSearch release notes.

To get started today, you can perform a maintenance upgrade on your OpenSearch clusters from the Aiven console. Happy searching!

Note: An earlier version of this post referred to custom local model support, which is not yet available on Aiven for OpenSearch®. Stay tuned for updates to ML Commons functionality