Zero downtime index deployment

v0.30 of Meilisearch was released earlier this month. With it, our primary goal was to make the Meilisearch workflow as smooth as possible. After all, we are on the road to v1, and our destination is closer than ever. For that, having index deployment with zero downtime was a crucial step. We’re happy to say that v0.30 solves this problem, making it easier than ever to update an index in production. Read on to learn more about how the feature works!

The challenge

Suppose you have changes in your database that you need to synchronize with your Meilisearch index in production. What do you do? Well, if you were working on Meilisearch v0.29, you might:

Update your index

🙅‍♀️  Simple, right? But if you’ve ever tried it, then you should already know…

Updating your production index while it receives search queries can result in inconsistencies with results or even missing information. You don’t want to offer that search experience, do you? Otherwise, you wouldn't be using Meilisearch 😁

Delete the index, create a new index with the same name, and re-index the data

⌛ Each step will take some time. Even if it’s just a few seconds, it’s a few seconds of downtime… And that’s at least three requests! A few seconds here plus a few seconds there can result in minutes of downtime! No, thank you very much.

Create a new index with another name, stage the changes there, then modify all the clients to point to the new index

🤔 Well, it’s not the worst option. But again, updating the clients leads to downtime 😞 In some cases, it isn’t even up to the developer when the client update will be pushed to production. Take an iOS app for example: Apple has to review the new version before you can release it, which can be a major drag on your release plan.

Create an intermediate redirection layer to avoid downtime

🧞 Very clever! While this is a good approach, it's not the most straightforward. It requires architectural knowledge, time, and extra tools.

Aren't you glad you never have to do any of these things again?

The silver bullet

Meilisearch has always been committed to offering the best developer experience.

Our team has been working hard for the past months, and after several iterations, they’ve come up with a solution that integrates seamlessly into the developer’s stack: index swapping.

Let me show you how it works.

Let’s say you have an index in production—indexA—in which your clients search. You want to sync the changes in your primary database with Meilisearch. To do so, you would follow the steps below.

Step 1: Create a new index with up-to-date data

First, you need to create an index—let’s call it indexA_new—representing the new version of indexA that you want to deploy to your search clients. Add up-to-date documents from your database, and update the index settings if necessary.

Don’t forget to check that all the tasks related to the index creation have successfully completed, including indexCreation, settingsUpdate, and documentAdditionOrUpdate task types. You can use the /tasks route to get information about their progress.

Step 2: Test the new index

Before sending your index to production, you want to be sure everything is working as expected.

Make sure to update the settings for any new fields you may have introduced with the updated data. For example, you may want to add new fields to searchableAttributes, filterableAttributes, and/or sortableAttributes. Don't forget to double-check the relevancy of search results any time you add new data!

👉 Keep in mind that the searchableAttributes list not only designates the fields that are searchable, but also dictates the attribute ranking order. Make sure any new fields you may have introduced are added to the list in the right order.

Step 3: Swap indexes

Once your indexA_new has been successfully created, filled with data, and tested, it’s ready to be deployed with an index swap. To do so, send a POST request to the /swap-indexes endpoint. Specify the indexes you want to swap in the payload. Since it’s a swap, the order doesn’t matter.

curl \
-X POST 'http://localhost:7700/swap-indexes' \
-H 'Content-Type: application/json' \
--data-binary '[
	{ "indexes": ["indexA", "indexA_new"] }
]'

👉 In a protected Meilisearch instance, the API key used to swap indexes must have access to the indexes.swap action as well as the indexes you want to swap. If not, Meilisearch will throw an invalid_api_key error. For more information about creating API keys with specific permissions, see the documentation.

You can use the response's taskUid to track the status of your request with the GET /tasks/{task_uid} endpoint. A successful index swap should look like this:

{
  "uid": 23,
  "indexUid": null,
  "status":"succeeded",
  "type":"indexSwap",
  "details":{
	"swaps": [
  	     {"indexes": ["indexA", "indexA_new"]},
	]
  },
  "duration": "PT1S",
  "enqueuedAt": "2021-08-10T14:29:17.000000Z",
  "startedAt": "2021-08-10T14:29:18.000000Z",
  "finishedAt": "2021-08-10T14:29:19.000000Z"
}

Your indexes have been swapped without any downtime! The documents, settings, and task history of indexA—except for any enqueued tasks—have been swapped with those of indexA_new. Every mention of indexA in the task history has been replaced by indexA_new and vice-versa (enqueued tasks are left unmodified).

After the swap, indexA_new holds the outdated content. You can delete it or keep it as a backup, should something go wrong and you need to swap back. Better safe than sorry!

And that’s it! Only three steps, two if you like to live dangerously. Could this get any better?

The cherries—yes, plural—on the cake 🍰

What if I tell you that you can swap several indexes with just one request?

I am not kidding. A single request can swap as many index pairs as you wish. Meilisearch can deploy all changes at the same time. Clients will access the new version of all indexes at once without any downtime.

curl \
  -X POST 'http://localhost:7700/swap-indexes' \
  -H 'Content-Type: application/json' \
  --data-binary '[
    {
        "indexes": ["indexA", "indexA_new"]
    },
    {
        "indexes": ["indexB", "indexB_new"]
    },
    {
        "indexes": ["indexC", "indexC_new"]
    }
]'

In the example above, three swap operations will occur simultaneously and atomically.

Wait, what?

Yes, you can read that again. It’s atomic! Either all indexes are successfully swapped, or none are. Either all the content is swapped, or none is.

Why is that important? It prevents partial changes in the database, ensuring consistency and, thus, a top-notch search experience.

Conclusion

As I mentioned earlier, this feature has been in the works for several months, and it all started with user feedback. The “swap indexes” card on our roadmap got 38 votes and almost as many comments explaining use cases where this feature is a must-have. Here are some of our favorites:

• “It would be very useful when trying to change rules on production database to tweak for the best results”
• “Would help if index was accidentally created with wrong name or need to rename/change it for various reasons.”
• “We need it to clean out all deleted items.”

These are just a few examples. This kind of insight is extremely helpful for our team, as it allows us to shape the product to fit the users' needs.

While very convenient—you can have an overall view of all the feature ideas submitted, in the making, and released—the roadmap is just one of the options to provide feedback.

The product discussions on GitHub are, in my opinion, one of the best places to explain your needs. You can have direct contact with the Meilisearch product team, and, being public, it allows anyone to participate and enrich the process.

The last option is our brand-new Discord server. Don’t hesitate to join us and talk about what you’ve built with Meilisearch, your use case, and your specific needs.

No matter which option you choose, we look forward to hearing from you!