Indexing Queue

Introduction

If any issues arise with your data, indices, or queue, please check the troubleshooting guide.

Before the Magento data is searchable, it needs to be uploaded to our servers and indexed. The indexing queue manages all the synchronization with our servers automatically.

To read more about the indexing process for Magento 2, please read the page on indexing.

The indexing queue processes updates to products, categories, pages, and any other data by sending the updates to our servers asynchronously. This way, the data in Magento and on our servers will be up to date at all times, providing the best user experience for the customers.

Because the queue processes asynchronously, the Magento administrator does not have to wait after every change until the index is updated.

Configuring the queue

To enable the indexing queue, navigate to System > Configuration > Algolia Search > Indexing Queue/Cron in the Magento administration.

Enabling the indexing queue is recommended for production environments.

All queued operations will be stored in the database, in a table called algoliasearch_queue. By default, the queue will run no more than 10 operations at a time. This amount can be scaled up or down in the settings, to better suit the resources available on the server.

To find out how many operations the server can handle, the following process can be followed:

1. Turn off the cron job.
2. Set the number of operations to process to 10.
3. Manually run the indexer.
4. Measure how long one run takes.
5. If it’s lower than 4 minutes, increase the number of jobs to process.
6. Repeat from point 3 until the highest possible value has been found.

Don’t forget to turn the cron job back on.

Note: the steps mention 4 minutes, and not 5. This is to keep a safe margin with the running of the cron job. It’s best practice to keep a margin in case the cron job is slower than normal due to unforeseen circumstances, like a high server load.

Failed operations

Whenever an operation fails during the processing of the queue, it will be re-triggered to be processed the next time the queue runs. In order to prevent operations from being re-triggered infinitely, the maximum amount of retries can be configured in the settings.

Processing the queue

Once the queue is enabled, the process to run it needs to be set up. There are two ways to achieve this.

Automatically

The preferred way to handle the queue is by processing it at a regular time interval. To do this, the following crontab entry has to be configured.

1
*/5 * * * * php absolute/path/to/magento/bin/magento indexer:reindex algolia_queue_runner

This crontab will run every five minutes, running the amount of operations set in the configuration (10 by default).

Manually

While the crontab entry is preferred to get regular updates to the data when necessary, it’s also possible to manually trigger the indexing jobs. To process the queue manually, run the following command from the command line:

$
php path/to/magento/bin/magento indexer:reindex algolia_queue_runner

Running the queue manually may not empty the whole queue, as it will only run the amount of operations set in the configuration.

Emptying the queue

When all operations queued need to be processed at once, the PROCESS_FULL_QUEUE=1 parameter needs to be passed when manually processing the queue. Run the following command in the command line to process the complete queue at once:

$
PROCESS_FULL_QUEUE=1 php path/to/magento/bin/magento indexer:reindex algolia_queue_runner

Please note that this is not recommended. When processing a large amount of operations, the chance of error increases - network timeouts, php timeouts, and memory problems could possibly happen. If these errors occur, the queue will not be completely empty, and the data will not be fully up-to-date. To resolve any encountered errors, please have a look at our troubleshooting pages.