Indexing
Indexing
In order to provide fast and relevant search, we restructure your data in a special way through a process called Indexing. The extension does this for you. It runs through all of your data - products, categories, and pages - and creates indexable objects out of it. These objects are then uploaded to our servers, either automatically via the extension’s queue, or manually via the Magento console or the command line. Once pushed to our servers, the objects go through an indexing process that transforms them into searchable data.
To learn about the indexing process, have a look at the documentation.
If you are having any issues with your data, indexes, or queue, please check our troubleshooting guide.
The extension automatically keeps all your data up to date to provide the best search experience for your users. To do this, we provide two indexing mechanisms in Magento:
- Section reindex - An entire section of the catalog (products, categories, pages) is pushed to our servers and reindexed.
- Single item reindex - A single resource (product, category, page) is pushed to our servers and reindexed. This happens when a resource is updated.
By default, the indexing operations run synchronously. This means the Magento administrator has to wait until the indexing process is finished before continuing. Since this is inconvenient, and can cause unexpected issues, we created the indexing queue. This will process all index operations in the background and has some fail-safes built in.
Indexing products
It is essential for e-commerce businesses to have exact and up-to-date product data in the search. To accommodate as many use-cases as possible, we provide a number of configurations for search and indexing operations.
Indexable products
To prevent too many products from being indexed, and therefore to lower the number of indexing operations performed, we only index products that will actually show up in the webshop. This results in a set of requirements a product has to meet before we index the product.
We only index products that are:
- Visible - either in the catalog, the search, or both
- Enabled
- Not deleted
- In-stock - unless the Magento settings tell us to show out-of-stock products too
If there’s ever a missing product in your index, make sure the product meets all these requirements.
More information about troubleshooting for missing data can be found here.
Searchable attributes
It’s possible to configure which attributes should be searched when users type their query. To configure the list of searchable attributes, navigate to the products tab through System > Configuration > Algolia Search.
In the products tab, it’s possible to configure per attribute if it is searchable, retrievable or ordered. By default, all attributes are set to be searched as unordered. In general, this is better for the relevance of the search and we don’t recommend changing it without a specific reason.
Read the dedicated documentation to learn more about the difference between ordered and unordered search.
Another column in the configuration screen allows for the indexing of an empty value. Most of the time, this will be set to ‘No’ to prevent empty values from being indexed. This comes in handy when using the attribute for faceting. If empty values are indexed, Magento will automatically convert it to a ‘No’ value, which will then show up in the filters.
Default searchable attributes
Some attributes are indexed regardless of what is specified in the configuration. These attributes are not all searchable, but can be used for filtering, sorting, customizing the ranking, and building the results page.
The attributes that are always indexed:
| name | The product’s name |
| url | The product’s URL |
| visibility_search | The product’s visibility in the search |
| visibility_catalog | The product’s visibility in the catalog |
| categories | The product’s categories, formatted as a tree path |
| categories_without_path | The product’s categories, without the tree path |
| thumbnail_url | The product’s thumbnail image |
| image_url | The product’s main image |
| in_stock | The product’s stock availability |
| price | The product’s price |
| type_id | The product’s type (simple, configurable, bundled, etc.) |
Facets
Facets are the attributes that will be used as filters on the results page. Common facets include price, color, categories, and brand. However, this will not work for every store. The facets have to be tuned to the products being sold and the way the end user searches for these products.
There are a couple of things that need to be specified for each facet:
- The attribute
- The label - this will be displayed above the filter
- The type of facet
By default, we provide facets on the price, categories and color attributes.
Any numeric attribute (like price) will be shown as a slider.
Attributes that are specified as facets are automatically indexed as retrievable but not searchable. There’s no need to specify them in the Searchable Attributes configuration.
Sorting strategies
Sorting is only available on the InstantSearch results page.
When searching for products, users may expect multiple ways to sort the result set. For example, they might want to sort by relevance, popularity, price, or date.
The default sorting strategy when searching is sorting by relevance. Any other sorting strategy needs to be defined in the Sort Settings. For each strategy, you’ll need to define an attribute, sort order (ascending or descending), and a label.
By default, there are three sorting strategies:
- From lowest price to highest price
- From highest price to lowest price
- From newest to oldest
Each sorting strategy will create a new index, which will increase the amount of records. More information can be found here.
Attributes that are configured in a sorting strategy are automatically indexed as retrievable but not searchable. There’s no need to specify them in the Searchable Attributes configuration.
Index Settings
Through the Magento dashboard, the following settings for an index can be configured:
searchableAttributescustomRankingunretrievableAttributesattributesForFacetingmaxValuesPerFacetremoveWordsIfNoResults
Additional index settings can be managed in the Algolia dashboard.
It’s also possible to modify the settings programmatically by hooking into the algolia_products_index_before_set_settings event provided by the extension.
A list of events provided by the extension can be found here.
Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.
Indexing Categories
To keep the number of records and indexing operations as low as possible, we only index categories that are actually active. This behavior can be changed through the settings.
If set to ‘Yes’, all categories will be shown in the autocomplete search and InstantSearch results page.
Searchable Attributes
It’s possible to configure which attributes should be searched when users type their query. To configure the list of searchable attributes, navigate to the Category configuration through System > Configuration > Algolia Search > Categories.
It’s possible to configure per attribute if it is searchable, retrievable or ordered. By default, all attributes are set to be searched as unordered. In general, this is better for the relevance of the search and we don’t recommend changing it without a specific reason.
Read the dedicated documentation to learn more about the difference between ordered and unordered search.
Default searchable category attributes
Some attributes are indexed, regardless of what is specified in the configuration. These attributes are not all searchable but can be used for filtering, sorting, customizing the ranking, and building the results page.
The attributes that are always indexed:
| name | The category’s name |
| url | The category’s URL |
| path | The category’s path (parent categories) |
| level | The category’s level in the category tree |
| include_in_menu | The category’s visibility in the menu |
| _tags | Filled automatically by the extension |
| popularity | The category’s popularity |
| product_count | The category’s amount of products |
Index Settings
Through the Magento dashboard, the following setting for an index can be configured.
Additional index settings can be managed in the Algolia dashboard.
It’s also possible to modify the settings programmatically by hooking into the algolia_categories_index_before_set_settings event provided by the extension.
A list of events provided by the extension can be found here.
Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.
Indexing Pages
CMS Pages will be automatically indexed by our extension, allowing the users to search for pages in the autocomplete menu. By default, all active pages are indexed.
The settings provide an option to exclude certain pages, like error pages, so they don’t show up in the search results.
The indexing of pages can be disabled altogether by navigating to the Additional Sections configuration, as shown below.
Searchable Attributes
For pages, it’s not possible to configure the searchable attributes through the admin interface.
However, it is possible to change them programmatically by hooking into the algolia_after_create_page_object event provided by the extension.
A list of events provided by the extension can be found here.
Default searchable page attributes
These attributes are indexed by default, and are not all searchable (some are). They can be used for filtering, sorting, customizing the ranking, and building the results page.
The attributes that are always indexed:
| name | The page’s name |
| url | The page’s URL |
| slug | The page’s slug |
| content | The page’s content |
Since records for our engine have to be smaller than 10 kilobytes, any page which is longer than 10,000 characters in content will not be indexed. In this case, only the page’s name would be searchable.
Read more here about the engine’s record limit.
Index settings
The following settings will always be sent to configure the index, and cannot be changed through the admin interface:
searchableAttributes:unordered(slug),unordered(name)andunordered(content)attributesToSnippet:content:7
Additional index settings can be managed in the Algolia dashboard.
It’s also possible to modify the settings programmatically by hooking into the algolia_pages_index_before_set_settings event provided by the extension.
A list of events provided by the extension can be found here.
Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.
Indexing Suggestions
Every query that is being executed on the Magento installation is stored by Magento in the database.
Magento automatically stores the query, the number of results, and the number of searches in the catalogsearch_query table, without any involvement from our extension.
Only back-end searches are stored by Magento. Search-as-you-type searches and instant search queries are not stored.
Our extension offers the possibility to index queries that are performed regularly on the Magento installation. In the settings, it’s possible to filter relevant queries (for example by the minimum number of results, minimum popularity, etc.). The resulting queries can be pushed into the suggestions index, providing an autocomplete on the most relevant queries for the Magento installation.
To ensure that the data in the suggestion index is good and relevant, the data in the catalogsearch_query table must be relevant as well.
This can be achieved by enabling back-end search with our extension, by turning on the Enable Search and Make SEO Request settings in the Magento Administration.
With these options enabled, searches in the back-end will be processed by our extension.
Since the data in catalogsearch_query will be updated as well, the queries in this table will become more relevant over time.
By default, suggestions are not indexed. When enabling the indexing of suggestions, a manual reindex needs to be triggered. Another way to start the indexing of suggestions is by adding a recurring job to the cron table:
1
1 * * * * php -f /absolute/path/to/magento/shell/indexer.php -- -reindex search_indexer_suggest
Searchable Attributes
For suggestions, it’s not possible to configure the searchable attributes through the admin interface.
However, it is possible to change them programmatically by hooking into the algolia_after_create_suggestion_object event provided by the extension.
A list of events provided by the extension can be found here.
Default searchable query attributes
These attributes are indexed by default, and are not all searchable (some are). They can be used for filtering, sorting, customizing the ranking, and building the results page.
The attributes that are always indexed:
| query | The query’s value |
| number_of_results | The query’s number of results |
| popularity | The query’s number of searches |
| updated_at | The query’s last update timestamp |
Index Settings
The following settings will always be sent to configure the index, and cannot be changed through the admin interface:
searchableAttributes:unordered(query)customRanking:desc(popularity),desc(number_of_results)typoTolerance:falseattributesToRetrieve:queryremoveWordsIfNoResults:lastWords
Additional index settings can be managed in the Algolia Dashboard.
It’s also possible to modify the settings programmatically by hooking into the algolia_suggestions_index_before_set_settings event provided by the extension.
A list of events provided by the extension can be found here.
Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.
Indexing Additional Sections
The autocomplete menu offers the possibility to display other sections from attributes, like colors and brands for example.
For this feature to work, the instant search page must be enabled
The attributes used for the additional sections have to be set to be attributes for faceting.
Searchable attributes
For Additional Settings, it’s not possible to configure the searchable attributes through the admin interface.
However, it is possible to change them programmatically by hooking into the algolia_additional_section_items_before_index event provided by the extension.
A list of events provided by the extension can be found here.
Default searchable attributes
These attributes are indexed by default, and are not all searchable (some are). They can be used for filtering, sorting, customizing the ranking, and building the results page.
The attributes that are always indexed:
| value | The attributes’ value (e.g. Red, XL, Nike, etc.) |
Index settings
The following settings will always be sent to configure the index, and cannot be changed through the admin interface:
searchableAttributes:unordered(value)
Additional index settings can be managed in the Algolia dashboard.
It’s also possible to modify the settings programmatically by hooking into the algolia_additional_sections_index_before_set_settings event provided by the extension.
A list of events provided by the extension can be found here.
Any changes done in the Algolia Dashboard will override these settings until a full reindex is performed from the Magento Dashboard.
