Skip to content

Update ElasticSearch and Elastica to 6.x#1880

Merged
melaniekung merged 35 commits into
qa/2.xfrom
dev/upgrade-elasticsearch
Dec 5, 2024
Merged

Update ElasticSearch and Elastica to 6.x#1880
melaniekung merged 35 commits into
qa/2.xfrom
dev/upgrade-elasticsearch

Conversation

@anvit
Copy link
Copy Markdown
Contributor

@anvit anvit commented Oct 9, 2024

Update ElasticSearch and Elastica and address issues introduced by breaking changes in ElasticSearch 6.x.

The most notable changes are:

  • Dropped support for multiple mapping types: ES 6.x no longer allows using multiple indices, They suggest handling this by creating multiple indices, each with its own type.
  • Mapping changes: The include_in_all mapping parameter is disallowed and the _all field is disabled by default.

Changes in Elastica:

  • Elastica only has a addDocuments, deleteDocuments, and updateDocuments methods in 6.x and there are no longer addDocument, deleteDocument, updateDocument methods for handling a singular documents.

@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 9f05a24 to 1e9114f Compare October 10, 2024 00:48
@anvit anvit self-assigned this Oct 10, 2024
@anvit anvit requested a review from a team October 10, 2024 16:22
@anvit anvit added this to the 2.9.0 milestone Oct 10, 2024
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 1e9114f to e6a7473 Compare October 10, 2024 22:11
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 7bdf937 to 0842eb9 Compare October 21, 2024 23:49
Copy link
Copy Markdown
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great @anvit!

I have not been able to look at everything in detail yet, but I have added a few comments that I think are important and may require some work already. Let me know your thoughts ;)

Comment thread apps/qubit/modules/search/actions/autocompleteAction.class.php Outdated
Comment thread lib/QubitLftSyncer.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchIndexDecorator.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPluginUtil.class.php Outdated
anvit and others added 10 commits November 15, 2024 15:53
Update ElasticSearch to 6.8.23 in the Dockerfile and update Elastica to
6.x in composer.
- 'inline' scripts are deprecated, changed to 'source'
Update arElasticSearchPlugin to use multiple indices instead of multiple
types since ES 6.x removed being able to add multiple types. Also update
mapping to remove include_in_all and add that to the _all field using
copy_to instead since include_in_all was also removed in ES 6.x
Update autocompleteAction to use multiple indices
Add the appropriate index types for ES requests that did not need to
specify explicit types in ES 5.x
Add missing document type to partialUpdate in arElasticSearchPlugin
Update condition in arElasticSearchPluginUtil to only get string
fields included in _all if copy_to is explicitly set to _all in the
mappping.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from abb1768 to 0259ed0 Compare November 15, 2024 23:53
Renamed arElasticSearchIndexDecorator as
arElasticSearchMultiIndexWrapper for clarity and added a few comments
about changes needed for ElasticSearch/Elastica 7.x.
Rename getType from arElasticSearchMultiIndexWrapper to getIndex for
clarity.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from bbf4bb1 to 0e0d87d Compare November 19, 2024 00:09
@anvit anvit requested a review from jraddaoui November 19, 2024 00:18
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from a37039a to 4b52458 Compare November 19, 2024 00:49
Add a dummy ElasticSearch index type since one is still required for
ES 6.x, instead of having a seperate type name for each of the
multiple indices.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 4b52458 to bc54543 Compare November 19, 2024 01:02
Copy link
Copy Markdown
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anvit! This looks a lot clearer to me now.

Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Update any references to ElasticSearch index type to index name.
Also add getType as an alias for getIndex for backwards compatibility
with custom themes that still might be making references to
ElasticSearch's getType.
Remove the index prefix from arElasticSearchMultiIndexWrapper and use
AtoM class names internally instead of using prefix to refer to them.
Copy link
Copy Markdown
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anvit! Getting there, just a few minor tweaks!

Comment thread apps/qubit/modules/search/actions/autocompleteAction.class.php Outdated
Comment thread lib/QubitLftSyncer.class.php Outdated
Comment thread lib/QubitLftSyncer.class.php
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchMultiIndexWrapper.class.php Outdated
Fix bug in search autocompleteAction which was showing duplicate results
for actors under repositories. Also address some of the CR feedback and
update/add methods for add, update, and deleteById in the
arElasticSearchMultiIndexWrapper.
Remove unnecessary foreach loop in arElasticSearchPlugin. Also move code
for adding filter for stripping md tags into a seperate method to make
the initialization code more readable.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from c5bbe4a to ef14555 Compare November 22, 2024 20:48
Copy link
Copy Markdown
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks @anvit!

Comment thread apps/qubit/modules/search/actions/autocompleteAction.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php
Comment thread plugins/arElasticSearchPlugin/config/mapping.yml
Comment thread plugins/arElasticSearchPlugin/lib/arElasticSearchMultiIndexWrapper.class.php Outdated
Update the index refresh method to accept an optional name param that
only refreshes that particular index in
arElasticSearchMultiIndexWrapper. Also address some small CR feedback.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from a94554a to 02b3c4b Compare November 25, 2024 22:49
@melaniekung melaniekung merged commit 9f553d3 into qa/2.x Dec 5, 2024
@anvit anvit linked an issue Dec 6, 2024 that may be closed by this pull request
@anvit anvit deleted the dev/upgrade-elasticsearch branch December 10, 2024 00:27
miceno pushed a commit to miceno/atom that referenced this pull request Feb 18, 2026
* Update ElasticSearch and Elastica dependencies

Update ElasticSearch to 6.8.23 in the Dockerfile and update Elastica to
6.x in composer.

* Remove arElasticSearchIndexDecorator.

- 'inline' scripts are deprecated, changed to 'source'

* Update index for ES 6.x

Update arElasticSearchPlugin to use multiple indices instead of multiple
types since ES 6.x removed being able to add multiple types. Also update
mapping to remove include_in_all and add that to the _all field using
copy_to instead since include_in_all was also removed in ES 6.x

* Fix autocompleteAction for ES 6.x

Update autocompleteAction to use multiple indices

* Add index types for ES Queries without a type

Add the appropriate index types for ES requests that did not need to
specify explicit types in ES 5.x

* Fix updateByQuery calls for ElasticSearch 6.x

* Fix broken publication status update

Add missing document type to partialUpdate in arElasticSearchPlugin

* Fix linting errors

* Switch elasticsearch to the oss docker image

* Update mapping condition for _all fields

Update condition in arElasticSearchPluginUtil to only get string
fields included in _all if copy_to is explicitly set to _all in the
mappping.

* Update ES mapping to copy relevant fields to _all

* Rename arElasticSearchIndexDecorator

Renamed arElasticSearchIndexDecorator as
arElasticSearchMultiIndexWrapper for clarity and added a few comments
about changes needed for ElasticSearch/Elastica 7.x.

* Rename getType function from ESMultiIndexWrapper

Rename getType from arElasticSearchMultiIndexWrapper to getIndex for
clarity.

* Add dummy type for ES index

Add a dummy ElasticSearch index type since one is still required for
ES 6.x, instead of having a seperate type name for each of the
multiple indices.

* Change references to index type for ElasticSearch

Update any references to ElasticSearch index type to index name.
Also add getType as an alias for getIndex for backwards compatibility
with custom themes that still might be making references to
ElasticSearch's getType.

* Fix copy to _all fields for ES mapping

Removed copy_to _all from all fields that previously had include_in_all
set to false.

* Remove unnecessary ElasticSearch type variables

Remove unnecessary $type ElasticSearch variables in QubitLftSyncer,
arElasticSearchPlugin, and updatePublicationStatusTask

* Remove ES index prefix from MultiIndexWrapper

Remove the index prefix from arElasticSearchMultiIndexWrapper and use
AtoM class names internally instead of using prefix to refer to them.

* Fix autocompleteAction bug, address CR feedback

Fix bug in search autocompleteAction which was showing duplicate results
for actors under repositories. Also address some of the CR feedback and
update/add methods for add, update, and deleteById in the
arElasticSearchMultiIndexWrapper.

* Remove non text fields from _all ES mapping

* Remove unnecessary loop in arElasticSearchPlugin

Remove unnecessary foreach loop in arElasticSearchPlugin. Also move code
for adding filter for stripping md tags into a seperate method to make
the initialization code more readable.

* Update ES index refresh to only refresh one index

Update the index refresh method to accept an optional name param that
only refreshes that particular index in
arElasticSearchMultiIndexWrapper. Also address some small CR feedback.

* Refactor ES initialize and populate methods (WIP)

* Tweak index recreation

* Initialize on populate

* Tweak flush/refresh operation

* Only ignore 404 errors when deleting indexes

* Prevent flush if current index name is not set

* Make loadDiacriticsMappings a private function

* Improve populate logging

* Configure filters once before indices creation

* Normalize loadDiacriticsMappings function

Check setting and update config global variable there.

* Create indices on update if they don't exist

* Trust recreate option on index creation

* Remove unused function

---------

Co-authored-by: melaniekung <mkung@artefactual.com>
Co-authored-by: José Raddaoui Marín <raddaouimarin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problem: ElasticSearch 5.x dependency

3 participants