Elastic6 work in progress#2373

Merged

tonytw1 merged 194 commits intomasterfrom

Jan 29, 2019

Commits on Dec 6, 2018

Rename create index method to make it explicit that this call only makes a specific type of index (one with image mappings).
Tony McCrae
committed
Remove duplicate suggestion mapper.
Tony McCrae
committed

Commits on Dec 7, 2018

Remove unused identifier settings function.
Tony McCrae
committed
Initial port of image index defination to elastic4s DSL.
Tony McCrae
committed
Fixes incorrectly migrated mappings and index settings.
Tony McCrae
committed
Introduce an interface to document the elastic search operations which need to be supported.
Tony McCrae
committed

Commits on Dec 11, 2018

Add a minimal Elastic search 6 test instance on port 9206.
Tony McCrae
committed
Spike. Thrall Syndication ops tests pass with against Elastic 1.7 and 6.4.
Tony McCrae
committed
Refactor; rename.
Tony McCrae
committed
Level up the to elastic4s 6.4.
Tony McCrae
committed
Clean up; deduplicate tests.
Tony McCrae
committed
Moving towards the restoration of the .executeAndLog functionality in the version 6 client.
Tony McCrae
committed
elastic4s generally returns happy failures; remap this into failed futures for consistancy with existing driver.
Tony McCrae
committed
Catch no match.
Tony McCrae
committed

Commits on Dec 12, 2018

Commits on Dec 17, 2018

Elastic6 implementation of update user meta data.
Tony McCrae
committed
Clean up; with tests in place we can now safely reduce this boolean expression.
Tony McCrae
committed
Restoring refresh usages rights scripts.
Tony McCrae
committed
Port photoshoot suggestion script.
Tony McCrae
committed
Coverage for update syndication rights.
Tony McCrae
committed

Commits on Dec 18, 2018

Test robustness; reinstate the wait for green cluster check.
Tony McCrae
committed
Ensure indexes before elastic search test.
Tony McCrae
committed
Elastic6 index needs to be single shade, no replicas in dev/test.
Tony McCrae
committed
Increase timeout for deployment of ES1 and ES6 docker images.
Tony McCrae
committed
Config for ES6 hostname.
Tony McCrae
committed
Clean up; push ES6 index shard and replicas count up to config.
Tony McCrae
committed
Elastic6 delete image implementation.
Tony McCrae
committed
Unroll deprecated count document type argument.
Tony McCrae
committed
Clean up; using count API for delete lookup seems to remove the need to the indexing wait.
Tony McCrae
committed
Delete image usages implementation.
Tony McCrae
committed
Clean up; push test data towards Fixtures.
Tony McCrae
committed

Commits on Dec 19, 2018

Update image usages elastic6 implementation.
Tony McCrae
committed
Delete syndication rights elastic 6 implementation.
Tony McCrae
committed
Elastic6 implementations for add and remove leases.
Tony McCrae
committed
Test waits.
Tony McCrae
committed
Elastic6 implementation of replace leases.
Tony McCrae
committed
Elastic6 implementation of set collections.
Tony McCrae
committed
Elastic6 implementation for add and delete exports.
Tony McCrae
committed
Test wait; pushing into one place ahead of optimisation.
Tony McCrae
committed
Revert commit of local changes
Tony McCrae
committed
Merge branch 'master' into elastic6
tonytw1
authored
Apply elastic search method tests to Elastic 1 and 6 implementations.
Tony McCrae
committed
Extracting an interface to document the elastic search operations which need to be ported to elastic 6.
Tony McCrae
committed
Push down private method.
Tony McCrae
committed
Refactor; move rendering of aggregation responses from elastic to controllers package.
Tony McCrae
committed
Refactor; collecting the model classes needed to talk to Elastic search into a model grouping.
Tony McCrae
committed

Commits on Dec 21, 2018

Commits on Jan 2, 2019

Clean up; consistent mapping style.
Tony McCrae
committed
Remove unused deleteIndex method.
Tony McCrae
committed
Logging; prefix elastic6 operations with ES6.
Tony McCrae
committed
Additonal documenting tests for existing indexImage behaviour.
Tony McCrae
committed
Reinstate indexing wait.
Tony McCrae
committed
Marking TODO for missing redaction of reindex requests test coverage.
Tony McCrae
committed
Extract reindex redaction which Elastic1 applies to non initial index requests and apply equally to Elastic6.
Tony McCrae
committed
Pick up elastic6 hostname from EC2 tags in non dev environments.
Tony McCrae
committed
No defaults for non dev elastic6 replica and shards; these are important enough to force them to be set.
Tony McCrae
committed
Imports.
Tony McCrae
committed
Document our updated understanding of how getCurrentIndices is used.
Tony McCrae
committed

Commits on Jan 3, 2019

Commits on Jan 4, 2019

Refactor - push EC2 host discovery into seperate package; not an elastic concern even if elastic is the only user.
Tony McCrae
committed
Level up to 6.5.4 latest elastic release.
Tony McCrae
committed
Elastic single node discovery flag for 6.5 in dev and test.
Tony McCrae
committed
Merge branch 'elastic6' into elastic6-media-api
Tony McCrae
committed
Refactor - collecting Elasticsearch 1 implementation specific classes into a package.
Tony McCrae
committed
Restor; push the init' of the elastic 1 search filters class down into the Elastic1 constructor.
Tony McCrae
committed
Marking private method to reduce dev cognitive load.
Tony McCrae
committed
WIP Begin porting filters DSL to elastic4s. Rename tests to avoid name clash with thrall in IDE.
Tony McCrae
committed
WIP Navive porting of SearchFilters; filters DSL continues to in insulate us the the underlying elastic version.
Tony McCrae
committed
WIP Restore unresolved WIP
Tony McCrae
committed

Commits on Jan 7, 2019

Commits on Jan 8, 2019

Setting up for implementation of QueryBuilder.
Tony McCrae
committed
Clean up; resolve not method clash between filter and Elastic DSLs.
Tony McCrae
committed
Refactor; move tests to match implementation package structure.
Tony McCrae
committed
Refactor; rename fixtures class.
Tony McCrae
committed
Correct test package name.
Tony McCrae
committed
Back filling QueryBuilder test coverage.
Tony McCrae
committed
Elastic6 QueryBuilder should return a Query equivalent to the Elastic1 implementation. Force the Elastic6 service to accomodate this so that the diffs between the 2 query builder implementations ar…
Tony McCrae
committed
Combine query builder and filter inputs in a manner similar to the ES1 implementation.
Tony McCrae
committed
Helper to render ES6 search request body JSON in tests; use as a sanity check to visually cross check that ES1 and ES6 are sending vaguely simalar requests over the wire.
Tony McCrae
committed
Spike; prove that we can test drive out the ES6 Query builder behaviour by observing the ES1 behaviour and then porting the tests.
Tony McCrae
committed
Port query builder negation behaviour.
Tony McCrae
committed
Port word field match.
Tony McCrae
committed
Port date range clause.
Tony McCrae
committed
Port has field filter.
Tony McCrae
committed
hierarchy field condition.
Tony McCrae
committed
Clean up; println
Tony McCrae
committed

Commits on Jan 9, 2019

Port any field query builder.
Tony McCrae
committed
Implement multiple fields query support in elastic6.
Tony McCrae
committed
Clean up; all make query bit cases implemented so we cab remove the not implemented back stop
Tony McCrae
committed
Clean up; on reading the source code a builder action for type field does exist in elastic4s and is called matchType.
Tony McCrae
committed
Port nested queries support to elastic6 query builder.
Tony McCrae
committed
Remove unused InvalidQuery case class which is superseded by the copy in lib query syntax.
Tony McCrae
committed
Clean ups; visually more aligned with the ES1 implementation.
Tony McCrae
committed
Refactor; extract match fields definition so that it can be tied across ES1 and ES6 implementations.
Tony McCrae
committed
Marking private to sorts fields and methods.
Tony McCrae
committed
WIP pick up pagination and default sort order.
Tony McCrae
committed
elastic4s can take a list of sorts which removes our responsibility for composing the subsorts ourselves.
Tony McCrae
committed
Added to collection date sort special case. Regression - the Elastic1 sort code mutates the query definition at this point. We'll need to pull this special case up.
Tony McCrae
committed
WIP Refactor; kick the date added to collection sort can further down the road by pushing knowledge if it's usage up.
Tony McCrae
committed
Restore elastic1 dateAddedToCollection sort restrictions by applying as an additional filter.
Tony McCrae
committed
Applies thrall style executeAndLog behaviour to the MediaAPI search calls.
Tony McCrae
committed
Image search metrics (time taken as per elastic1).
Tony McCrae
committed
Clean up; elastic6 executions should be bring your own execution context.
Tony McCrae
committed
Implement ES6 getImageById
Tony McCrae
committed
Initial attempt at porting an aggregation related end point to Elastic6.
Tony McCrae
committed
Implement date histogram based end point.
Tony McCrae
committed

Commits on Jan 10, 2019

WIP Test for supplier usages end point.
Tony McCrae
committed
Developer experience; fail Elastic1 tests if the tear down fails.
Tony McCrae
committed
Stable purge test images for ES1 index.
Tony McCrae
committed
Clean up; remove duplication of ES6 execution.
Tony McCrae
committed
Additional asserts to document our understanding of deny leases involvement in blocked syndication status.
Tony McCrae
committed
With some more understanding of what the syndication filter tests are doing we can game our agency test data to stay out of the way.
Tony McCrae
committed
ES6 implementation for usageForSupplier.
Tony McCrae
committed
Naive implementation of suggest completions.
Tony McCrae
committed
Marking observed bug.
Tony McCrae
committed
Elastic6 bug fix; missing mappings.
Tony McCrae
committed
Elastic6 bug fix; missing mappings.
Tony McCrae
committed

Commits on Jan 11, 2019

Commits on Jan 14, 2019

Commits on Jan 15, 2019

Commits on Jan 16, 2019

Commits on Jan 17, 2019

Commits on Jan 18, 2019

Use scan search type as recommended optimisation of Elastic1.7 scroll when order is not important.
Tony McCrae
committed
Build fix; only apply Main.class farjar setting to migration project.
Tony McCrae
committed
Merge branch 'elastic6-migration' into elastic6-media-api
Tony McCrae
committed
Merge branch 'elastic6-media-api' into elastic6
Tony McCrae
committed
Revert SearchType.SCAN optimisation; setting this resulted in loss of controller of the page size. Seemed to be persisting the pagesize (5000) of a previous scroll.
Tony McCrae
committed
Import Elastic1 source rather than serialized image to migrate the fields the Elastic scripts have been populating (ie. suggesters etc).
Tony McCrae
committed
Restore SearchType.Scan optimisation. The page size confusion related to the page size been results returned per shard; hence 5000 was 1000 x 5 shards.
Tony McCrae
committed
Clean up; be more target with the JSON cleaning we're doing on the suggestMetadataCredit field; use a JSON transformer to document it.
Tony McCrae
committed
README
Tony McCrae
committed
Lower maximum pagination limit. UI has a limit of 20k so we can come down to just above that.
Tony McCrae
committed
Clean up; leases.current is marked as unused in the ES1 mappings file; dropping it.
Tony McCrae
committed
Clean up; leases.current is marked as unused in the ES1 mappings so lets not migrate it.
Tony McCrae
committed
Clean up.
Tony McCrae
committed
Logging.
Tony McCrae
committed
Merge branch 'master' into elastic6
Tony McCrae
committed
Tweaking for index speed; replicas = 0 seems like the most impactful change ~ 40%
Tony McCrae
committed

Commits on Jan 21, 2019

Import tweaks.
Tony McCrae
committed
Merge branch 'master' into elastic6
Tony McCrae
committed

Commits on Jan 22, 2019

Merge branch 'master' into elastic6
Tony McCrae
committed

Commits on Jan 24, 2019

Merge branch 'master' into elastic6
Tony McCrae
committed

Commits on Jan 25, 2019

Merge branch 'master' into elastic6
Tony McCrae
committed