Skip to content

[Connector API] Support soft-deletes of connectors - #118669

Merged
jedrazb merged 38 commits into
elastic:mainfrom
jedrazb:support-soft-deletes-connectors-change-mapping
Jan 9, 2025
Merged

jedrazb merged 38 commits into
elastic:mainfrom
jedrazb:support-soft-deletes-connectors-change-mapping

Conversation

@jedrazb

@jedrazb jedrazb commented Dec 13, 2024 •

Copy link
Copy Markdown
Contributor

Soft deletes for connectors

Add support for soft-deletes of connectors. Why?

  • In order to improve UX on connectors on agentless we need to track what connectors were deleted vs. what connector records are not yet created

Changes

  • introduce deleted flag in the connector index mappings, indicates whether connector has been soft deleted
  • adapt delete, get and list operations logic to support this feature
    • if we pass include_deleted=true flag we return also deleted connectors in the response
  • update docs
  • add yaml tests and unit tests

Related work

  • In a follow-up PR I will address adding index mapping migrations for connector index. I'm coordinating here with @navarone-feekery as he might be adding SystemIndexDecriptor for connector indices soon, this will help updating mappings a lot as there is SystemIndexMappingUpdateService
  • follow-up: support force deletes, e.g pass force=true to delete endpoint to completely remove the connector

@github-actions

Copy link
Copy Markdown
Contributor

Documentation preview:

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @jedrazb, I've created a changelog YAML for you.

@jedrazb

jedrazb commented Dec 17, 2024

Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

@jedrazb
jedrazb marked this pull request as ready for review December 17, 2024 16:02

@navarone-feekery navarone-feekery left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jedrazb

jedrazb commented Dec 19, 2024

Copy link
Copy Markdown
Contributor Author

can a new Connector have an index_name value identical to a soft-deleted Connector

That was the single edge case I didn't cover because we have custom check (with query) for index names already in use ... fixed in 7525c99

@timgrein timgrein left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments, but I think nothing preventing this PR from being merged

Comment thread docs/reference/connector/apis/get-connector-api.asciidoc Outdated
Comment thread rest-api-spec/src/main/resources/rest-api-spec/api/connector.list.json Outdated
@jedrazb

jedrazb commented Dec 30, 2024

Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

Comment thread docs/reference/connector/apis/delete-connector-api.asciidoc
@jedrazb

jedrazb commented Jan 8, 2025

Copy link
Copy Markdown
Contributor Author

New changes:

  • remove serialisation by wire for Connector class and get/list actions (since it's not needed) same as in Make the Get API keys API local only #105497
  • rename REST param deleted to include_deleted in get/list actions as IMO it conveys its purpose better

@jedrazb

jedrazb commented Jan 9, 2025

Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

@navarone-feekery navarone-feekery left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a few questions, nothing blocking.

@jedrazb

jedrazb commented Jan 9, 2025

Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

@elasticmachine

Copy link
Copy Markdown
Collaborator

merge conflict between base and head

…ub.com:jedrazb/elasticsearch into support-soft-deletes-connectors-change-mapping
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>feature :SearchOrg/Extract&Transform Label for the Search E&T team Team:Search - Extract & Transform Team:SearchOrg Meta label for the Search Org (Enterprise Search) v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants