Repository navigation
[ti_crowdstrike] Add Support of Deep Pagination in Intel Data Stream - #9200
Merged
Merged
Conversation
Update the changelog entry for the version 0.3.0 too.
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
Contributor
|
/test |
1 similar comment
Contributor
|
/test |
efd6
reviewed
Feb 20, 2024
Contributor
|
Failure appears to be a builder issue. |
Change the want_more condition, removed as. Run system test.
Contributor
|
/test |
|
🚀 Benchmarks reportTo see the full report comment with |
💚 Build Succeeded
History
|
efd6
approved these changes
Feb 21, 2024
|
Package ti_crowdstrike - 0.4.0 containing this change is available at https://epr.elastic.co/search?package=ti_crowdstrike |
gizas
pushed a commit
that referenced
this pull request
Mar 13, 2024
…9200) Currently, the CrowdStrike Intel API limits the pagination to fetch the data up to 50,000 records only. If the sum of the offset and limit exceeds the value of 50,000 then the API throws an error. Using Deep Pagination, we can fetch all the data through the Intel Data Stream. So, modify the data collection logic to perform pagination through the _marker in the filter instead of the offset and last_updated to overcome the limit of Intel API. Also fix the pull request URL in the changelog version for 0.3.0.
qcorporation
pushed a commit
that referenced
this pull request
Feb 3, 2025
…9200) Currently, the CrowdStrike Intel API limits the pagination to fetch the data up to 50,000 records only. If the sum of the offset and limit exceeds the value of 50,000 then the API throws an error. Using Deep Pagination, we can fetch all the data through the Intel Data Stream. So, modify the data collection logic to perform pagination through the _marker in the filter instead of the offset and last_updated to overcome the limit of Intel API. Also fix the pull request URL in the changelog version for 0.3.0.
qcorporation
pushed a commit
that referenced
this pull request
Feb 4, 2025
…9200) Currently, the CrowdStrike Intel API limits the pagination to fetch the data up to 50,000 records only. If the sum of the offset and limit exceeds the value of 50,000 then the API throws an error. Using Deep Pagination, we can fetch all the data through the Intel Data Stream. So, modify the data collection logic to perform pagination through the _marker in the filter instead of the offset and last_updated to overcome the limit of Intel API. Also fix the pull request URL in the changelog version for 0.3.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type of change
What does this PR do?
Add Support of Deep Pagination in Intel Data Stream
Currently, the CrowdStrike Intel API limits the pagination to fetch the data upto 50,000 records only. If the sum of the offset and limit exceeds the value of 50000 then the API throws an error.
Using Deep Pagination, we can fetch all the data through the Intel Data Stream. Hence, Modified the data collection logic to perform pagination through the
_markerin the filter instead of theoffsetandlast_udpatedto overcome the limit of Intel API.A quick fix- Add the pull request URL in the changelog version for 0.3.0.
Checklist
changelog.ymlfile.All changes
How to test this PR locally
Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/ti_crowdstrike directory.
Run the following command to run tests.
elastic-package test -vRelated issues
Automated Test
test-ti_crowdstrike.log