Repository navigation
[Swimlane] New integration #13499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[Swimlane] New integration #13499
Changes from all commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
4e0606c
Intial commit of swimlane package
kevinmata92 65cf7f7
Updated CODEOWNERS
kevinmata92 1f32d54
Updated PR in changelog
kevinmata92 1531b01
Update github owner team
kevinmata92 2d61640
Updated ecs version reference
kevinmata92 c43b9e3
Updated policy template name
kevinmata92 c04445d
Create README in correct directory
kevinmata92 25eb126
Deleted and updated format for test events files
kevinmata92 fbdef72
Added sample events for each datastream
kevinmata92 bc7b265
Updated README after running build
kevinmata92 4cbe999
Updated pipelines as recommended
kevinmata92 7cfb1d3
Added extra case for handling multiple / single Ips
kevinmata92 8907aab
Fixed event.type type to list
kevinmata92 67dcb4c
Re-ran build and updated README
kevinmata92 a3991fb
Updated expected json files after running locally
kevinmata92 b90cecb
Add ECS version 8.17.0
kevinmata92 6e8c125
Fixed typos in README and removed unwated info
kevinmata92 7e3ec4b
Added related.user to pipelines
kevinmata92 e7675cc
Fixed title for swimlane_api and tenant_api
kevinmata92 324ab57
Bump version to 0.1.0
kevinmata92 81d7918
Remove dynamic dataset and namespace
kevinmata92 b00c627
Cleaned pipeline and remapped tenant and account id
kevinmata92 0dc36af
Cleaned tenant_api pipeline and generated sample event
kevinmata92 a7bbec0
Cleaned turbine_api pipeline and generated sample event
kevinmata92 f41692f
Updated base fields for cloud.origin.project.id
kevinmata92 3c5d9c1
Fix spacing in swimlane.api pipeline
kevinmata92 16aa78d
Fixed spacing in tenant_api pipeline
kevinmata92 38e195f
Fixed spacing in turbine_api pipeline
kevinmata92 4d822a1
Remove new lines in swimlane_api pipeline
kevinmata92 2a39343
New lines in sample event
kevinmata92 5a574c8
Remove unnecessary values from manifest
kevinmata92 1932b77
Adding system test for swimlane-api
kevinmata92 07681b2
Change to k8s format
kevinmata92 8c0d32b
Updated pipeline after running system test
kevinmata92 e2f1e0e
Updated base fields to include user_agent.original
kevinmata92 89f192b
Updated pipeline to remove fields which are not needed
kevinmata92 6442d80
Added remaining system test and updated sample_events and README
kevinmata92 ad563ca
Add new line character for test-events.logs
kevinmata92 8f138b8
Add new line character for tests
kevinmata92 9eb1ed6
Updated pipeline formatting and use null-safe operators
kevinmata92 d708705
Updated description and added link
kevinmata92 6c6131e
Updated pipelines and removed null safe for ctx
kevinmata92 81f9602
Added additional log to swimlane_api to test the message portion
kevinmata92 f4e0681
Remove extra new line in test configs
kevinmata92 21a85b3
Updated pipeline formatting
kevinmata92 fe450a9
fix final new-lines
efd6 ebb8b23
build package
efd6 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| dependencies: | ||
| ecs: | ||
| reference: git@v8.17.0 | ||
| import_mappings: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # Swimlane Turbine | ||
|
|
||
| The [Swimlane Turbine](https://swimlane.com/swimlane-turbine/) integration allows you to ingest on-prem audit logs from Swimlane Turbine, the Enterprise AI Hyperautomation & Orchestration Platform. | ||
|
|
||
| Use the Swimlane Turbine integration to stream container pod logs into your Elastic deployment. | ||
|
|
||
| ## Data streams | ||
| The Swimlane Turbine integration collects one type of data streams: logs. | ||
|
|
||
| ### swimlane.api | ||
| Swimlane API help logs keep a record of events happening in Swimlane API which are related to Workspaces, Dashboards, Reports, Application, Applets, Records, Role Based Access Control (RBAC). | ||
| All fields ingested to this data stream are stored under `swimlane.api` as an event. | ||
|
|
||
| ### swimlane.tenant | ||
| Tenant API help logs keep a record of events happening in Tenant API which are related to Account & Tenant Management, Settings, and Authentication. | ||
| All fields ingested to this data stream are stored under `swimlane.tenant` as an event. | ||
|
|
||
| ### turbine.api | ||
| Turbine API help logs keep a record of events happening in Turbine API which are related to Connectors, Assets, Sensors, Solutions, Playbook, Schema Definitions, and Components. | ||
| All fields ingested to this data stream are stored under `turbine.api` as an event. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### For Turbie Platform Installs (TPI) | ||
| TPI settings can be configured in the administrator dashboard as seen below: | ||
|
|
||
|  | ||
|
|
||
| ### For Helm or Kustomize Installs | ||
| The following environment variables will need to be set for Audit logs to be outputted into the container pod logs. | ||
|
|
||
| ``` | ||
| swimlane-api & swimlane-tenant: | ||
| "SWIMLANE_Logging__Level=Info" | ||
| "SWIMLANE_Logging__IncludeAudit=true" | ||
|
|
||
| turbine-api: | ||
| "LOG_LEVEL_API=info" | ||
| "LOG_LEVEL_DEFAULT=info" | ||
| "LOG_LEVEL_SYSTEM=info" | ||
| "LOG_FILES_ENABLED=false" | ||
| ``` | ||
| ## Data Stream | ||
|
|
||
| ### swimlane.api | ||
|
|
||
| {{event "swimlane_api"}} | ||
|
|
||
| {{fields "swimlane_api"}} | ||
|
|
||
| ### swimlane.tenant | ||
|
kevinmata92 marked this conversation as resolved.
|
||
|
|
||
| {{event "tenant_api"}} | ||
|
|
||
| {{fields "tenant_api"}}. | ||
|
|
||
| ### turbine.api | ||
|
|
||
| {{event "turbine_api"}} | ||
|
|
||
| {{fields "turbine_api"}} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| services: | ||
| filestream-swimlane-api: | ||
| image: alpine | ||
| volumes: | ||
| - ./sample_logs:/sample_logs:ro | ||
| - ${SERVICE_LOGS_DIR}:/var/log | ||
| command: /bin/sh -c "cp /sample_logs/* /var/log/" | ||
|
|
||
| filestream-tenant-api: | ||
| image: alpine | ||
| volumes: | ||
| - ./sample_logs:/sample_logs:ro | ||
| - ${SERVICE_LOGS_DIR}:/var/log | ||
| command: /bin/sh -c "cp /sample_logs/* /var/log/" | ||
|
|
||
| filestream-turbine-api: | ||
| image: alpine | ||
| volumes: | ||
| - ./sample_logs:/sample_logs:ro | ||
| - ${SERVICE_LOGS_DIR}:/var/log | ||
| command: /bin/sh -c "cp /sample_logs/* /var/log/" | ||
|
|
1 change: 1 addition & 0 deletions
1
packages/swimlane/_dev/deploy/docker/sample_logs/test-swimlane-api.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 2025-04-10T19:26:36.081039081Z stdout F { "EventTime": "2025-04-10T19:26:36.0810390Z", "User": "admin@domain.tld", "UserId": "1af3bd48-d46e-490f-c015-004d198d0558", "Category": "Settings", "LogSource": "api", "LogType": "Audit", "Description": "admin@domain.tld created a new permission", "AccountId": "5fbee706-0909-4t1a-ada7-3e8e2a1f3117", "TenantId": "5941becf-5ac4-493e-97eb-50da36f80582", "SourceIp": "::ffff:81.2.69.144", "UserAgent": "Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/135.0.0.0 Safari\/537.36", "ActionType": "Create", "Id": "a7SAfL4_XghbfVNQm", "NewValue": "{\"Permission\":{\"Access\":12743,\"Id\":\"a6VKjVcgS7I5a3sl0\",\"Fields\":{}},\"Source\":{\"Type\":\"Role\",\"Id\":\"0196201d-ba14-76a3-9a0c-2fbf1510f458\",\"Name\":\"Tier-2 IR Specialist\",\"Disabled\":false},\"Target\":{\"Type\":\"Application\",\"Id\":\"aK_JIlwxET4gA7RWN\",\"Name\":\"Swimlane System of Record\",\"Disabled\":false},\"Id\":\"a7SAfL4_XghbfVNQm\",\"Name\":null,\"Uid\":null,\"Disabled\":false}", "EventOutcome": "Success", "Endpoint": "\/app\/aK_JIlwxET4gA7RWN", "IsAdmin": "True", "AuthenticationType": "JWT" } |
1 change: 1 addition & 0 deletions
1
packages/swimlane/_dev/deploy/docker/sample_logs/test-tenant-api.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 2025-04-09T22:09:49.893647581Z stdout F {"EventTime": "2025-04-09T22:09:49.89364Z", "User": "admin@domain.tld", "UserId": "1af3bd48-d46e-490f-c015-004d198d0558", "Category": "Login", "LogSource": "Tenant Service", "LogType": "Audit", "LoggedInUserId": "1af3bd48-d46e-490f-c015-004d198d0558", "Description": "Failed login attempt registered for the swimlane user admin@domain.tld, current failed login attempts:1", "RequestEndPoint": "://", "SourceIp": "81.2.69.144, 10.42.134.224", "EndPoint": "turbine.domain.tld/api/users/login", "UserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36", "IsAdmin": "False", "ActionType": "Login", "EventOutcome": "Failure", "LogLevel": "INFO"} |
1 change: 1 addition & 0 deletions
1
packages/swimlane/_dev/deploy/docker/sample_logs/test-turbine-api.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 2025-04-10T20:32:26.677647581Z stdout F {"level": "info", "time": "2025-04-10T20:32:26.677Z", "instance": "api-turbine-api-55674d58c-xphlb-d56157c8-db02-4c48-a6a7-02c1abf40598", "host": "turbine-api-55674d58c-xphlb", "service": "turbine-api", "version": "25.0.7 (gh-readonly-queue/release/25.0/pr-9272-d5359d7a7f3aa093359ca89a42a5c009fd000b17:b6f5d273)", "LogType": "Audit", "EventTime": "2025-04-10T20:32:26.677Z", "TenantId": "5941becf-5ac4-493e-97eb-50da36e80582", "AccountId": "5fbee706-0909-4f1a-ada7-3e8e2a1f3117", "User": "admin@domain.tld", "UserId": "1af3bd48-d46e-490f-b015-004c198d0558", "Category": "Asset", "Description": "admin@domain.tld Created asset 67f82ada9deafb2d6ec46987", "ActionType": "Create", "Id": "67f82ada9deafb2d6ec46987", "NewValue": "{\"name\":\"ECK\",\"title\":\"ECK\",\"description\":\"\",\"testingEnabled\":false,\"interval\":15,\"poolId\":\"67f702c8921c9ee6262ed50a\",\"status\":\"inactive\",\"docSchemaVersion\":1,\"connectorAsset\":\"elasticsearch.http_basic\",\"params\":{\"url\":\"https://eck.domain.tld\",\"username\":\"turbine-audit\",\"password\":\"**********\"},\"paramSchema\":{\"type\":\"object\",\"properties\":{\"url\":{\"title\":\"URL\",\"description\":\"A URL to the target host.\",\"type\":\"string\"},\"username\":{\"title\":\"Username\",\"description\":\"Username\",\"type\":\"string\"},\"password\":{\"title\":\"Password\",\"description\":\"Password\",\"type\":\"string\",\"format\":\"password\"},\"verify_ssl\":{\"title\":\"Verify SSL Certificates\",\"description\":\"Verify SSL certificate\",\"type\":\"boolean\"},\"http_proxy\":{\"title\":\"HTTP(s) Proxy\",\"description\":\"A proxy to route requests through.\",\"type\":\"string\"}},\"required\":[\"url\",\"username\",\"password\"]},\"testParams\":{},\"testParamSchema\":{\"type\":\"object\",\"properties\":{\"url\":{\"title\":\"URL\",\"description\":\"A URL to the target host.\",\"type\":\"string\"},\"username\":{\"title\":\"Username\",\"description\":\"Username\",\"type\":\"string\"},\"password\":{\"title\":\"Password\",\"description\":\"Password\",\"type\":\"string\",\"format\":\"password\"},\"verify_ssl\":{\"title\":\"Verify SSL Certificates\",\"description\":\"Verify SSL certificate\",\"type\":\"boolean\"},\"http_proxy\":{\"title\":\"HTTP(s) Proxy\",\"description\":\"A proxy to route requests through.\",\"type\":\"string\"}},\"required\":[\"url\",\"username\",\"password\"]},\"isCustom\":false,\"meta\":{\"sharingUid\":\"2a6cedfe-2818-4a69-89f8-c6c1853fa433\"},\"audit\":{\"version\":1,\"created\":{\"date\":\"2025-04-10T20:32:26.533Z\",\"user\":{\"id\":\"1af3bd48-d46e-490f-b015-004c198d0558\",\"username\":\"admin@domain.tld\"},\"authProvider\":{\"id\":\"\",\"title\":\"Swimlane\"}},\"modified\":{\"date\":\"2025-04-10T20:32:26.533Z\",\"user\":{\"id\":\"1af3bd48-d46e-490f-b015-004c198d0558\",\"username\":\"admin@domain.tld\"},\"authProvider\":{\"id\":\"\",\"title\":\"Swimlane\"}}},\"id\":\"67f82ada9deafb2d6ec46987\"}", "SourceIp": [ "81.2.69.144" ], "UserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36", "EventOutcome": "Success", "Endpoint": "/v1/asset", "Referer": "https://turbine.domain.tld/account/5fbee706-0909-4f1a-ada7-3e8e2a1f3117/tenant/5941becf-5ac4-493e-97eb-50da36e80582/canvas/assets", "IsAdmin": true, "AuthenticationType": "JWT"} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # newer versions go on top | ||
| - version: "0.1.0" | ||
| changes: | ||
| - description: Initial draft of the package | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/13499 |
2 changes: 2 additions & 0 deletions
2
packages/swimlane/data_stream/swimlane_api/_dev/test/pipeline/test-events.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| { "EventTime": "2025-04-10T19:26:36.0810390Z", "User": "admin@domain.tld", "UserId": "1af3bd48-d46e-490f-c015-004d198d0558", "Category": "Settings", "LogSource": "api", "LogType": "Audit", "Description": "admin@domain.tld created a new permission", "AccountId": "5fbee706-0909-4t1a-ada7-3e8e2a1f3117", "TenantId": "5941becf-5ac4-493e-97eb-50da36f80582", "SourceIp": "::ffff:81.2.69.144", "UserAgent": "Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/135.0.0.0 Safari\/537.36", "ActionType": "Create", "Id": "a7SAfL4_XghbfVNQm", "NewValue": "{\"Permission\":{\"Access\":12743,\"Id\":\"a6VKjVcgS7I5a3sl0\",\"Fields\":{}},\"Source\":{\"Type\":\"Role\",\"Id\":\"0196201d-ba14-76a3-9a0c-2fbf1510f458\",\"Name\":\"Tier-2 IR Specialist\",\"Disabled\":false},\"Target\":{\"Type\":\"Application\",\"Id\":\"aK_JIlwxET4gA7RWN\",\"Name\":\"Swimlane System of Record\",\"Disabled\":false},\"Id\":\"a7SAfL4_XghbfVNQm\",\"Name\":null,\"Uid\":null,\"Disabled\":false}", "EventOutcome": "Success", "Endpoint": "\/app\/aK_JIlwxET4gA7RWN", "IsAdmin": "True", "AuthenticationType": "JWT" } | ||
| { "EventTime": "2025-05-01T00:06:25.3472006Z", "User": "admin@domain.tld", "UserId": "6b76c4cc-cb91-401d-807a-096e51bda097", "Category": "Applications", "LogSource": "api", "LogType": "Audit", "Description": "admin@domain.tld updated application aDSkpLIhPw7BJgraj", "AccountId": "8b1cb48d-bfd6-4a4c-8e87-703555319925", "TenantId": "231d2ae4-f78e-4c98-be5f-1cd2f4807e0b", "SourceIp": "::ffff:81.2.69.144", "UserAgent": "Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/135.0.0.0 Safari\/537.36", "ActionType": "Update", "Id": "aDSkpLIhPw7BJgraj", "NewValue": "{\"Description\":null,\"Acronym\":\"KTA\",\"TrackingFieldId\":\"680ac03329077207d515802b\",\"Layout\":[{\"Id\":\"6811680f479a681095973524\",\"ParentId\":null,\"Row\":1,\"Col\":1,\"Sizex\":4,\"Sizey\":0,\"LayoutType\":1},{\"Id\":\"6811696fc0a3487ab8f0ab6f\",\"ParentId\":null,\"Row\":1,\"Col\":2,\"Sizex\":4,\"Sizey\":0,\"LayoutType\":1},{\"Id\":\"6812bafec66e6c56bee451fe\",\"ParentId\":null,\"Row\":1,\"Col\":4,\"Sizex\":4,\"Sizey\":0,\"LayoutType\":1}],\"Fields\":[{\"Id\":\"a9dyt\",\"Name\":\"Text\",\"Description\":null,\"Key\":\"text\",\"SourceAppletFieldId\":null,\"SourceAppletId\":null,\"FieldType\":1,\"Required\":false,\"ReadOnly\":false,\"SupportsMultipleOutputMappings\":false,\"VisibleToHeroAi\":false},{\"Id\":\"a01tt\",\"Name\":\"Text (2)\",\"Description\":null,\"Key\":\"text-2\",\"SourceAppletFieldId\":null,\"SourceAppletId\":null,\"FieldType\":1,\"Required\":false,\"ReadOnly\":false,\"SupportsMultipleOutputMappings\":false,\"VisibleToHeroAi\":false},{\"Id\":\"av25h\",\"Name\":\"Text (3)\",\"Description\":null,\"Key\":\"text-3\",\"SourceAppletFieldId\":null,\"SourceAppletId\":null,\"FieldType\":1,\"Required\":false,\"ReadOnly\":false,\"SupportsMultipleOutputMappings\":false,\"VisibleToHeroAi\":false},{\"Id\":\"680ac03329077207d515802b\",\"Name\":\"Tracking Id\",\"Description\":\"Tracking Id\",\"Key\":\"tracking-id\",\"SourceAppletFieldId\":null,\"SourceAppletId\":null,\"FieldType\":7,\"Required\":null,\"ReadOnly\":true,\"SupportsMultipleOutputMappings\":false,\"VisibleToHeroAi\":false}],\"MaxTrackingId\":0,\"Workspaces\":[],\"CreateWorkspace\":false,\"CreatedDate\":\"2025-04-24T22:50:27.109Z\",\"CreatedByUser\":{\"Id\":\"6b76c4cc-cb91-401d-807a-096e51bda097\",\"Name\":\"admin@domain.tld\"},\"ModifiedDate\":\"2025-05-01T00:06:25.3119561Z\",\"ModifiedByUser\":{\"Id\":\"6b76c4cc-cb91-401d-807a-096e51bda097\",\"Name\":\"admin@domain.tld\"},\"TimeSpentFieldId\":null,\"TimeTrackingEnabled\":false,\"VisibleToHeroAiEnabled\":false,\"Permissions\":{},\"SelectionFields\":{},\"Uid\":\"test-app-145fa\",\"Version\":4,\"Id\":\"aDSkpLIhPw7BJgraj\",\"Name\":\"Test App\",\"Disabled\":false}", "EventOutcome": "Success", "Endpoint": "\/app\/aDSkpLIhPw7BJgraj", "IsAdmin": "True", "AuthenticationType": "JWT" } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.