…e & project level config (#226)

* feat: read namespace config in replay

* feat: add unit test, export into function

* fix: lint

* feat: also add project-level config & do not mutate config in param

* feat: add map of supported plugins

* feat: add to optimus.sample.yaml

* feat: modify logic & update tests

* refactor: remove map usage & use config server instead

* fix: lint

* feat: add to config.sample.yaml

* feat: add changelog

* fix: integration tests

* fix: added test cases for myers diff

* fix: add metrics of errors for better observability

* fix: lint

* feat: add job deploy command

* chore: fix lint issue

* refactor: remove unused code in job deploy command

* refactor: read specs at once in job deploy

* feat: add keepalive enforcement policy 2m on server side

* fix: linter magic number on config keepalive enforcement policy

* feat: get changelog API

* fix: lint

* fix: undo migration change

* fix: refactor to create a new changelog service

* fix: update proton

* feat: draft job plan command

* feat: add command dif and tree list

* feat: add job plan command

* feat: add command plan for resource

* feat: update plan command description

* feat: add executed on plan object

* feat: refactor git api name and support personal token

* feat: unified git diff get directories

* feat: change order csv kind at first

* feat: removed unused git API

* feat: update message for output file on plan command

* feat: removed unused attribute of git

* feat: support github for plan command

* refactor: move git from ext to client extension provider

* feat: add some information related to abstraction contract

* feat: add unit test on provider git api

* feat: handle when git response is nil

* feat: update message log for plan subcommand

* feat: plan for migrate command

* feat: append additional plan update for every migrate operation

* feat: update file creation and format

* fix: return error missing on open file

* feat: plan operation json serialization

* feat: add composite plan for handle migrate plan

* chore: add comment and update verbose log on plan command

* refactor: update plan coordinator to compositor

* feat: update compositor.GetAll as Merge result

* feat: removing plan sorting and simplify Plan Operation

* chore: remove unused go mod

* refactor: remove compositor from plan and move to plans

* chore: update method name from merge to specific merge migrate operation

* refactor: job and resource plan generation

* feat: add logging on verbose plan

* fix: wording on source and target git diff plan

* feat: change destination to target and update get spec error msg

* feat: append plan if exist and improve source target definition

* feat: check project name use method

* feat: add unit test

* refactor: logic append plan on save file

* feat: add get all namespaces on plan class

* refactor: update list by namespace

* chore: remove unrelated changes of diff plan

* refactor: update git related flag not support short flag

* refactor: update command message on source and target plan

* fix: logging plan on verbose flag is not working

* feat: loop directory based on value instead of index

* feat: enhance check job directory

* feat: add unit test on plan directory function

* fix: update package unit test using suffix _test

* feat: update var twice and returned plan result directly

---------

Co-authored-by: Dery Rahman Ahaddienata <dery.ra@gmail.com>

* feat: draft job plan command

* feat: add command dif and tree list

* feat: add job plan command

* feat: add command plan for resource

* feat: update plan command description

* feat: add executed on plan object

* feat: refactor git api name and support personal token

* feat: unified git diff get directories

* feat: change order csv kind at first

* feat: removed unused git API

* feat: update message for output file on plan command

* feat: removed unused attribute of git

* feat: support github for plan command

* refactor: move git from ext to client extension provider

* feat: add some information related to abstraction contract

* feat: add unit test on provider git api

* feat: handle when git response is nil

* feat: update message log for plan subcommand

* feat: plan for migrate command

* feat: append additional plan update for every migrate operation

* feat: update file creation and format

* fix: return error missing on open file

* feat: plan operation json serialization

* feat: add composite plan for handle migrate plan

* chore: add comment and update verbose log on plan command

* refactor: update plan coordinator to compositor

* feat: update compositor.GetAll as Merge result

* feat: removing plan sorting and simplify Plan Operation

* chore: remove unused go mod

* refactor: remove compositor from plan and move to plans

* chore: update method name from merge to specific merge migrate operation

* refactor: job and resource plan generation

* feat: add logging on verbose plan

* fix: wording on source and target git diff plan

* feat: change destination to target and update get spec error msg

* feat: append plan if exist and improve source target definition

* feat: check project name use method

* feat: add unit test

* refactor: logic append plan on save file

* feat: add get all namespaces on plan class

* feat: add job validation using plan

* refactor: update list by namespace

* fix: condition invalidPlanFlag

* refactor: update method name on job validate command

* chore: remove unrelated changes of diff plan

* refactor: update git related flag not support short flag

* refactor: update command message on source and target plan

* feat: add job validation namespace on normal flow

* feat: draft job plan command

* feat: add command dif and tree list

* feat: add job plan command

* feat: add command plan for resource

* feat: update plan command description

* feat: add executed on plan object

* feat: refactor git api name and support personal token

* feat: unified git diff get directories

* feat: change order csv kind at first

* feat: removed unused git API

* feat: update message for output file on plan command

* feat: removed unused attribute of git

* feat: support github for plan command

* refactor: move git from ext to client extension provider

* feat: add some information related to abstraction contract

* feat: add unit test on provider git api

* feat: handle when git response is nil

* feat: update message log for plan subcommand

* feat: plan for migrate command

* feat: append additional plan update for every migrate operation

* feat: update file creation and format

* fix: return error missing on open file

* feat: plan operation json serialization

* feat: add composite plan for handle migrate plan

* chore: add comment and update verbose log on plan command

* refactor: update plan coordinator to compositor

* feat: update compositor.GetAll as Merge result

* feat: removing plan sorting and simplify Plan Operation

* chore: remove unused go mod

* refactor: remove compositor from plan and move to plans

* chore: update method name from merge to specific merge migrate operation

* refactor: job and resource plan generation

* feat: add logging on verbose plan

* fix: wording on source and target git diff plan

* feat: change destination to target and update get spec error msg

* feat: append plan if exist and improve source target definition

* feat: check project name use method

* feat: add unit test

* refactor: logic append plan on save file

* feat: add get all namespaces on plan class

* refactor: update list by namespace

* chore: remove unrelated changes of diff plan

* refactor: update git related flag not support short flag

* refactor: update command message on source and target plan

* fix: logging plan on verbose flag is not working

* fix: lint defer function not deferred

---------

Co-authored-by: Dery Rahman Ahaddienata <dery.ra@gmail.com>

* feat: add apply command

* feat: implement add job specifications with response job name successes

* feat: implement cache on reading spec by name

* feat: checkout plan from branch feat-diff-plan

* feat: execute apply on job and resource operations

* fix: lint

* feat: add internal/plan from plan branch

* feat: use json reading instead of csv

* feat: returning success status on change namespace job + resource

* feat: add job + resource migrate command

* refactor: get proto requests

* feat: cherry-pick from plan feature

* feat: add apply command on cmd root

* feat: returning job success on update operations

* feat: add log for every execution

* fix: lint

* feat: add log for fail and success operations

* revamp: log summary based on namespace

* fix: specs reading cache based on path

* fix: resource operations with kind name split

* fix: spec test and linter

* fix: return status for resource add, update and migrate

* remove: unused plan operations

* feat: cherry-pick new plan class to handle new contract

* feat: modify apply command to use new plan contract

* fix: lint

* fix: reading from plan and write the summary

* refactor: remove success on create resource response + refactor success bulk get names + rafactor plan apply format

* refactor: print apply logs

* fix: save executed job plan

* feat: remove unused success returned

* refactor: rename field to successful_job_names

* fix: lint

* refactor: remove unecessary success returm

* fix : grant monitoring role select access on change-log

* feat: support routine bigquery on job validation

* feat: add unit test routine handle

* feat: change init mock BigQuery Routine

* feat: update lint on routine

* feat: add more test case and update KindFunction to KindRoutine

* feat: add negative test case

* fix: unit test name

* fix: unit test error assertion after EntityRoutine value changes

* feat: add job upsert in job repository

* test: add cases in job repository upsert

* feat: add job upsert in job service

* fix: issue when differentiating spec when upserting

* Revert "test: add cases in job repository upsert"

This reverts commit 43f1c3c.

* Revert "feat: add job upsert in job repository"

This reverts commit 6dc7bb6.

* feat: add upsert job specifications in job handler

* refactor: use upsert in job deploy command

* refactor: reword log messages in job upsert

* feat: add batch size in job deploy command

* chore: update to latest proto

* chore: update to latest proto with unary changes

* refactor: add successfull job names in job upsert response

* fix: include job name information in error message when fail to generate jobs

* fix: lint issue in job service

* fix: wrong spec mentioned in log in job service parallel runner

* fix: should not log job creation/update in upsert, if there is 0 affected job

* chore: remove unnecessary server logging for catchup in job add and upsert

* fix: avoid changelog insertion if there is no change

* refactor: checking spec difference on job upsert

* test: fix assertion to handle intermittent test failures in job upsert

* refactor: break down job service interface

* refactor: return job names as part of job upsert response

* chore: update proton commit

* feat: add upsert resource in resource service

* feat: add upsert resource in resource handler

* feat: accept multiple resources in upsert resource api

* feat: add resource upload command

* feat: add error message to the upsert response

* fix: return error when failure found in resource upload command

* chore: update latest proto

* chore: update to latest proto with unary changes

* test: fix broken test cases in resource upsert

* refactor: reuse resource upsert & diff detection logic from deploy, in new upsert api

* chore: reword resource upload command examples

* feat: add successful resource names in resource upsert response

* chore: update proton commit

* fix: skip plan migrate on same namespace

* feat: improve handle for multiple namespace same name

* feat: support resource migration based on datastore

* fix: update unit test name based on assertion

* feat: auto dependency capability when destination is changed

* refactor: appending job to be resolved

* test: add test when add new upstream

* test: add test when update upstream

* feat: suport downstream auto deps on upsert api

* fix: support downstream autodeps on replace all

* fix: test on replace all

* feat: deduplicate jobs before resolve

* test: add testcase fror deduplicate + resolve deps

* test: fix ordering issue on test

* refactor: only show direct downstream when deleting jobs

* refactor: add 2 case unit test

* refactor: use list of list of downstreams

* refactor: skip lint

* feat: add resource changelog API service & repo

* feat: unit test for service

* feat: move to a separate service

* feat: add handler & proto

* feat: reorganize functions & track resource updates

* feat: fix metric counter & fix job changelog query

* feat: skip when there's no changes

* feat: update queries structure

* feat: fix query & handler function name

* fix: handler naming

* feat: update on code review

* feat: remove sorting from get query

* fix: lint

* feat: update proton commit

* fix: duplicate unique constraint issue on job deployment

* chore: fix lint issue in job service test

* fix: re-create deleted resource fetch

* feat: passing onlyActive param as const

…255)

* feat: add service logic

* feat: integrate bulk delete to client

* feat: update job service logic & handler

* feat: update proton

* feat: fix client

* feat: fix test

* feat: refactor approach

* feat: restructure downstream deletion

* feat: fix after sync with master

* feat: fix tracker already exists

* feat: unit test

* feat: fix variable naming & remove excessive logs

* feat: update logic to only use deletionTrakerMap

* fix: intermittent unit test fix

* fix: handling empty job

* feat: update proton commit

* fix: add logical date validation for replay

* fix: add a seperate duration extimator for Auto SLA duration calculations

* fix: tests

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* feat: make severity configurable from request

* feat: make percentile configurable

* fix: linter

* fix: linter

* fix: config load teest

* fix : alerts logger bug fix

* fix: typo

* fix: cleanup

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* fix: query

* refactor: segregate task and hook

* refactor: query to get task duration and hook duration

* refactor: make the hook filter as array

* fix: multiple hooks duration

* fix: tests

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* fix: add debug logs for auto sla

* fix: fetch task and hook when operators not set

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>
Co-authored-by: Dery Rahman Ahaddienata <dery.ra@gmail.com>

* fix: buffer duration calculation algorithm

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* return 404 on GetJobSpecification job not found

* remove not found handling on if

* feat: store potential sla breach + add dampering coefficient

* refactor: remove prev scheduled range

* feat: reference time to calculate estimated duration

* feat: use range in hours

* refactor: remove unecessary log

* feat: update proton commit

…ons (#488)

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* fix: use bfs iterative for pruneLineage

* move pruning function for job lineage summary

* refactor: move prunelineage to joblineagesummary, add tests

* fix limit

* add more unit test for cases

* fix: prev scheduled calculation within range in hours

* test: add test cases for prev schedule and next schedule

* test: fix test cases

* feat: add retry on client run input

* feat: put retry when calling the api

* feat: multi team alert

* refactor: support multi team alert

* feat: support multiple notify

* feat: add sensor support for schedule changes

* add test & fix code

* also fetch older runs on schedule change

* fix linter

* split get old runs schedule into multiple functions

* restructure

* add indexing on changelog project name, name, and created_at

* fix structure for lint

* restructure for early return

* fix: use correct alert in alertManager

* use local reference in goroutine

* feat: validate job with upstreams schedule

* only validate for more than daily cron, and use jobs in request if any

* move validation config parsing to server level

* use a granular function

* fix linter

* adjust time reference

* provide reference timezone in the message

* debug: disable db migrate

* feat: initial dummy dex resolver

* feat: initial configurable dummy 3rd party resolvers

* feat: resolve dex by string matching + resolve it on different function call

* feat: add placeholder for creating sensor in lib.py

* feat: persist third party type

* fix: lint

* fix: column naming for 3rd party

* fix: test + resource urn naming

* feat: use dedicated third party table

* fix: lint

* fix: lint

* fix: bring back migrate db

* refactor: rename table migration for 3rd party resolver

* Dex api integration (#489)

* feat: resolve dex by string matching + resolve it on different function call

* feat: add placeholder for creating sensor in lib.py

* feat: persist third party type

* fix: test + resource urn naming

* feat: use dedicated third party table

* fix: lint

* fix: integrate dex API

* fix

* refactor: use url parser + producer type check

* refactor: abstracting out sensor service

* refactor: returning the response object for third party client

---------

Co-authored-by: Dery Rahman Ahaddienata <dery.ra@gmail.com>
Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* refactor: move window logic to optimus server (#496)

* refactor: remove unecessary code

* refactor: reuse window interval calculation on server side

* refactor: lowercase response

* add feature flag, and metrics

* fix: return iscomplete status

* refactor: const status, remove identifier, refactor poke

* update: proto

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* fix: lint

* fix: test case

* fix: lint

* feat: add auth email support

* fix: integration test

* fix: duplicate db migrate file

* fix: config parsing

* fix: parseDateTime properly

* fix: __lib.py

* fix: add more logs

* fix

* fix: test

---------

Co-authored-by: Yash Bhardwaj <imyashbhardwaj@gmail.com>
Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* feat: enhance potential sla log + response

* feat: enhance with job run information

* feat: update proto commit

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

* fix: dex resolver

* refactor: fix dex resolver

* feat: improve parallel bul resolve 3rd party

* refactor: enhance upstream error

* refactor: append unresolved

* fix: add test cases for DEX resolver

* fix: lint

* fix: add test for bulk resolve

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>

…each (#500)

* feat: add neccessary logs

* feat: update proto + add dampering coeff through api

* feat: include target job as calculation

* feat: deduplicate if alert already exist

* fix: linter

* refactor: full causes to store in persistent repo

* feat: skip detection for job on the list

* fix: return no breach

* fix: temp solution to skip preprod project

* feat: bump proton version

* fix: only fill end times if job/operators succeed

* fix query

* group runs by project

* explicitly exclude preprod

* add brackets for or cases

* fix: temp solution ignore backfill alert

* fix: prevent nil pointer

* refactor: use alerrules and expose it on constructor

* fix: linter

* test: add test case detect backfill

* fix: dex interval fix

* fix: tests

---------

Co-authored-by: Yash Bhardwaj <yash.bhardwaj@gojek.com>