Skip to content

Skip 3 expensive calls for jobs saving in 'waiting' status on UnifiedJob#15174

Merged
kdelee merged 1 commit into
ansible:develfrom
kdelee:only_skip_parent_update
May 13, 2024
Merged

Skip 3 expensive calls for jobs saving in 'waiting' status on UnifiedJob#15174
kdelee merged 1 commit into
ansible:develfrom
kdelee:only_skip_parent_update

Conversation

@kdelee
Copy link
Copy Markdown
Member

@kdelee kdelee commented May 9, 2024 •
edited
Loading

SUMMARY

Alternative to #15171
@AlanCoding suggested most speed up could be gained in just not updating parent for waiting jobs

Put all 39,000 jobs in my system back in pending and triggered task manager. Had START_TASK_LIMIT set to 10,000

Result was ~ 133 jobs/second -- close to the 142 jobs/second we get from bulk_update approach with exact same workload in #15171 (compared to ~ 20 jobs/second in devel)

# HELP task_manager_get_tasks_seconds Time spent in loading tasks from db
# TYPE task_manager_get_tasks_seconds gauge
task_manager_get_tasks_seconds{node="awx-1"} 18.265242898953147
# HELP task_manager_start_task_seconds Time spent starting task
# TYPE task_manager_start_task_seconds gauge
task_manager_start_task_seconds{node="awx-1"} 54.556849607266486
# HELP task_manager_process_running_tasks_seconds Time spent processing running tasks
# TYPE task_manager_process_running_tasks_seconds gauge
task_manager_process_running_tasks_seconds{node="awx-1"} 1.7329584807157516e-06
# HELP task_manager_process_pending_tasks_seconds Time spent processing pending tasks
# TYPE task_manager_process_pending_tasks_seconds gauge
task_manager_process_pending_tasks_seconds{node="awx-1"} 57.23840588296298
# HELP task_manager__schedule_seconds Time spent in running the entire _schedule
# TYPE task_manager__schedule_seconds gauge
task_manager__schedule_seconds{node="awx-1"} 75.57144767208956
# HELP task_manager__schedule_calls Number of calls to _schedule, after lock is acquired
# TYPE task_manager__schedule_calls gauge
task_manager__schedule_calls{node="awx-1"} 4
# HELP task_manager_recorded_timestamp Unix timestamp when metrics were last recorded
# TYPE task_manager_recorded_timestamp gauge
task_manager_recorded_timestamp{node="awx-1"} 0.0
# HELP task_manager_tasks_started Number of tasks started
# TYPE task_manager_tasks_started gauge
task_manager_tasks_started{node="awx-1"} 10000
# HELP task_manager_running_processed Number of running tasks processed
# TYPE task_manager_running_processed gauge
task_manager_running_processed{node="awx-1"} 0
# HELP task_manager_pending_processed Number of pending tasks processed
# TYPE task_manager_pending_processed gauge
task_manager_pending_processed{node="awx-1"} 39197
# HELP task_manager_tasks_blocked Number of tasks blocked from running
# TYPE task_manager_tasks_blocked gauge
task_manager_tasks_blocked{node="awx-1"} 67
# HELP task_manager_commit_seconds Time spent in db transaction, including on_commit calls
# TYPE task_manager_commit_seconds gauge
task_manager_commit_seconds{node="awx-1"} 0.04217028617858887
ISSUE TYPE
  • Bug, Docs Fix or other nominal change
COMPONENT NAME
  • API
ADDITIONAL INFORMATION

@AlanCoding has indicated he'd like to solve this in potentially more universal way leveraging django-ansible-base

@kdelee kdelee mentioned this pull request May 9, 2024
Closed
@kdelee kdelee force-pushed the only_skip_parent_update branch 2 times, most recently from 1130e34 to 03530c8 Compare May 9, 2024 14:47
@kdelee kdelee changed the title [draft] skip update parent logic for 'waiting' on UnifiedJob [draft] skip 3 expensive calls for jobs saving in 'waiting' status on UnifiedJob May 9, 2024
@kdelee
skip update parent logic for 'waiting' on UnifiedJob
3a029c3 
by not looking up "status_before" from previous instance
we save 2 to 3 expensive calls (the self lookup of old state, the lookup
of parent, and the update to parent if allow_simultaneous == False)
@kdelee kdelee force-pushed the only_skip_parent_update branch from 03530c8 to 3a029c3 Compare May 9, 2024 14:50
@kdelee kdelee changed the title [draft] skip 3 expensive calls for jobs saving in 'waiting' status on UnifiedJob Skip 3 expensive calls for jobs saving in 'waiting' status on UnifiedJob May 9, 2024
@kdelee kdelee merged commit 439c3f0 into ansible:devel May 13, 2024
@kdelee kdelee deleted the only_skip_parent_update branch May 13, 2024 14:26
djyasin pushed a commit to djyasin/awx that referenced this pull request Sep 16, 2024
@kdelee @djyasin
Skip 3 expensive calls for jobs saving in 'waiting' status on Unified…
33b344f 
…Job (ansible#15174)

skip update parent logic for 'waiting' on UnifiedJob

by not looking up "status_before" from previous instance
we save 2 to 3 expensive calls (the self lookup of old state, the lookup
of parent, and the update to parent if allow_simultaneous == False or status == 'waiting')
djyasin pushed a commit to djyasin/awx that referenced this pull request Nov 11, 2024
@kdelee @djyasin
Skip 3 expensive calls for jobs saving in 'waiting' status on Unified…
77ec57c 
…Job (ansible#15174)

skip update parent logic for 'waiting' on UnifiedJob

by not looking up "status_before" from previous instance
we save 2 to 3 expensive calls (the self lookup of old state, the lookup
of parent, and the update to parent if allow_simultaneous == False or status == 'waiting')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrismeyersfsu chrismeyersfsu chrismeyersfsu approved these changes

@AlanCoding AlanCoding AlanCoding approved these changes

Assignees

No one assigned

Labels

component:api

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kdelee @chrismeyersfsu @AlanCoding