Page MenuHomePhabricator
Create Task
Maniphest T346264

Improve wikifunctions logging
Closed, ResolvedPublic
Actions

Description

Currently wikifunctions pods log ~2500 lines per hour in production (in one DC). The absolute majority of those logs are from "levelPath: trace/req" for the /metrics and /_info endpoints, which could probably suppressed. There are also a bunch of "levelPath: debug" logs logging memory and cpu usage, lacking any other identifier (like a request-id). Same goes for "levelPath: info" which seems to print the function call (plus arguments?) and some "levelPath: notice".

But especially the amount of trace/req logs probably made it hard to spot the problem that lead to T344998 right away, see https://logstash.wikimedia.org/goto/aa3497c585530d231916e0d2b479c026 vs. https://logstash.wikimedia.org/goto/d9f2a3f0cab9474b5d48bed41cf7b8c1

Details

SubjectRepoBranchLines +/-
wikifunctions: Use function-orchestrator image with better loggingoperations/deployment-chartsmaster+2 -2
Update function-schemata sub-module to HEAD (958de5c)mediawiki/extensions/WikiLambdamaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm renamed this task from Improcve wikifunctions logging to Improve wikifunctions logging.Sep 13 2023, 4:59 PM
JMeybohm created this task.
CodeReviewBot added a project: Patch-For-Review.Sep 26 2023, 2:07 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/39

LoggerWrapper: Pass data object to logger service too

CodeReviewBot added a comment.Sep 26 2023, 2:09 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/60

Logging: Actually use the logger in the correct way, passing data objects

Jdforrester-WMF changed the task status from Open to In Progress.Sep 26 2023, 2:09 PM
Jdforrester-WMF claimed this task.
Jdforrester-WMF triaged this task as High priority.
Jdforrester-WMF moved this task from To triage to In Progress on the Abstract Wikipedia team board.
Jdforrester-WMF added a project: function-orchestrator.
CodeReviewBot added a comment.Sep 28 2023, 3:21 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/39

LoggerWrapper: Pass data object to logger service too

CodeReviewBot added a comment.Sep 28 2023, 4:25 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/15

Update function-schemata sub-module to HEAD (958de5c)

CodeReviewBot added a comment.Sep 28 2023, 4:25 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/56

Update function-schemata sub-module to HEAD (958de5c)

gerritbot added a comment.Sep 28 2023, 4:26 PM

Change 961848 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (958de5c)

https://gerrit.wikimedia.org/r/961848

CodeReviewBot added a comment.Sep 28 2023, 4:26 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/64

Update function-schemata sub-module to HEAD (958de5c)

CodeReviewBot added a comment.Sep 28 2023, 5:07 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/15

Update function-schemata sub-module to HEAD (958de5c)

CodeReviewBot added a comment.Sep 28 2023, 5:09 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/56

Update function-schemata sub-module to HEAD (958de5c)

gerritbot added a comment.Sep 28 2023, 5:16 PM

Change 961848 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (958de5c)

https://gerrit.wikimedia.org/r/961848

CodeReviewBot added a comment.Sep 28 2023, 5:30 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/64

Update function-schemata sub-module to HEAD (958de5c)

Jdforrester-WMF moved this task from In Progress to To deploy on the Abstract Wikipedia team board.Sep 28 2023, 8:26 PM
ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.29; 2023-10-03).Sep 29 2023, 10:01 AM
CodeReviewBot added a comment.Oct 2 2023, 1:18 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/60

Logging: Actually use the logger in the correct way, passing data objects

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2023, 1:31 PM
gerritbot added a comment.Oct 2 2023, 4:55 PM

Change 962650 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Use function-orchestrator image with better logging

https://gerrit.wikimedia.org/r/962650

gerritbot added a project: Patch-For-Review.Oct 2 2023, 4:55 PM
gerritbot added a comment.Oct 10 2023, 7:21 PM

Change 962650 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Use function-orchestrator image with better logging

https://gerrit.wikimedia.org/r/962650

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2023, 7:30 PM
CodeReviewBot added a project: Patch-For-Review.Oct 10 2023, 7:43 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/71

Replace all remaining log writes to trace/req with real values

CodeReviewBot added a comment.Oct 10 2023, 7:53 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/61

Replace all remaining log writes to trace/req with real values

CodeReviewBot added a comment.Oct 10 2023, 9:46 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/71

Replace all remaining log writes to trace/req with real values

Jdforrester-WMF added a parent task: T343458: Allow logged-out users to run (community-approved) implementations on wikifunctions.org via the `wikilambda-execute` right.Oct 12 2023, 4:34 PM
CodeReviewBot added a comment.Oct 12 2023, 9:53 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/61

Replace all remaining log writes to trace/req with real values

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 10:10 PM
Jdforrester-WMF moved this task from To deploy to Verify in production on the Abstract Wikipedia team board.Oct 16 2023, 3:20 PM
Etonkovidova subscribed.EditedOct 17 2023, 11:21 PM

When I check https://logstash.wikimedia.org/goto/dbb318a11eaeb4b54503b9878c805f15 for the last 1 hour and remove NOT levelPath:trace/req, the number of events is unchanged, so the levelPath:trace/req is suppressed.
But levelPath:trace is still present on Apps Logs (Kubernetes) (about 3,600 in 1 hour). We don't need to suppress them too?

Also, the dashboard wikifunctions has events with level 10 (`levelPath
trace`), ~8,404; and level INFO accounts for 4,551 events. Overall (without filtering), wikifunctions dashboard presents 13,110 events in 1 hour. Should the dashboard filters be adjusted further?

Additional improvements for logging will be addressed in other tasks if necessary. trace/req is not present anymore. Closing this task as Resolved.

Etonkovidova closed this task as Resolved.Oct 23 2023, 9:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits