tests/llm/fixtures/test_ask_holmes/42_dns_issues_steps_old_tools/test_case.yaml (1)

19-21: kubectl apply -f./manifest.yaml will fail – missing space

kubectl treats -f as a flag whose value must be provided after a space (or with =). Without the space, it is parsed as an unknown shorthand flag and the test will error out.

-  kubectl apply -f./manifest.yaml
+  kubectl apply -f ./manifest.yaml

Same fix is needed for the corresponding kubectl delete line.

tests/llm/fixtures/test_ask_holmes/42_dns_issues_result_all_tools/test_case.yaml (1)

6-9: Missing space after -f flag

Same kubectl apply/delete -f./manifest.yaml issue as in the sister file.

tests/llm/fixtures/test_ask_holmes/42_dns_issues_steps_new_all_tools/test_case.yaml (1)

2-4: Duplicate feedback: see the earlier comment about network vs networking marker ambiguity.

tests/llm/fixtures/test_ask_holmes/42_dns_issues_result_old_tools/test_case.yaml (1)

2-4: Duplicate feedback: see the earlier comment about network vs networking marker ambiguity.

tests/llm/fixtures/test_ask_holmes/42_dns_issues_result_new_tools/test_case.yaml (1)

2-4: Duplicate feedback: see the earlier comment about network vs networking marker ambiguity.

tests/llm/fixtures/test_ask_holmes/42_dns_issues_result_all_tools/test_case.yaml (1)

2-4: Same taxonomy collision as above

Ensure only one of network / networking survives in the constants file and in all test cases.

tests/llm/fixtures/test_ask_holmes/21_job_fail_curl_no_svc_account/test_case.yaml (1)

18-20: Minor typo in comment

Currenlty → Currently.

-# Currenlty holmes is able to answer the question correctly only around 10% of the time.
+# Currently Holmes is able to answer the question correctly only around 10 % of the time.

tests/llm/fixtures/test_ask_holmes/42_dns_issues_steps_new_tools/test_case.yaml (1)

2-4: Potential confusion between network vs networking tags

network is already an established marker across the suite, and now we introduce networking.
Unless the two convey distinct semantics, having both will fragment filtering (pytest -m network vs pytest -m networking) and make dashboards harder to search.

If the intent is simply to rename/clarify the old marker, consider:

-  - networking
+  - network

and then deprecate networking from pyproject.toml.
Otherwise, document the precise difference in CONTRIBUTING.md and ensure every test uses the right one.

pyproject.toml (1)

83-87: Minor ordering nit

For readability the marker list has so far been alphabetical.
Keeping that convention helps future reviewers spot duplicates quickly.

-    "networking: Tests involving networking functionality",
-    "misleading-history: Tests with misleading historical data",
-    "k8s-misconfig: Tests involving Kubernetes misconfigurations",
-    "chain-of-causation: Tests involving chain-of-causation analysis",
-    "slackbot: Tests involving Slack bot functionality",
+    "chain-of-causation: Tests involving chain-of-causation analysis",
+    "k8s-misconfig: Tests involving Kubernetes misconfigurations",
+    "misleading-history: Tests with misleading historical data",
+    "networking: Tests involving networking functionality",
+    "slackbot: Tests involving Slack bot functionality",

tests/llm/fixtures/test_ask_holmes/42_dns_issues_steps_old_tools/test_case.yaml (1)

5-17: Block scalar starts with an empty line – unintended leading newline

expected_output: | is immediately followed by a blank line, which introduces a leading newline into the value. If the evaluation logic performs exact or trimmed comparisons, this may cause false negatives. Delete the empty line or switch to |- to strip.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: nherment
PR: robusta-dev/holmesgpt#408
File: holmes/plugins/toolsets/kubernetes_logs.py:90-97
Timestamp: 2025-05-15T05:13:43.169Z
Learning: In the Kubernetes logs toolset for Holmes, both current and previous logs are intentionally fetched and combined for each pod, even though this requires more API calls. This design ensures all logs are captured even when pods restart but retain their name, providing complete diagnostic information.

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.

Learnt from: Sheeproid
PR: robusta-dev/holmesgpt#586
File: tests/llm/fixtures/test_ask_holmes/03_what_is_the_command_to_port_forward/test_case.yaml:4-4
Timestamp: 2025-07-02T10:27:17.231Z
Learning: In LLM-as-judge test cases for HolmesGPT, expected outputs should be descriptive rather than prescriptive when testing for flexible responses like port numbers. Using specific values in expected outputs can cause unnecessary test failures when the AI generates different but equally valid responses.