In PR #879 some functions of Schemathesis testing were disabled:
|
schema.config.checks.positive_data_acceptance.enabled = False |
|
schema.config.generation.allow_extra_parameters = False |
If these are enabled, two tests fail:
________________________________________________ test_openapi_fuzzy[POST /projects/{project_id}/suggest] ________________________________________________
+ Exception Group Traceback (most recent call last):
| File "/home/jmminkin/git/Annif/.venv/lib/python3.11/site-packages/hypothesis/core.py", line 2287, in wrapped_test
| raise the_error_hypothesis_found
| schemathesis.core.failures.FailureGroup: Schemathesis found 1 distinct failure
|
| - API rejected schema-compliant request
|
| Valid data should have been accepted
| Expected: 2xx, 401, 403, 404, 409, 5xx
| Hint: The request body contains 1 additional property not defined in the schema (``). The server likely rejects unexpected fields. Add `additionalProperties: false` to your schema to prevent this.
|
| [400] Bad Request:
|
| `{"type": "about:blank", "title": "Bad Request", "detail": "language \"d\ud9d9\uddbc\" not supported by vocabulary", "status": 400}`
|
| Reproduce with:
|
| curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d 'text=d%C2%9F%C2%AD%C3%A6%1B&=%C3%8F%0A&limit=3200206&threshold=1.2279522352668104e-107&language=d%F2%86%96%BC' http://localhost/v1/projects/dummy-en/suggest
|
| (1 sub-exception)
+-+---------------- 1 ----------------
| schemathesis.openapi.checks.RejectedPositiveData: API rejected schema-compliant request
|
| Valid data should have been accepted
| Expected: 2xx, 401, 403, 404, 409, 5xx
| Hint: The request body contains 1 additional property not defined in the schema (``). The server likely rejects unexpected fields. Add `additionalProperties: false` to your schema to prevent this.
+------------------------------------
and
_____________________________________________ test_openapi_fuzzy[POST /projects/{project_id}/suggest-batch] _____________________________________________
+ Exception Group Traceback (most recent call last):
| File "/home/jmminkin/git/Annif/tests/test_openapi.py", line 77, in test_openapi_fuzzy
| case.call_and_validate()
| File "/home/jmminkin/git/Annif/.venv/lib/python3.11/site-packages/schemathesis/generation/case.py", line 601, in call_and_validate
| self.validate_response(
| File "/home/jmminkin/git/Annif/.venv/lib/python3.11/site-packages/schemathesis/generation/case.py", line 560, in validate_response
| raise FailureGroup(_failures, message) from None
| schemathesis.core.failures.FailureGroup: Schemathesis found 1 distinct failure
|
| - API accepted schema-violating request
|
| Invalid data should have been rejected
| Expected: 400, 401, 403, 404, 405, 406, 409, 422, 428, 5xx
| Invalid component: in query - object with unexpected properties
|
| [200] OK:
|
| `[]`
|
| Reproduce with:
|
| curl -X POST -H 'Content-Type: application/json' -d '{"documents": []}' 'http://localhost/v1/projects/dummy-fi/suggest-batch?threshold=0&x-schemathesis-unknown-property=42'
|
| (1 sub-exception)
+-+---------------- 1 ----------------
| schemathesis.core.failures.AcceptedNegativeData: API accepted schema-violating request
|
| Invalid data should have been rejected
| Expected: 400, 401, 403, 404, 405, 406, 409, 422, 428, 5xx
| Invalid component: in query - object with unexpected properties
+------------------------------------
I think it should be possible to restrict the generation of languages to resolve the first failure (the error response "not supported by vocabulary" is correct here, it just is not possible to be expressed in OpenAPI spec?).
For the second failure, I speculated on an resolution:
In addition to upgrading to Schemathesis 4, this PR currently enables strict_validation in Connexion "to disallow any extra parameters that are not defined in your specification". I think this would be a good feature, because then typos in API requests parameters are noticed at once, not after debugging (with strict validation e.g. using theshold instead of threshold gives 400 Bad request error, without it the extra, nonfunctional parameter is just silently ignored).
However, Annif 1.4 introduced metadata_* parameters to suggest/, suggest_batch/ and learn/ API methods (#867), and to return a 400 error for requests with parameters not starting with metadata_ I made this: Add Custom request body validator.
Originally posted by @juhoinkinen in #879 (comment)
Also, on some runs a third test has failed (can't remember of find again which or reproduce); to not make this happen surprisingly, Schemathesis could be made generate the test cases determistically: schemathesis/schemathesis#3531 (comment)
In PR #879 some functions of Schemathesis testing were disabled:
Annif/tests/test_openapi.py
Lines 12 to 13 in ef5d8ae
If these are enabled, two tests fail:
and
I think it should be possible to restrict the generation of languages to resolve the first failure (the error response "not supported by vocabulary" is correct here, it just is not possible to be expressed in OpenAPI spec?).
For the second failure, I speculated on an resolution:
Originally posted by @juhoinkinen in #879 (comment)
Also, on some runs a third test has failed (can't remember of find again which or reproduce); to not make this happen surprisingly, Schemathesis could be made generate the test cases determistically: schemathesis/schemathesis#3531 (comment)