Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.66%. Comparing base (140a1b6) to head (9592eef).
⚠️ Report is 15 commits behind head on main.

@@           Coverage Diff            @@
##             main     #872    +/-   ##
========================================
  Coverage   99.66%   99.66%            
========================================
  Files         102      103     +1     
  Lines        7665     7838   +173     
========================================
+ Hits         7639     7812   +173     
  Misses         26       26            

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@juhoinkinen

jsonschema could be used for validating the JSON data and for error handling. However, for such a simple schema, it might be unnecessary.

Yes, that's a great idea! I think that the schema is just about complex enough to justify JSON Schema validation. I'll look into it. (I already added basic error handling for cases where the file is empty or has broken JSON syntax)

Note to self: there's now a subtle bug in annif index. The list of paths returned by DocumentDirectory may contain broken/empty files, because their content is not checked during initial iteration. That means that the list of file names may get out of sync with the list of suggestion results, causing results to be written into the wrong files.

EDIT: Should be fixed by 3031971

@juhoinkinen I added JSON Schema validation, made it possible to specify subjects via labels instead of URIs (as in the .txt/.tsv/.key format), moved the JSON support into its own submodule json.py, and fixed some issues with annif index. I can't think of anything else right now, so from my side, this is getting ready for merging. Do you want to take another look?

Also @c-poley @RietdorfC feel free to test and comment!

Quality Gate failed

Failed conditions
24.1% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

Hi @osma,
we started testing the JSON fulltext format but unfortunally encountered one error and one problem.
The error occured when we tried to run the annif index command. We prepared two files from the same document: a .txt file containing simply the full text of the document (you can find this file attached as 1000694364.txt) and one .json file with an additional metadata field for the title and its subjects (you can find this file attached as 1000694364.json). We placed both files in directories to use the annif index command. In our test, we used the same model to run the index command (an omikuji model trained on full texts). To test if both files could be processed, we ran the following commands:

annif index A-omikuji-FT tmp-txt/
annif index A-omikuji-FT tmp-json/

The first one completed without any problems and produced the expected output, but the second one, which refers to the JSON-file, generated an error massage (you can find the message in the file error.txt). We couldn’t trace the error back to our input, but maybe we overlooked something?

A question occured when we tried to run the annif suggest command on a JSON file. First, we tested the annif suggest command with an omikuji model without the transform=select(title) parameter. We tested the model on a file from the classic full text corpus format (same .txt file as above) and got a valid result. Then we tested the same model but on a JSON file (same as mentioned above) and also got valid results.
Next, we tested a omikuji model with the new parameter transform=select(title) set. First, we tested it on the .txt file and, as expected got no result, as there is no title for the model to process. Second, we tested the model on the JSON file but again got no result, even though there is a title given in the metadata section of the JSON file. Why didn't it work? You can find the complete input and output in the file suggest.txt.

Thanks a lot and best regards
Clemens

1000694364.txt
1000694364.json
error.txt
suggest.txt

Thank you @RietdorfC for testing and for reporting the problems.

For the first problem, it appears that subject_index is not properly set for some reason. For the second problem, I'm not sure what happened.

I will investigate and see if I can reproduce these problems.

@RietdorfC just verifying - the testing was performed using the final PR code that was merged to main yesterday and not an earlier version?

Hi @osma,
Yes, we used the newest version with the final pull request merged.

I found the first problem. There is a bugfix in PR #874 . Would you like to test it?

As for the second problem, this happens because annif suggest is not aware of JSON files. It expects to be given a plain text file and does not care about the filename extension. If you give it a JSON file, it will still parse it as plain text. Therefore there is no title and the suggestion result is empty. Maybe this could be changed (open an issue?) but for now this is how things stand.

Hi @osma ,
Yes, we will test the bugfix!

Thanks for the clarification! The JSON support for the suggest command is not essential for us. So there is no need for us to change it.

Hi @osma,

We tested the index command again and now everything works perfectly. Thank you! We also tested the train, eval, hyperopt and optimize commands with the new JSON fulltext corpus format and did not encounter any problems. Everything seems to work. In addition, tried using the content of the JSON files directly as (part of) the request body for the suggest_batch function with the REST API. That worked really well too.

Thanks again to your and your team!
Best regards
Clemens