Skip to content

Conversation

@schlawg
Copy link
Contributor

@schlawg schlawg commented Jun 5, 2025

this code assumes the automod subdocument has the integer based "automod.quality" field rather than the string based "automod.classification". data using the older schema is live on prod as of 05/06/2025

that data must be migrated before ublog ingests & queries will function.

@lenguyenthanh
Copy link
Member

that data must be migrated before ublog ingests & queries will function.

Do we have anything to track down this migration?

textField(text),
shortField(quality).copy(docValues = Some(true)),
keywordField(language).copy(docValues = Some(false)),
dateField(date).copy(docValues = Some(true))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a curious question, why does some fields keep values, some don't?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that disabling docValues on fields you don’t sort/aggregate keeps segment size smaller. language would only ever be filtered or searched. it wouldn't be sorted.

@schlawg schlawg requested a review from lenguyenthanh June 9, 2025 17:02
intro <- doc.getString(F.intro)
body <- doc.getString(F.markdown)
author <- doc.getString(F.blog).map(_.split(":")(1))
language <- doc.getString("language")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
language <- doc.getString("language")
language <- doc.getString(F.language)

Copy link
Member

@lenguyenthanh lenguyenthanh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to do the migration before merging, to avoid accidental deploy. otherwise lgtm. great job!

@lenguyenthanh
Copy link
Member

something fishy with git history 🤔

@ornicar ornicar merged commit be4d2f6 into lichess-org:master Jul 16, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants