Description
Patterns in the blacklist and whitelist are normally written in their natural spelling, for example "hello". However, in the actual use, the input text is first passed through a chain of transformers. These transformers can prevent some patterns from matching. For example, collapseDuplicatesTransformer can transform "hello" into "helo", which will prevent the pattern "hello" from matching. This is highlighted in the documentation. To avoid the issue, the default English preset leaves several letters duplicated (up to 2 characters). But this causes another issue -- now it is possible to evade the filter for words that have these characters by duplicating them. The original issue is also not fully fixed -- if someone adds a pattern that has a duplicate of another character (not one of "beolsg"), their pattern will not match with the default preset.
Solution
The idea is to run the patterns through the transformers. This way duplicate characters in the patterns will also be collapsed and so they will match deduplicated text. Patterns contain more than literal text, so one has to be careful not to break metacharacters, but I think this is quite doable.
Code of Conduct
Description
Patterns in the blacklist and whitelist are normally written in their natural spelling, for example "hello". However, in the actual use, the input text is first passed through a chain of transformers. These transformers can prevent some patterns from matching. For example,
collapseDuplicatesTransformercan transform "hello" into "helo", which will prevent the pattern "hello" from matching. This is highlighted in the documentation. To avoid the issue, the default English preset leaves several letters duplicated (up to 2 characters). But this causes another issue -- now it is possible to evade the filter for words that have these characters by duplicating them. The original issue is also not fully fixed -- if someone adds a pattern that has a duplicate of another character (not one of "beolsg"), their pattern will not match with the default preset.Solution
The idea is to run the patterns through the transformers. This way duplicate characters in the patterns will also be collapsed and so they will match deduplicated text. Patterns contain more than literal text, so one has to be careful not to break metacharacters, but I think this is quite doable.
Code of Conduct