Repository navigation
CSV rules do not interpolate non-ASCII field names #1809
Description
When declaring a field name with non-ASCII characters in it in a .rules file, that field won't interpolate correctly when calling hledger print.
Example CSV file (test.csv) with accompanying rules:
Date, Description, Id, Amount
12/11/2019, Foo, 123, 10.23
#test.csv.rules
skip 1
fields date, description, unicöde, amount
date-format %d/%m/%Y
code %unicöde
% hledger -f test.csv print
2019-11-12 (%unicöde) Foo
expenses:unknown 10.23
income:unknown -10.23
I'm currently on macOS Catalita v10.15.7
This is the output of hledger --version, which I installed using Homebrew:
% hledger --version
hledger 1.24.1, mac-x86_64
Activity
[-]CSV rules do not interpolate unicode field names[/-][+]CSV rules do not interpolate non-ASCII field names[/+]Xitian9 commentedon Jan 22, 2022
Indeed, it looks like field references are specifically restricted to be alphanumeric ascii characters (plus
-and_). SeeisFieldNameCharbelow.https://github.com/simonmichael/hledger/blob/e16fa51dd54687e957e9342c361f3b9ef6776efa/hledger-lib/Hledger/Read/CsvReader.hs#L1239-L1248
The question is whether this was a specific choice, in which case it should be documented, or if it was just a convenient way of getting at something else, in which case it should be generalised. @simonmichael, do you happen to know about this?
simonmichael commentedon Jan 22, 2022
I expect it was a deliberate (& anglocentric) simplification as these are just convenient aliases for %1 etc. seen only infrequently (?), and full unicode is ever evolving and contains dragons.
But I can see you might want to use names with proper local spelling or script, so I guess I'm in favour of limited demand-driven enabling of progressively larger subsets of unicode, when someone is willing to do the testing and fixing. Guessing: Accents for european scripts, ok. CJK and other scripts, ok when requested, attending to double-width display issues. More punctuation, probably not. Semigraphics, weird spaces, etc, no.
Xitian9 commentedon Jan 22, 2022
The obvious generalisation would be to just drop
isAscii, leavingisFieldNameChar c = isAlphaNum c || c == '_' || c == '-'. Is there a reason not to do this? I don't think these are ever displayed unless the corresponding field is missing from the csv file: what issues do you foresee arising?simonmichael commentedon Jan 22, 2022
simonmichael commentedon Jan 22, 2022
Xitian9 commentedon Jan 22, 2022
Unfortunately
isAlphaNumis implemented as a foreign function call tou_iswalnum. The definition is here and is a bit dense. Basically, any unicode character of class Lt (Letter, titlecase), Lu (Letter, uppercase), Ll (Letter, lowercase), Lm (Letter, modifier), Lo (Letter, other), Mc (Mark, spacing, combining), Me (Mark, enclosing), Mn (Mark, nonspacing), No (Number, other), Nd (Number, decimal digit), or Nl (Number, letter). Enlightening…joelhed commentedon Jan 23, 2022
As mentioned, it would be nice for the CSV reader to support more unicode characters, however, in the meantime, there should probably be some error messaging telling the user that non-ASCII characters are not supported yet. Maybe even when the rules file is parsed and the
fieldsdirective is encountered instead of at the rendering stage. If it is a rule, why not make it explicit and disallow it right when it is broken?simonmichael commentedon Jan 23, 2022
Agreed, if it's possible to detect this reliably ?
fix: csv: Allow unicode in field references for csv. (hledgerorg#1809)
simonmichael commentedon Feb 6, 2022
#1815 enables all 133022 Unicode "alphanumeric" characters. Here's the first screenful:
I wondered if included any problematic exploit-friendly chars, like the kind that are invisible, or change the direction of rendering, or look like things they're not. Too many to audit. These did at least corrupt my emacs minibuffer rendering:
isLatin1 would allow slightly more flexibility than ASCII, but not eg CJK characters. I don't know if using CJK characters in CSV rules makes sense, but it sounds nice to offer.
A better rendering of the first 30000:
simonmichael commentedon Feb 7, 2022
Alright.. let's assume Unicode letters and numbers categories are probably free of the worst issues like invisible characters (though they do obviously contain lookalike/mistakable ones), I guess I'm ok with allowing them all. i18n is always something we like.
fix: csv: Allow unicode in field references for csv. (#1809)
fix: csv: Allow unicode in field references for csv. (#1809)
hledger, hledger-lib: Update to 1.28
hledger, hledger-lib: Update to 1.28
hledger, hledger-lib: Update to 1.28