Inconsistent rounding for balance CSV rules (due to amount1 0)

hledger 1.18.1 on Fedora 33:

$ cat >f.csv <<EOF
12/31/2018,21.058,$256.1300
EOF
$ cat >f.rules <<EOF
fields date, quantity, price
date-format %m/%d/%Y

amount1 0
balance %quantity @ %price
EOF
$ hledger print --rules-file f.rules --file f.csv
2018-12-31
    expenses:unknown               0 = 21 @ $256.1300

I've ommited the 'currency' statement for a minimal example, in my actual example I have a currency1 STOCKSYMBOL, reproduces both ways.

There is some documentation here on how the display style is guessed
https://hledger.org/hledger.html#commodity-display-style

However it is very unexpected that one commodity has no decimals guessed, and the other one has 4
(and this will lead to a balance assertion failure further down the line because the 21.058 is the correct value).
There is no way to specify a 'commodity' in a rules file, so have to rely on hledger guessing.

If I modify amount1 0 to amount1 0.000 then I get the correct result, this was very unexpected. Perhaps hledger should print a warning if it ever does truncation on a line that has values with larger number of decimals?
FWIW -O json has the correct amount printed.

I only added amount1 0 because without it balance assertions wouldn't work.

Thanks for the report. I think the same thing happens in latest hledger.

Minimal examples are good, but I'm finding this one a little hard to debug - could you provide one that's a little more realistic ? Eg one record, but the full record and rules you're using ?

Or maybe this is realistic ? Are you actually hard coding the amount to 0 ?

My CSV files have 2 kinds of entries: transactions without balance, and balances.
Balances I want to translate to balance assertions, and for that the only way I found it to work with hledger was to hardcode the amount to 0 in the rules file (otherwise hledger complains:

real postings' sum should be 0 but is: $21.058
2018-12-31
    expenses:unknown         $21.058 = $21.05

). If there is a better way, please let me know.

Here is a more realistic CSV where it prints '21':

12/31/2018,STOCK (MYSTOCK),,BALANCE,21.058,$102.4600

And here is another more detailed CSV that prints 21.058:

12/31/2018,STOCK (MYSTOCK) 1.370%,,BALANCE,21.058,$102.4600
12/21/2018,STOCK TRADE DATE,17,Reinvestment,0.058,$107.73850,,$,-$6.2
12/21/2018,STOCK,17,Dividend Received,-,,,$,7.35
12/21/2018,STOCK,,Non-Resident Tax,,,,$,-$1.1

(The CSV files are converted by a script of mine from Fidelity's PDF statements, and there are statements for various periods,
that sometimes overlap)

and more realistic rules (they're a bit hacked together, I'm in the process of refining them):

fields date, desc, cusip,kind, quantity, price, txbasis, txcost, xamount
date-format %m/%d/%Y

account1 Assets:fidelity:shares
description %desc:%kind

if %desc STOCK
    currency1 MYSTOCK

# must be first in case it is overriden
if %xamount ^-?$
    amount1-in 0

if %kind BALANCE
    amount1 0
    balance %quantity @ %price

if ESPP.*You Bought
    account2 Assets:fidelity:espp-cash
    amount1-in %quantity @ %price
    amount2 %xamount

if RSU.*You Bought
    account2 income:salary:myemployer:rsu
    amount1-in %quantity @ %price

if %kind You Sold
    account2 Assets:transfers:fidelity:cashout
    amount1 %quantity @ %price
    account3 expenses:fidelity:txcost
    amount3-out %txcost

# rounding errors in quantity*price vs amount goes into equity:conversion
if %kind Reinvestment
    account2 Assets:transfers:fidelity:dividends
    amount1-in %quantity @ %price
    amount2 %xamount
    account3 equity:conversion

if %quantity ^-$
    account1 Assets:fidelity:espp-cash
    amount1-in %xamount
    account2 Assets:transfers:fidelity

if Journaled,-,,,
    amount1-in 0

if %kind Dividend
    currency1 $
    account1 Assets:transfers:fidelity:dividends
    account2 income:dividends:ctxs
    amount2 -%xamount
    amount1-in %xamount

if %kind Non-Resident Tax
    account1 Assets:transfers:fidelity:dividends
    account2 expenses:tax:fidelity:us
    amount2 -%xamount

if %desc CASH WITHDRAWAL
    account1 Assets:transfers:fidelity:cashout
    account2 Assets:edwin:mybank:current
    amount1 %xamount

if %kind In Lieu of Frx Share
    amount1 %quantity
    account2 Assets:transfers:fidelity:cashout
    amount2 %xamount

If hledger's rules file would accept a commodity declaration then perhaps the guessing could be avoided? (i.e. I would declare that MYSTOCK has 3 decimal places), but even when it is guessing perhaps it should guess independently for the amount and the balance? (the balance has no decimal places, fine, but the balance has 3 decimal places, throwing it away looses information).

@edwintorok

@edwintorok, thanks. I will read in more detail when I can. But in case this moves us forward at all: does the following example (possibly tweaked further) capture (one of) the issues ? (I ignored the price field, thinking it may not be essential.)

2020-01-01,21.058,$21.0580

fields date, quantity, price
amount1  0
balance1 %quantity

$ ls a.csv* | entr hledger -f a.csv print
2020-01-01
    expenses:unknown               0 = 21

Or better:

Setting the posting amount to 0 with no commodity symbol or decimal places also hides the balance amount's decimal places (which is bad):

2020-01-01,1.234

fields date, quantity
amount1  0
balance1 %quantity

2020-01-01
    expenses:unknown               0 = 1

It doesn't happen if the posting amount has a commodity symbol:

2020-01-01,1.234

fields date, quantity
amount1  $0
balance1 %quantity

2020-01-01
    expenses:unknown               0 = 1.234

Possible interim workaround: in the amount1 rule, either add a commodity symbol (any), or add the maximum number of decimal places seen in balance amounts ?

PS and yes you are doing the right thing by setting amount to 0 to create a standalone balance assertion.

Setting the posting amount to 0 with no commodity symbol or decimal places also hides the balance amount's decimal places

Assigning a literal amount value in rules sets a commodity style which can interfere in unwanted ways with other amounts, eg hiding decimal places in a balance assertion amount.

Thanks, perhaps it is worth documenting this at least in the CSV import page, to say that for balance assertions you need to provide an amount of 0 with same commodity symbol, or with the desired number of decimal places.
Not sure whether there is anything hledger itself could do better, perhaps warn if an amount of 0 is used in a rules file, since it is likely not what the user intended?

Oh I do think it's a bug, the behaviour is surprising. More detailed debugging needed. Good find.

Thanks, perhaps it is worth documenting this at least in the CSV import page, to say that for balance assertions you need to provide an amount of 0 with same commodity symbol, or with the desired number of decimal places.

Currently, hledger print -x produces lines that say 0 = $20.25 for balance assertions: without a commodity symbol and without decimals. So I don't think that's a mistake in the docs.

Contrary to what I said above, today I think ~~the~~ a basic issue here is that balance assertion amounts get their precision limited by the commodity display style, like all other amounts. Which seems to contradict https://hledger.org/hledger.html#commodity-display-style, which says "For the amounts in each commodity, hledger chooses a consistent display style. (Except for price amounts, which are always displayed as written)."
The print command does not seem to implement that. [Correction: I misread. Price amounts indeed were not rounded.]

Here's another example using just a journal file:

# display A with no decimal places:
commodity A 1.

2021-01-01
  (a)   0.001A
  (a)   1.122A = 1.123A

$ hledger print
2021-01-01
    (a)               0
    (a)             A 1 = A 1

If I make print preserve the full precision of assertion/assignment amounts, it looks like this:

$ hledger print
2021-01-01
    (a)               0
    (a)             A 1 = A 1.123

Which I guess is an improvement, and makes it doc-compliant. [Fixed in master.]

The entry still looks weird of course - though it is actually balanced, it doesn't look that way when displayed with the requested commodity display style. And hledger won't be able to re-parse this entry. That is issue #931.

Two more issues:

However it is very unexpected that one commodity has no decimals guessed, and the other one has 4

This is normal hledger behaviour, at least. Each commodity has its own display style. The amount1 and quantity amounts are in the "no-symbol" commodity, while the price is in the $ commodity, and their styles are inferred separately.

Secondly: for the no-symbol commodity it has inferred zero decimal places, based on only the 0 amount in the amount1 0 rule I guess. But we would expect it to also consider the amounts in the CSV, eg the 21.058, and therefore to infer three decimal places for the no-symbol commodity.

However.. numbers in the CSV don't necessarily have a commodity symbol attached. Sometimes it's in another field, sometimes we add it in the rules. So in general it's a little unclear which CSV amounts should be used to infer the number of decimal places for which commodities. Any thoughts ? [And consider that CSV can contain many number fields, with different numbers of decimal places.]

That said.. in the usual case where you're using the CSV amount as a posting amount, all the decimals are copied to the journal entry and then the display style is inferred as you'd expect.

Here the trouble arises because 1. the posting amount is hard coded to 0 with no decimals and 2. there are no other records being processed, which if some of them used the CSV amount as posting amount would cause the display style to use more decimal places. Eg:

12/31/2018,20.000,
12/31/2018,1.058,
12/31/2018,,21.058

date-format %m/%d/%Y
fields date, amt, balance

account1 a
amount  %amt

if %balance .
 amount1 0

$ hledger print
2018-12-31
    a                       20.000
    income:unknown         -20.000

2018-12-31
    a                        1.058
    income:unknown          -1.058

2018-12-31
    a               0 = 21.058

So I think this part of it is specific to this minimal example, and not much of a problem in practice.

print: show all decimal places in assertion/assignment amounts (#1465)

So I think that fix I just pushed is enough to resolve this issue, let me know.

;csv: doc: note how CSV amounts affect display style (#1465)

print: don't add decimal places in assertion/assignment amounts either (

;journal: doc: wording tweak for #1465

Update/summary:

It would be better for print to always show all decimal places. In master, now it does.
For other commands: the number of decimal places displayed is influenced by the decimal places in the rule-generated journal amounts, but not by the original amounts in the CSV. This has been clarified in the CSV Tips doc.

;journal: doc: wording tweak for #1465

Navigation Menu

Repository navigation

Inconsistent rounding for balance CSV rules (due to `amount1 0`) #1465

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Inconsistent rounding for balance CSV rules (due to amount1 0) #1465

Description

Activity

simonmichael commented on Jan 17, 2021

simonmichael commented on Jan 17, 2021

edwintorok commented on Jan 17, 2021

edwintorok commented on Jan 17, 2021

simonmichael commented on Jan 18, 2021

simonmichael commented on Jan 18, 2021

simonmichael commented on Jan 18, 2021

simonmichael commented on Jan 18, 2021

simonmichael commented on Jan 18, 2021

edwintorok commented on Jan 18, 2021

simonmichael commented on Jan 18, 2021

the-solipsist commented on Jan 26, 2021

simonmichael commented on Feb 4, 2021

simonmichael commented on Feb 4, 2021

simonmichael commented on Feb 4, 2021

simonmichael commented on Feb 4, 2021

simonmichael commented on Feb 4, 2021

simonmichael commented on Feb 4, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Inconsistent rounding for balance CSV rules (due to `amount1 0`) #1465