okay, so this is made for nixpkgs right. technically maybe this makes more sense to open in nixpkgs but uhh the kdl format is not merged and it's not a "real" issue in nixpkgs yet. this is the closest thing to a formats.kdl repo so i'm posting it on your issue tracker; it's purely an implementation detail of the precise conversion semantics, and don't affect the usability of that PR; this can and should be fixed without derailing that discussion.

let's create a backlink, though:

• formats.kdl: init (usage ergonomics) NixOS/nixpkgs#426828

let
  exp_2 = n: builtins.foldl' (x: _: x * x) 2.0 (builtins.genList throw n);
  inf = exp_2 10;
  nan = inf / inf;

  exotics = {
    inf = inf;
    nan = nan;
    "-inf" = -inf;
  };
in
exotics

these values can and do exist in Nix.

KDL can also represent these values.

The existence of these keywords does not imply that any numbers be represented as IEEE 754 floats. These are simply for clarity and convenience for any implementation that chooses to represent their numbers in this way.

They're technically optional, and i don't think there's any application that needs to be able use these exotic values. However, Nix does represent numbers as floats. Those values can exist in a user's configuration; so we should be able to serialize them.

When passing to this utility, we convert to JSON, which only supports real numbers, and has no notion of IEEE-754. This has made a lot of people very angry and been widely regarded as a bad move.

Nix doesn't throw an error if you use such exotic values in JSON; it just silently coerces them:

let
  document = [
   { name = "exotics"; properties = exotics; }
  ];
in
builtins.toJSON document

{"-inf":null,"inf":null,"nan":null}

So, of course, when converted to KDL...

let
  pkgs = import <nixpkgs> {};

  document-kdl-file = pkgs.runCommand "exotics.kdl" {
    document = builtins.toJSON document;
    passAsFile = ["document"];
  } ''
    jsonkdl --kdl-v2 -- "$documentPath" "$out"
  '';
in
builtins.readFile document-kdl-file

exotics -inf=#null inf=#null nan=#null {
}

...we just get #null, and not #inf/#-inf/#nan, like we're supposed to.

- -inf=#null false=#false inf=#null nan=#null null=#null true=#true {
}

solution 1

jsonkdl could implement a special exotic value syntax. these exotic scalars are only allowed in "value" position, where we already accept an object form, like scalar | { type = str, value = scalar; }. it would be fairly trivial to add a key and extend that schema to scalar | { type = str, value = scalar; } | { type = str; @ = "nan" | "inf" | "-inf"; }.

then, as far as jsonkdl is concerned, this problem would be "resolved". You can now represent exotic IEEE-754 values.

but how do we handle that in nixpkgs?? how do we coerce the exotic floats into the sentinels? 'cause, the fact that these values exist in Nix are kind of a bug? NaN is by definition not equal to itself, but Nix sometimes "assumes" it is:

let
  exp_2 = n: builtins.foldl' (x: _: x * x) 2.0 (builtins.genList throw n);
  inf = exp_2 10;

  nan = inf / inf;
in
[ nan ] == [ nan ] # true

let
  exp_2 = n: builtins.foldl' (x: _: x * x) 2.0 (builtins.genList throw n);
  inf = exp_2 10;

  nan  = inf / inf;
  nan' = inf / inf;
in
[ nan ] == [ nan' ] # false

this is not okay. we're not relying on that in nixpkgs. no fucking way.

also, builtins.toString nan == "-nan". i don't know how to make it stringify as unsigned nan. i have no idea why it does that but this is painful to do on purpose so there's no way it's intentional. so, even with toString, we'd be relying on bugs in the underlying representation of floats. or at least i'd consider this buggy.

solution 2

okay, let's say "these values existing in Nix are a bug and you shouldn't use them". sure. then, nixpkgs can expose sentinel values like kdl.lib.exotic_inf, kdl.lib.exotic_-inf, kdl.lib.exotic_nan. (note: these names suck, and can be bikeshedded in nixpkgs if we choose this solution). their value would be like { _type = "nan"; } or something like that; they can be transparently mapped to whatever schema jsonkdl exposes for exotic values.

it would also be abundantly clear with these values that they're not intended to do math with; they are sentinels for the KDL format and that's it.

solution 3

but it kinda sucks that standard Nix nan, inf, -inf values don't work in solution 2. it sucks even more that they will be encoded as null, instead of throwing any error.

JSON is not the only "builtin converter" in Nix. there's also builtins.toXML.

builtins.toXML 0.0

?xml version='1.0' encoding='utf-8'?>
<expr>
  <float value="0" />
</expr>

we can give it all our exotic values:

builtins.toXML {
  "inf" = inf;
  "-inf" = -inf;
  "nan" = nan;
}

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <attrs>
    <attr name="-inf">
      <float value="-inf" />
    </attr>
    <attr name="inf">
      <float value="inf" />
    </attr>
    <attr name="nan">
      <float value="-nan" />
    </attr>
  </attrs>
</expr>

which is yay! it serializes inf and -inf and nan in non-hacky ways

it still calls nan -nan for some reason? like, again, can't really do much about that:

builtins.toXML {
  "nan" = nan;
  "-nan" = -nan;
}

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <attrs>
    <attr name="-nan">
      <float value="-nan" />
    </attr>
    <attr name="nan">
      <float value="-nan" />
    </attr>
  </attrs>
</expr>

the Obvious Solution is then to deprecate jsonkdl in favor of a new contender: nix-xml-kdl (fictional). because yay! xml does it all! let's just builtins.toXML the whole document.

solution 4

actually solution 3 kinda sucks, because builtins.toXML is very different from builtins.toJSON. in particular, builtins.toXML /path/to/whatever actually discards the string context of a path? ???? what. i discovered that just now. why does that discard the path context. this is so broken. lmao.

anyways the big dealbreaker is recursive stuff. if i do let x.x=x; in builtins.toJSON x, the evaluator just dies immediately with error: stack overflow (possible infinite recursion), and crashes the repl. but if i do let x.x=x; in builtins.toXML x, then the evaluator spikes to like way-too-much-memory-usage and takes significantly longer.

builtins.toXML ({ x, y?0, ...}@z: 0)

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <function>
    <attrspat ellipsis="1" name="z">
      <attr name="x" />
      <attr name="y" />
    </attrspat>
  </function>
</expr>

builtins.functionArgs ({ x, y?0, ...}@z: 0)

{
  x = false;
  y = true;
}

builtins.toJSON (builtins.substring 0 1 (builtins.fromJSON "\"\\u01FF\""))

error:
       … while calling the 'toJSON' builtin
         at «string»:1:1:
            1| builtins.toJSON (builtins.substring 0 1 (builtins.fromJSON "\"\\u01FF\""))
             | ^

       error: JSON serialization error: [json.exception.type_error.316] incomplete UTF-8 string; last byte: 0xC7

builtins.toXML (builtins.substring 0 1 (builtins.fromJSON "\"\\u01FF\""))  

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <string value="�" />
</expr>

builtins.toXML "&;<\">\\"

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <string value="&amp;;&lt;&quot;&gt;\" />
</expr>

so it's not okay to do builtins.toXML on the whole document: this sucks in several ways.

what we could do to "alleviate" the pain on the Nix side, is to make the jsonkdl values shaped like scalar | { type = str; value = scalar; } | { type = str; nix-xml-encoded-float = str; } and this weird type exists solely to accept the output to builtins.toXML when encoding a float.

it would accept an XML document with precisely this structure:

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <float value="0" />
</expr>

the only thing that can vary is the value property.

because this is a "normal" serialization of the float, i have much more faith in this not being buggy like Nix otherwise is (for instance, 0.0 is encoded as 0, as opposed to 0.000000 as you'd get from toString).

so of course, it can be -nan, nan, inf, -inf, and in any other case, this allows us to use the exact float representation of the Nix serializer, without roundtripping to a f64 parse of serde_json. that's pretty neat, since kdl-rs can otherwise represent "original text form of numeric literals" pretty well.

we could also do this for ints; which would allow a Nix implementation with arbitrarily large integers (i.e. not github:NixOS/nix) to give us those huge ints, and they'd be preserved in the KDL data. neato!

solution 5

but ahahahahahaha there is a big issue with solution 4 and i am going insane. we lose PRECISION because FOR SOME REASON even though there is no FORCED PRECISION (like in builtins.toString), there is still a MAX PRECISION in builtins.toXML

builtins.toXML 1.000000001

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <float value="1" />
</expr>

it is Deleting Significant Figures. I think this is because it is actually roundtripping to f32 for conversion??? What the fuck??????? I am losing my mind.

JSON doesn't have this issue.

builtins.toJSON 1.000000001

1.000000001

So, actually, we Must use JSON for real numbers. else, it is just simply not precise enough.

Conclusion

i'm thinking the way we handle this is something like the following:

jsonkdl adds a sentinel field with a type like @ = "inf" | "-inf" | "nan". these keywords in KDL cannot otherwise be represented as a JSON scalar value. it would be mutually exclusive with the value field, and exactly one of them must be present. do not support other keywords, because they can already be specified as JSON primitives.

before invoking jsonkdl, nixpkgs should map over each value. if it was a float, it tries to serialize with builtins.toJSON. if that returns the string null, then we know it's actually inf, -inf, nan, or some other arcane bullshit they cooked up while i wasn't looking. in that case, it should try to serialize with builtins.toXML, and compare it against the values returned from nan, inf, and -inf. if it is equal to any of these, it should completely automatically with no interaction necessary convert to the jsonkdl sentinel field, so that jsonkdl knows to generate the given keyword. if the JSON value wasn't null, then just pass it as-is, because the value is already encoded correctly and precisely.