| draft-dutta-gcmf-00.txt | draft-dutta-gcmf-01.txt | |||
|---|---|---|---|---|
| Independent Research S. Dutta | Independent Research S. Dutta | |||
| Internet-Draft The ZSMC Co. | Internet-Draft The ZSMC Co. | |||
| Intended status: Experimental 29 August 2026 | Intended status: Experimental 29 August 2026 | |||
| Expires: 2 March 2027 | Expires: 2 March 2027 | |||
| General-Purpose Compression via Mathematical Functions (GCMF) | General-Purpose Compression via Mathematical Functions (GCMF) | |||
| draft-dutta-gcmf-00 | draft-dutta-gcmf-01 | |||
| Abstract | Abstract | |||
| This document specifies General-Purpose Compression via Mathematical | This document specifies General-Purpose Compression via Mathematical | |||
| Functions (GCMF), a lossless compression format that represents | Functions (GCMF), a lossless compression format that represents | |||
| sequences of data using mathematical functions and associated | sequences of data using mathematical functions and associated | |||
| parameters. | parameters. | |||
| GCMF attempts to represent a sequence using a compact mathematical | GCMF attempts to represent a sequence using a compact mathematical | |||
| representation rather than storing every value explicitly. This | representation rather than storing every value explicitly. This | |||
| skipping to change at page 2, line 10 ¶ | skipping to change at page 2, line 10 ¶ | |||
| Provisions Relating to IETF Documents (https://trustee.ietf.org/ | Provisions Relating to IETF Documents (https://trustee.ietf.org/ | |||
| license-info) in effect on the date of publication of this document. | license-info) in effect on the date of publication of this document. | |||
| Please review these documents carefully, as they describe your rights | Please review these documents carefully, as they describe your rights | |||
| and restrictions with respect to this document. | and restrictions with respect to this document. | |||
| Table of Contents | Table of Contents | |||
| 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
| 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 | 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
| 2. File Extension and MIME Type . . . . . . . . . . . . . . . . 3 | 2. File Extension and MIME Type . . . . . . . . . . . . . . . . 3 | |||
| 3. Function Representation Model . . . . . . . . . . . . . . . . 3 | 3. Function Representation Model . . . . . . . . . . . . . . . . 4 | |||
| 3.1. Source Size Comparison . . . . . . . . . . . . . . . . . 4 | 3.1. Source Size Comparison . . . . . . . . . . . . . . . . . 5 | |||
| 4. GCMF Binary Format Specification . . . . . . . . . . . . . . 5 | 4. GCMF Binary Format Specification . . . . . . . . . . . . . . 5 | |||
| 4.1. File Structure . . . . . . . . . . . . . . . . . . . . . 5 | 4.1. File Structure . . . . . . . . . . . . . . . . . . . . . 6 | |||
| 4.2. Magic Number . . . . . . . . . . . . . . . . . . . . . . 6 | 4.2. Magic Number . . . . . . . . . . . . . . . . . . . . . . 6 | |||
| 4.3. Version . . . . . . . . . . . . . . . . . . . . . . . . . 6 | 4.3. Version . . . . . . . . . . . . . . . . . . . . . . . . . 7 | |||
| 4.4. Function Address . . . . . . . . . . . . . . . . . . . . 6 | 4.4. Function Address . . . . . . . . . . . . . . . . . . . . 7 | |||
| 4.4.1. Residual Flag . . . . . . . . . . . . . . . . . . . . 6 | 4.4.1. Residual Flag . . . . . . . . . . . . . . . . . . . . 7 | |||
| 4.4.2. Function Type Address . . . . . . . . . . . . . . . . 7 | 4.4.2. Function Type Address . . . . . . . . . . . . . . . . 7 | |||
| 4.5. Bit Packing . . . . . . . . . . . . . . . . . . . . . . . 7 | 4.5. Bit Packing . . . . . . . . . . . . . . . . . . . . . . . 7 | |||
| 4.6. Variable Length Integer Encoding . . . . . . . . . . . . 7 | 4.6. Variable Length Integer Encoding . . . . . . . . . . . . 8 | |||
| 4.7. Signed Integer Encoding . . . . . . . . . . . . . . . . . 8 | 4.7. Signed Integer Encoding . . . . . . . . . . . . . . . . . 8 | |||
| 4.8. CUSTOM Function . . . . . . . . . . . . . . . . . . . . . 8 | 4.8. CUSTOM Function . . . . . . . . . . . . . . . . . . . . . 8 | |||
| 4.9. Predefined Functions . . . . . . . . . . . . . . . . . . 9 | 4.9. Predefined Functions . . . . . . . . . . . . . . . . . . 10 | |||
| 4.9.1. Predefined Function Parameter Encoding . . . . . . . 10 | 4.9.1. Predefined Function Parameter Encoding . . . . . . . 11 | |||
| 4.10. Function Bytecode . . . . . . . . . . . . . . . . . . . . 11 | 4.10. Function Bytecode . . . . . . . . . . . . . . . . . . . . 12 | |||
| 4.11. Stack Machine Requirements . . . . . . . . . . . . . . . 11 | 4.11. Stack Machine Requirements . . . . . . . . . . . . . . . 12 | |||
| 4.11.1. Numerical Evaluation . . . . . . . . . . . . . . . . 12 | 4.11.1. Numerical Evaluation . . . . . . . . . . . . . . . . 13 | |||
| 4.12. Example Custom Function . . . . . . . . . . . . . . . . . 12 | 4.12. Example Custom Function . . . . . . . . . . . . . . . . . 13 | |||
| 4.13. X_start and X_end . . . . . . . . . . . . . . . . . . . . 12 | 4.13. X_start and X_end . . . . . . . . . . . . . . . . . . . . 13 | |||
| 4.14. Base Encoding . . . . . . . . . . . . . . . . . . . . . . 13 | 4.14. Base Encoding . . . . . . . . . . . . . . . . . . . . . . 14 | |||
| 4.15. Base Interpretation . . . . . . . . . . . . . . . . . . . 13 | 4.15. Base Interpretation . . . . . . . . . . . . . . . . . . . 14 | |||
| 4.16. Residual Representation . . . . . . . . . . . . . . . . . 14 | 4.16. Residual Representation . . . . . . . . . . . . . . . . . 15 | |||
| 4.17. Residual Function Container . . . . . . . . . . . . . . . 14 | 4.17. Residual Function Container . . . . . . . . . . . . . . . 15 | |||
| 4.18. Literal . . . . . . . . . . . . . . . . . . . . . . . . . 15 | 4.18. Literal . . . . . . . . . . . . . . . . . . . . . . . . . 16 | |||
| 4.19. Physical End Of File . . . . . . . . . . . . . . . . . . 16 | 4.19. Physical End Of File . . . . . . . . . . . . . . . . . . 17 | |||
| 4.20. Complete Canonical Layout . . . . . . . . . . . . . . . . 16 | 4.20. Complete Canonical Layout . . . . . . . . . . . . . . . . 17 | |||
| 4.21. Encoding Example . . . . . . . . . . . . . . . . . . . . 17 | 4.21. Encoding Example . . . . . . . . . . . . . . . . . . . . 18 | |||
| 4.22. Encoder Selection Algorithm . . . . . . . . . . . . . . . 18 | 4.22. Encoder Selection Algorithm . . . . . . . . . . . . . . . 19 | |||
| 4.23. Literal Selection . . . . . . . . . . . . . . . . . . . . 18 | 4.23. Literal Selection . . . . . . . . . . . . . . . . . . . . 19 | |||
| 4.24. Decoder State Machine . . . . . . . . . . . . . . . . . . 18 | 4.24. Decoder State Machine . . . . . . . . . . . . . . . . . . 20 | |||
| 4.25. Decoder Validation . . . . . . . . . . . . . . . . . . . 19 | 4.25. Decoder Validation . . . . . . . . . . . . . . . . . . . 21 | |||
| 4.26. Canonical Encoding . . . . . . . . . . . . . . . . . . . 21 | 4.26. Canonical Encoding . . . . . . . . . . . . . . . . . . . 23 | |||
| 4.27. Versioning . . . . . . . . . . . . . . . . . . . . . . . 21 | 4.27. Versioning . . . . . . . . . . . . . . . . . . . . . . . 23 | |||
| 4.28. Reserved Space . . . . . . . . . . . . . . . . . . . . . 22 | 4.28. Reserved Space . . . . . . . . . . . . . . . . . . . . . 24 | |||
| 4.29. Losslessness Requirement . . . . . . . . . . . . . . . . 22 | 4.29. Losslessness Requirement . . . . . . . . . . . . . . . . 24 | |||
| 4.30. Summary of Binary Grammar . . . . . . . . . . . . . . . . 22 | 4.30. Summary of Binary Grammar . . . . . . . . . . . . . . . . 24 | |||
| 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 | 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 | |||
| 6. Security Considerations . . . . . . . . . . . . . . . . . . . 24 | 6. Security Considerations . . . . . . . . . . . . . . . . . . . 26 | |||
| 7. Normative References . . . . . . . . . . . . . . . . . . . . 25 | 7. Normative References . . . . . . . . . . . . . . . . . . . . 27 | |||
| 8. Informative References . . . . . . . . . . . . . . . . . . . 25 | 8. Informative References . . . . . . . . . . . . . . . . . . . 27 | |||
| Appendix A. Example GCMF Representation . . . . . . . . . . . . 25 | Appendix A. Example GCMF Representation . . . . . . . . . . . . 27 | |||
| Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 26 | Appendix B. Changes from draft-dutta-gcmf-00 . . . . . . . . . . 30 | |||
| Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 32 | ||||
| 1. Introduction | 1. Introduction | |||
| General-Purpose Compression via Mathematical Functions (GCMF) is a | General-Purpose Compression via Mathematical Functions (GCMF) is a | |||
| lossless compression method that represents data using mathematical | lossless compression method that represents data using mathematical | |||
| functions. | functions. | |||
| Instead of storing every value in a sequence explicitly, GCMF may | Instead of storing every value in a sequence explicitly, GCMF may | |||
| store a function and a range over which the function can reproduce | store a function and a range over which the function can reproduce | |||
| the original values. | the original values. | |||
| The objective of GCMF is to reduce the amount of information required | The objective of GCMF is to reduce the amount of information required | |||
| to represent sequences that exhibit mathematical regularity, in the | to represent sequences that exhibit mathematical regularity, in the | |||
| spirit of minimum-description-length approaches to modeling such as | spirit of minimum-description-length approaches to modeling such as | |||
| [J_RISSANEN_1978]. | [J_RISSANEN_1978]. | |||
| A GCMF encoder MUST evaluate a set of valid candidate representations | A GCMF encoder MUST evaluate a set of valid candidate representations | |||
| supported by the implementation and SHOULD select the candidate | supported by the implementation and SHOULD select the candidate | |||
| having the minimum complete encoded length, within the closed | having the minimum complete encoded length, as described in | |||
| interval [X_start, X_end], as described in Section 3. | Section 3. | |||
| 1.1. Terminology | 1.1. Terminology | |||
| The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | |||
| SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL in this | "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and | |||
| document are to be interpreted as described in [RFC2119]. | "OPTIONAL" in this document are to be interpreted as described in | |||
| BCP 14 [RFC2119] when, and only when, they appear in all capitals, as | ||||
| shown here. | ||||
| This document also uses "OPTIONAL" as a structural label within | ||||
| binary layout diagrams (Section 4) to mark a field that is absent | ||||
| under some, but not all, values of a preceding discriminant field | ||||
| (for example, a Function Address or a predefined-function identity). | ||||
| A field so marked is still mandatory whenever the applicable | ||||
| discriminant requires it; "OPTIONAL" in a diagram never means an | ||||
| implementation may omit the field at its own discretion. Section 4.1 | ||||
| identifies, for each representation, which fields are conditional and | ||||
| on what. | ||||
| 2. File Extension and MIME Type | 2. File Extension and MIME Type | |||
| GCMF currently utilizes a working MIME type of application/gcmf. | GCMF currently utilizes a working MIME type of application/gcmf. | |||
| This is not an officially registered MIME type under [RFC2046], or | This is not an officially registered MIME type under [RFC2046], or | |||
| under any current standard or working group as of the publication of | under any current standard or working group as of the publication of | |||
| this document. It is intended only to identify the file type to | this document. It is intended only to identify the file type to | |||
| adopting implementations. | adopting implementations. | |||
| GCMF also currently utilizes a working file extension of .gcmf. | GCMF also currently utilizes a working file extension of .gcmf. | |||
| skipping to change at page 4, line 4 ¶ | skipping to change at page 4, line 16 ¶ | |||
| Until formal registration occurs, this extension SHOULD be treated | Until formal registration occurs, this extension SHOULD be treated | |||
| with the same significance as the MIME type above. | with the same significance as the MIME type above. | |||
| 3. Function Representation Model | 3. Function Representation Model | |||
| GCMF represents a numerical data sequence as the evaluation of a | GCMF represents a numerical data sequence as the evaluation of a | |||
| mathematical function over a finite domain, optionally augmented by a | mathematical function over a finite domain, optionally augmented by a | |||
| residual function. The original sequence MUST be: | residual function. The original sequence MUST be: | |||
| D = d0, d1, ..., d(n-1) | D = d0, d1, ..., d(n-1) | |||
| A GCMF encoder MUST attempt to determine a function (f), a domain | A GCMF encoder MUST attempt to determine a function (f), a domain | |||
| sequence (X = x0, x1, ..., x(n-1)), and MAY also determine a residual | sequence (X = x0, x1, ..., x(n-1)), and MAY also determine a residual | |||
| function. For a generalized representation: | function. For a generalized representation: | |||
| d(i) = f(x(i)) | d(i) = f(x(i)) | |||
| For an exact representation: | For an exact representation: | |||
| d(i) = f(x(i)) + r(x(i)) | d(i) = f(x(i)) + r(x(i)) | |||
| The exact representation is therefore a special case of the | The exact representation is therefore a special case of the | |||
| generalized representation for which: | generalized representation for which: | |||
| r(x) = 0 | r(x) = 0 | |||
| Let C be the theoretical set of all valid GCMF candidate | Let C be the theoretical set of all valid GCMF candidate | |||
| representations and let C_I be the subset of candidates generated by | representations, and let C_I be the subset of candidates that a | |||
| a particular implementation, where C_I is a subset of C. The encoder | particular implementation is capable of generating, where C_I is a | |||
| optimization objective is: | subset of C. Because a real implementation can only search C_I, its | |||
| encoder optimization objective is: | ||||
| C*_I = arg min L(C), for C in C_I | C*_I = arg min L(C), for C in C_I | |||
| where L(C) is the complete encoded length of candidate C in bytes. | where L(C) is the complete encoded length of candidate C in bytes, as | |||
| defined in Section 3.1. | ||||
| An implementation MAY use heuristic, statistical, symbolic, or other | An implementation MAY use heuristic, statistical, symbolic, or other | |||
| candidate-discovery methods. An implementation is not required to | candidate-discovery methods. An implementation is not required to | |||
| discover every mathematically valid representation. | discover every mathematically valid representation; consequently an | |||
| implementation is not required to find C*, the theoretical global | ||||
| optimum over C, and is only required to find C*_I, the best candidate | ||||
| it is able to construct. | ||||
| A candidate is considered compressive only when its complete encoded | A candidate is considered compressive only when its complete encoded | |||
| representation is smaller than the original input under the source- | representation is smaller than the original input under the source- | |||
| size comparison defined in Section 3.1. The global optimization | size comparison defined in Section 3.1. | |||
| objective is: | ||||
| C* = arg min L(C), for C in C | ||||
| where C is the set of all valid GCMF candidate representations and | ||||
| L(C) is the number of bytes required to encode candidate C. | ||||
| 3.1. Source Size Comparison | 3.1. Source Size Comparison | |||
| For the purpose of determining whether a GCMF representation is | For the purpose of determining whether a GCMF representation is | |||
| compressive, the original input is considered as a sequence of N | compressive, the original input is considered as a sequence of N | |||
| bytes and has a source size of N bytes. | bytes and has a source size of N bytes. | |||
| A GCMF representation is compressive if and only if: | A GCMF representation is compressive if and only if: | |||
| L(C) < N | L(C) < N | |||
| skipping to change at page 5, line 26 ¶ | skipping to change at page 5, line 37 ¶ | |||
| This section defines the binary representation of GCMF. A conforming | This section defines the binary representation of GCMF. A conforming | |||
| GCMF decoder MUST be capable of parsing all mandatory structure | GCMF decoder MUST be capable of parsing all mandatory structure | |||
| defined in this specification. Unless otherwise stated, all bit | defined in this specification. Unless otherwise stated, all bit | |||
| fields are packed from the most-significant bit to the least- | fields are packed from the most-significant bit to the least- | |||
| significant bit. | significant bit. | |||
| GCMF is a lossless representation format. A valid GCMF stream MUST | GCMF is a lossless representation format. A valid GCMF stream MUST | |||
| decode to exactly the byte sequence represented by the encoded data. | decode to exactly the byte sequence represented by the encoded data. | |||
| Every GCMF representation is one of exactly two kinds: a LITERAL | ||||
| representation (Section 4.18) or a mathematical representation | ||||
| (Section 4.8 through Section 4.17). A mathematical representation is | ||||
| further one of exactly two kinds: a CUSTOM function representation, | ||||
| in which the function is defined inline by bytecode, or a PREDEFINED | ||||
| function representation, in which the function is identified by | ||||
| Function Address alone and looked up in the registry of Section 4.9. | ||||
| These branches are mutually exclusive: a single representation never | ||||
| contains both a CUSTOM bytecode body and a PREDEFINED parameter body, | ||||
| and never contains both a mathematical body and a LITERAL payload. | ||||
| The field tables in Section 4.8, Section 4.9, and Section 4.18 give | ||||
| the exact, non-overlapping field sequence for each of these three | ||||
| branches; the combined view in Section 4.20 shows them side by side | ||||
| for comparison only and is not itself a fourth, independent layout. | ||||
| 4.1. File Structure | 4.1. File Structure | |||
| A GCMF file consists of a fixed header followed by a representation- | A GCMF file consists of a fixed header followed by a representation- | |||
| specific payload. The general structure is: | specific payload. The general structure is: | |||
| +--------------------------------+ | +--------------------------------+ | |||
| | Magic | 4 bytes | | | Magic | 4 bytes | | |||
| +--------------------------------+ | +--------------------------------+ | |||
| | Version | 1 byte | | | Version | 1 byte | | |||
| +--------------------------------+ | +--------------------------------+ | |||
| | Function Address | 5 bits | | | Function Address | 5 bits | | |||
| +--------------------------------+ | +--------------------------------+ | |||
| | Representation Data | variable| | | Representation Data | variable, see Sections 4.8, 4.9, 4.18 | | |||
| +--------------------------------+ | +--------------------------------+ | |||
| | Physical EOF | | | Physical EOF | | |||
| +--------------------------------+ | +--------------------------------+ | |||
| The Function Address determines the interpretation of Representation | The Function Address determines the interpretation of Representation | |||
| Data. For mathematical representations, Representation Data consists | Data. For mathematical representations, Representation Data consists | |||
| of the function definition or predefined-function parameters, | of the function body (either inline CUSTOM bytecode or PREDEFINED | |||
| X_start, X_end, BASE, and, when present, the residual representation. | parameters), X_start, X_end, BASE, and, when the residual flag is | |||
| set, the residual representation. For LITERAL, Representation Data | ||||
| For LITERAL, Representation Data consists entirely of the original | consists entirely of the original source bytes after the required | |||
| source bytes after the required alignment padding. | alignment padding. | |||
| Physical EOF is not encoded as a field or byte. The end of the | Physical EOF is not encoded as a field or byte; it is the point at | |||
| physical GCMF stream terminates the final representation. | which the physical GCMF stream ends, and it terminates the final | |||
| representation. There is exactly one Magic field and one Physical | ||||
| EOF per file: unlike some of the illustrative field tables later in | ||||
| this section, the file as a whole never repeats the Magic field or | ||||
| contains an in-band EOF marker (see Section 4.19). | ||||
| The exact fields present after the function address depend on the | The exact fields present after the Function Address depend on the | |||
| selected function type. The LITERAL function type is an exception | selected function type. The LITERAL function type is an exception | |||
| and terminates the structured portion of the stream. | and terminates the structured portion of the stream after three | |||
| padding bits. | ||||
| 4.2. Magic Number | 4.2. Magic Number | |||
| Every GCMF file MUST begin with the following 4 bytes: 0x47 0x43 0x4D | Every GCMF file MUST begin with the following 4 bytes: 0x47 0x43 0x4D | |||
| 0x46. These bytes correspond to the ASCII string "GCMF". A decoder | 0x46. These bytes correspond to the ASCII string "GCMF". A decoder | |||
| MUST reject a file whose first 4 bytes do not equal 0x47 0x43 0x4D | MUST reject a file whose first 4 bytes do not equal 0x47 0x43 0x4D | |||
| 0x46. | 0x46. | |||
| 4.3. Version | 4.3. Version | |||
| skipping to change at page 6, line 49 ¶ | skipping to change at page 7, line 35 ¶ | |||
| where R is the residual flag and AAAA is the function type address. | where R is the residual flag and AAAA is the function type address. | |||
| 4.4.1. Residual Flag | 4.4.1. Residual Flag | |||
| The most-significant bit MUST be the residual flag, where 0 indicates | The most-significant bit MUST be the residual flag, where 0 indicates | |||
| no residual function is present and 1 indicates a residual function | no residual function is present and 1 indicates a residual function | |||
| is present. | is present. | |||
| If R = 1, the residual representation immediately follows the primary | If R = 1, the residual representation immediately follows the primary | |||
| mathematical representation according to the residual encoding rules | mathematical representation according to the residual encoding rules | |||
| in Section 4.16. | in Section 4.16. R MUST be 0 for LITERAL (Section 4.18). | |||
| 4.4.2. Function Type Address | 4.4.2. Function Type Address | |||
| LITERAL + RESIDUAL is reserved and MUST be rejected by a conforming | The four-bit function type address AAAA selects one of the sixteen | |||
| function types listed in Table 1. LITERAL with the residual flag set | ||||
| (address 10001) is reserved and MUST be rejected by a conforming | ||||
| decoder unless a future GCMF version explicitly defines semantics for | decoder unless a future GCMF version explicitly defines semantics for | |||
| it. | it. | |||
| 4.5. Bit Packing | 4.5. Bit Packing | |||
| The Function Address occupies exactly five bits and MUST NOT | The Function Address occupies exactly five bits and MUST NOT | |||
| automatically consume a complete byte. | automatically consume a complete byte. | |||
| The bit immediately following the Function Address is the first bit | The bit immediately following the Function Address is the first bit | |||
| of the next field unless alignment is explicitly required by the | of the next field unless alignment is explicitly required by the | |||
| skipping to change at page 8, line 27 ¶ | skipping to change at page 9, line 11 ¶ | |||
| 4.8. CUSTOM Function | 4.8. CUSTOM Function | |||
| A CUSTOM function has Function Type Address: | A CUSTOM function has Function Type Address: | |||
| 0000 | 0000 | |||
| or, when residual data is present: | or, when residual data is present: | |||
| 10000 | 10000 | |||
| The structure is: | A CUSTOM representation always carries an inline function body; the | |||
| Function Length and Function Bytecode fields below are therefore | ||||
| mandatory whenever Function Address selects CUSTOM, and are never | ||||
| present under any other function type. The structure is: | ||||
| +--------------------------------------------------------------+ | ||||
| | Magic | 4 bytes (0x47434D46) | | ||||
| +--------------------------------------------------------------+ | ||||
| | Version | 1 byte (0x00) | | ||||
| +--------------------------------------------------------------+ | ||||
| | Function Address | 5 bits | | ||||
| +--------------------------------------------------------------+ | ||||
| | Function Length | variable VLI | | ||||
| +--------------------------------------------------------------+ | ||||
| | Function Bytecode | Function Length bits | | ||||
| +--------------------------------------------------------------+ | ||||
| | X_start | variable VLI | | ||||
| +--------------------------------------------------------------+ | ||||
| | X_end | variable VLI | | ||||
| +--------------------------------------------------------------+ | ||||
| | Base | variable VLI | | ||||
| +--------------------------------------------------------------+ | ||||
| | [Residual] | present iff R = 1, see Section 4.17 | | ||||
| +--------------------------------------------------------------+ | ||||
| Physical EOF follows the final field; per Section 4.19 no additional | ||||
| Magic or EOF field is encoded at the end of the representation. | ||||
| +-------------------------------------------+ | ||||
| | Magic | 4 bytes (0x47434D46) | | ||||
| +-------------------------------------------+ | ||||
| | Version | 1 byte (0x00) | | ||||
| +-------------------------------------------+ | ||||
| | Function Address | 5 bits | | ||||
| +-------------------------------------------+ | ||||
| | Function Length | OPTIONAL VLI | | ||||
| +-------------------------------------------+ | ||||
| | Function Bytecode | OPTIONAL, Function | | ||||
| | | Length bits | | ||||
| +-------------------------------------------+ | ||||
| | X_start | variable VLI | | ||||
| +-------------------------------------------+ | ||||
| | X_end | variable VLI | | ||||
| +-------------------------------------------+ | ||||
| | Base | variable VLI | | ||||
| +-------------------------------------------+ | ||||
| | Magic | 4 bytes (0x47434D46)| | ||||
| +-------------------------------------------+ | ||||
| | EOF | | ||||
| +-------------------------------------------+ | ||||
| The Function Length specifies the number of bits occupied by the | The Function Length specifies the number of bits occupied by the | |||
| function bytecode, including opcode bits, operand bits, and the END | function bytecode, including opcode bits, operand bits, and the END | |||
| instruction. | instruction. | |||
| The Function Length MUST be sufficient to contain a complete bytecode | The Function Length MUST be sufficient to contain a complete bytecode | |||
| sequence ending in an END instruction. | sequence ending in an END instruction. | |||
| The decoder MUST NOT read beyond the declared function length while | The decoder MUST NOT read beyond the declared function length while | |||
| executing the function. | executing the function. | |||
| skipping to change at page 9, line 28 ¶ | skipping to change at page 10, line 15 ¶ | |||
| The function bytecode MUST contain a valid END instruction within the | The function bytecode MUST contain a valid END instruction within the | |||
| specified function length. | specified function length. | |||
| 4.9. Predefined Functions | 4.9. Predefined Functions | |||
| A predefined function does not contain its mathematical definition in | A predefined function does not contain its mathematical definition in | |||
| the file. The Function Address identifies the mathematical function | the file. The Function Address identifies the mathematical function | |||
| through the GCMF predefined-function registry. The decoder obtains | through the GCMF predefined-function registry. The decoder obtains | |||
| the function definition from the version-specific registry. | the function definition from the version-specific registry. | |||
| A predefined function therefore has the general structure: | A predefined representation never carries a Function Length or | |||
| Function Bytecode field; those fields are exclusive to CUSTOM | ||||
| (Section 4.8). Depending on which predefined function is selected, | ||||
| the Function Parameters field itself may be empty: Section 4.9.1 | ||||
| defines five predefined functions (FIBONACCI, FACTORIAL, TRIANGULAR, | ||||
| SQUARE, CUBE) that take no parameters, in which case this field is | ||||
| zero-length and X_start immediately follows the Function Address. A | ||||
| predefined function therefore has the general structure: | ||||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | Magic | 4 bytes (0x47434D46) | | | Magic | 4 bytes (0x47434D46) | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | Version | 1 byte (0x00) | | | Version | 1 byte (0x00) | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | Function Address | 5 bits | | | Function Address | 5 bits | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | Function Parameters | variable | | | [Function Parameters] | OPTIONAL, variable -- empty for the | | |||
| +-------------------------------------------+ | | | zero-parameter functions listed | | |||
| | X_start | variable VLI | | | | above, otherwise per Section 4.9.1 | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | X_end | variable VLI | | | X_start | variable VLI | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | Base | variable VLI | | | X_end | variable VLI | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | Magic | 4 bytes (0x47434D46)| | | Base | variable VLI | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| | EOF | | | [Residual] | present iff R = 1, see Section 4.17 | | |||
| +-------------------------------------------+ | +--------------------------------------------------------------+ | |||
| Physical EOF follows the final field; per Section 4.19 no additional | ||||
| Magic or EOF field is encoded at the end of the representation. | ||||
| Each predefined function MUST have a precisely defined parameter | Each predefined function MUST have a precisely defined parameter | |||
| encoding. | encoding. | |||
| 4.9.1. Predefined Function Parameter Encoding | 4.9.1. Predefined Function Parameter Encoding | |||
| Unless otherwise specified, all integer parameters of predefined | Unless otherwise specified, all integer parameters of predefined | |||
| functions are encoded using GCMF-VLI. Parameters that may be | functions are encoded using GCMF-VLI. Parameters that may be | |||
| negative MUST use ZigZag encoding before GCMF-VLI encoding. | negative MUST use ZigZag encoding before GCMF-VLI encoding. | |||
| skipping to change at page 14, line 28 ¶ | skipping to change at page 15, line 28 ¶ | |||
| output. For every value: | output. For every value: | |||
| r(i) = d(i) - f(x(i)) | r(i) = d(i) - f(x(i)) | |||
| The original datum is reconstructed using: | The original datum is reconstructed using: | |||
| d(i) = f(x(i)) + r(i) | d(i) = f(x(i)) + r(i) | |||
| The residual representation MUST itself be encoded using a valid GCMF | The residual representation MUST itself be encoded using a valid GCMF | |||
| function representation. Therefore a residual MAY be a predefined | function representation. Therefore a residual MAY be a predefined | |||
| function, a custom function, or another mathematically represented | function or a custom function, subject to the restrictions imposed by | |||
| sequence, subject to the restrictions imposed by the GCMF version. | this version of GCMF (see Section 4.17, which prohibits a residual | |||
| from itself carrying a further nested residual). | ||||
| The residual representation does not independently require a second | The residual representation does not independently require a second | |||
| Function Address byte unless specified by the residual container | Function Address byte; it uses the four-bit Residual Function Type | |||
| structure. | defined in Section 4.17 instead. | |||
| 4.17. Residual Function Container | 4.17. Residual Function Container | |||
| When R = 1, the residual representation immediately follows the | When R = 1, the residual representation immediately follows the | |||
| complete primary mathematical representation. The residual section | complete primary mathematical representation. The residual section | |||
| is encoded as: | is encoded as: | |||
| +-----------------------------+ | +----------------------------------------------------------+ | |||
| | Residual Function Type | 4 bits | | | Residual Function Type | 4 bits | | |||
| +-----------------------------+ | +----------------------------------------------------------+ | |||
| | Residual Function Data | variable | | | Residual Function Data | variable, mandatory -- see below | | |||
| +-----------------------------+ | +----------------------------------------------------------+ | |||
| The Residual Function Type uses the same function-type address table | The Residual Function Type uses the same function-type address table | |||
| as the primary function, excluding the residual flag. The residual | as the primary function (Table 1), excluding the residual flag. The | |||
| function therefore contains exactly four function-type bits. | residual function therefore contains exactly four function-type bits, | |||
| and Residual Function Data is always present: a custom residual's | ||||
| data is its Function Length and Function Bytecode, and a predefined | ||||
| residual's data is its parameter fields, which may be zero-length for | ||||
| the parameterless functions of Section 4.9.1 but the field itself is | ||||
| never omitted outright. | ||||
| For a custom residual, 0000 is used as the Residual Function Type. A | For a custom residual, 0000 is used as the Residual Function Type. A | |||
| custom residual MUST contain its Function Length followed by its | custom residual MUST contain its Function Length followed by its | |||
| Function Bytecode according to the CUSTOM function encoding rules. | Function Bytecode according to the CUSTOM function encoding rules. | |||
| A predefined residual MUST contain the parameter fields specified for | A predefined residual MUST contain the parameter fields specified for | |||
| its function type. | its function type. | |||
| The Residual Function Type MUST NOT encode LITERAL (0001): a residual | ||||
| is always a mathematical representation, never a literal payload. | ||||
| The residual function MUST NOT contain a residual flag and MUST NOT | The residual function MUST NOT contain a residual flag and MUST NOT | |||
| contain another residual representation. Recursive residual nesting | contain another residual representation. Recursive residual nesting | |||
| is therefore prohibited in GCMF v1. | is therefore prohibited in GCMF v1. | |||
| The residual representation MUST generate exactly the same number of | The residual representation MUST generate exactly the same number of | |||
| values as the primary representation. | values as the primary representation. | |||
| 4.18. Literal | 4.18. Literal | |||
| The LITERAL function has Function Type Address: | The LITERAL function has Function Type Address: | |||
| skipping to change at page 15, line 36 ¶ | skipping to change at page 17, line 5 ¶ | |||
| Address is therefore: | Address is therefore: | |||
| 00001 | 00001 | |||
| LITERAL is a terminal representation. Immediately following the | LITERAL is a terminal representation. Immediately following the | |||
| Function Address, the decoder MUST consume three zero-valued padding | Function Address, the decoder MUST consume three zero-valued padding | |||
| bits. These three padding bits align the stream to the next byte | bits. These three padding bits align the stream to the next byte | |||
| boundary. The remainder of the GCMF stream is then interpreted as | boundary. The remainder of the GCMF stream is then interpreted as | |||
| raw source bytes. | raw source bytes. | |||
| +----------------------+ | +--------------------------+ | |||
| | Magic | | | Magic | | |||
| +----------------------+ | +--------------------------+ | |||
| | Version | | | Version | | |||
| +----------------------+ | +--------------------------+ | |||
| | Function Address | | | Function Address | | |||
| +----------------------+ | +--------------------------+ | |||
| | Raw source bytes | | | Zero padding (3 bits) | | |||
| | Raw source bytes | | +--------------------------+ | |||
| | Raw source bytes | | | Raw source bytes | | |||
| | ... | | +--------------------------+ | |||
| +----------------------+ | | Physical EOF | | |||
| | EOF | | +--------------------------+ | |||
| +----------------------+ | ||||
| No function length is stored. No X_start is stored. No X_end is | No function length is stored. No X_start is stored. No X_end is | |||
| stored. No BASE field is stored. No residual representation is | stored. No BASE field is stored. No residual representation is | |||
| stored. | stored. There is no trailing Magic field or in-band EOF field; per | |||
| Section 4.19, the physical end of the stream is the only termination | ||||
| condition. | ||||
| The decoder MUST copy the literal payload directly to the output | The decoder MUST copy the literal payload directly to the output | |||
| without interpretation or transformation. | without interpretation or transformation. | |||
| 4.19. Physical End Of File | 4.19. Physical End Of File | |||
| The physical end of the GCMF file is the termination condition for | The physical end of the GCMF file is the termination condition for | |||
| the final field. No EOF byte or other in-band EOF marker is defined | the final field. No EOF byte or other in-band EOF marker is defined | |||
| by GCMF. | by GCMF, and the Magic field defined in Section 4.2 is never repeated | |||
| within a representation. | ||||
| This is particularly important for LITERAL, because every byte value | This is particularly important for LITERAL, because every byte value | |||
| from 0x00 through 0xFF is valid source data. Consequently, no byte | from 0x00 through 0xFF is valid source data. Consequently, no byte | |||
| value MAY be reserved as an EOF marker. | value MAY be reserved as an EOF marker. | |||
| For LITERAL, EOF is the physical end of the GCMF stream. The final | For LITERAL, EOF is the physical end of the GCMF stream. The final | |||
| byte of the file is therefore the final literal source byte. | byte of the file is therefore the final literal source byte. | |||
| A transport or container system embedding GCMF MUST preserve the | A transport or container system embedding GCMF MUST preserve the | |||
| exact file length. | exact file length. | |||
| 4.20. Complete Canonical Layout | 4.20. Complete Canonical Layout | |||
| For a normal mathematical representation without residual data: | This section restates, side by side and for reference only, the three | |||
| mutually exclusive field sequences already given in full and without | ||||
| ambiguity in Section 4.8 (CUSTOM), Section 4.9 (PREDEFINED), and | ||||
| Section 4.18 (LITERAL). A single GCMF representation always matches | ||||
| exactly one of these three sequences; the bracketed items below are | ||||
| conditional exactly as specified in the section cited, not | ||||
| independently optional. | ||||
| CUSTOM mathematical representation (Section 4.8): | ||||
| GCMF | GCMF | |||
| VERSION | VERSION | |||
| FUNCTION ADDRESS | FUNCTION ADDRESS | |||
| [FUNCTION LENGTH] | FUNCTION LENGTH | |||
| [FUNCTION BYTECODE] | FUNCTION BYTECODE | |||
| [FUNCTION PARAMETERS] | ||||
| X_START | X_START | |||
| X_END | X_END | |||
| BASE | BASE | |||
| [RESIDUAL FUNCTION TYPE, RESIDUAL FUNCTION DATA] -- iff R = 1 | ||||
| EOF | EOF | |||
| For a representation containing a residual: | PREDEFINED mathematical representation (Section 4.9): | |||
| GCMF | GCMF | |||
| VERSION | VERSION | |||
| FUNCTION ADDRESS | FUNCTION ADDRESS | |||
| [FUNCTION LENGTH] | [FUNCTION PARAMETERS] -- zero-length for the parameterless | |||
| [FUNCTION BYTECODE] | functions listed in Section 4.9.1 | |||
| [FUNCTION PARAMETERS] | ||||
| X_START | X_START | |||
| X_END | X_END | |||
| BASE | BASE | |||
| RESIDUAL FUNCTION | [RESIDUAL FUNCTION TYPE, RESIDUAL FUNCTION DATA] -- iff R = 1 | |||
| RESIDUAL DATA | ||||
| EOF | EOF | |||
| For LITERAL: | LITERAL representation (Section 4.18): | |||
| GCMF | GCMF | |||
| VERSION | VERSION | |||
| 00001 | 00001 | |||
| ZERO PADDING (3 BITS) | ||||
| RAW DATA | RAW DATA | |||
| EOF | EOF | |||
| 4.21. Encoding Example | 4.21. Encoding Example | |||
| Consider the sequence: | Consider the sequence: | |||
| 10, 13, 16, 19, 22, 25 | 10, 13, 16, 19, 22, 25 | |||
| It can be represented by the linear function f(x) = 3x + 10 over the | It can be represented by the linear function f(x) = 3x + 10 over the | |||
| domain X = 0, 1, 2, 3, 4, 5. The logical GCMF representation is | domain X = 0, 1, 2, 3, 4, 5. This is a PREDEFINED representation | |||
| therefore: | (LINEAR), so it follows the PREDEFINED field sequence of Section 4.9 | |||
| with Function Parameters = (a=3, b=10) and no residual. The logical | ||||
| GCMF representation is therefore: | ||||
| MAGIC | MAGIC | |||
| VERSION | VERSION | |||
| FUNCTION ADDRESS = LINEAR | FUNCTION ADDRESS = LINEAR, R = 0 | |||
| a = 3 | a = 3 | |||
| b = 10 | b = 10 | |||
| X_START = 0 | X_START = 0 | |||
| X_END = 5 | X_END = 5 | |||
| BASE | BASE | |||
| PHYSICAL EOF | PHYSICAL EOF | |||
| The decoder evaluates: | The decoder evaluates: | |||
| f(0) = 10 | f(0) = 10 | |||
| skipping to change at page 19, line 16 ¶ | skipping to change at page 21, line 16 ¶ | |||
| | | | | |||
| v | v | |||
| READ MAGIC | READ MAGIC | |||
| | | | | |||
| v | v | |||
| READ VERSION | READ VERSION | |||
| | | | | |||
| v | v | |||
| READ FUNCTION ADDRESS | READ FUNCTION ADDRESS | |||
| | | | | |||
| +-----> LITERAL -----> COPY REMAINDER -----> END | +-----> LITERAL --> SKIP 3 PADDING BITS --> COPY REMAINDER --> END | |||
| | | | | |||
| v | v | |||
| READ FUNCTION | [not LITERAL] | |||
| | | | | |||
| v | v | |||
| READ PARAMETERS | READ FUNCTION (CUSTOM: length + bytecode; PREDEFINED: parameters) | |||
| | | | | |||
| v | v | |||
| READ X_start | READ X_start | |||
| | | | | |||
| v | v | |||
| READ X_end | READ X_end | |||
| | | | | |||
| v | v | |||
| READ BASE | READ BASE | |||
| | | | | |||
| +-----> R = 0 -----> EVALUATE | +-----> R = 0 -----> EVALUATE -----> END | |||
| | | | | |||
| +-----> R = 1 -----> READ RESIDUAL | +-----> R = 1 -----> READ RESIDUAL | |||
| | | | | |||
| v | v | |||
| EVALUATE RESIDUAL | EVALUATE RESIDUAL | |||
| | | | | |||
| v | v | |||
| RECONSTRUCT | RECONSTRUCT | |||
| | | | | |||
| v | v | |||
| END | END | |||
| 4.25. Decoder Validation | 4.25. Decoder Validation | |||
| A decoder MUST reject a GCMF stream if any of the following occurs: | A decoder MUST reject a GCMF stream if any of the following occurs: | |||
| * The magic number is invalid. | * The magic number is invalid. | |||
| * The version is unsupported. | * The version is unsupported. | |||
| * The Function Address is invalid. | * The Function Address is invalid, including LITERAL with the | |||
| residual flag set (Section 4.4.2). | ||||
| * A required VLI is truncated. | * A required VLI is truncated. | |||
| * A VLI is malformed. | * A VLI is malformed. | |||
| * A custom function exceeds its declared length. | * A custom function exceeds its declared length. | |||
| * A custom function lacks a valid END. | * A custom function lacks a valid END. | |||
| * A function causes stack underflow. | * A function causes stack underflow. | |||
| * A function terminates with an invalid stack state. | * A function terminates with an invalid stack state. | |||
| * X_end is less than X_start. | * X_end is less than X_start. | |||
| * BASE is less than 2. | * BASE is less than 2. | |||
| * A required function parameter is missing. | * A required function parameter is missing. | |||
| * A residual representation is malformed. | * A residual representation is malformed, including a Residual | |||
| Function Type of LITERAL (Section 4.17). | ||||
| * A prohibited recursive residual is encountered. | * A prohibited recursive residual is encountered. | |||
| * The mathematical evaluation produces a value that cannot be | * The mathematical evaluation produces a value that cannot be | |||
| represented by the selected numerical representation. | represented by the selected numerical representation. | |||
| * The stream terminates before all mandatory fields are read. | * The stream terminates before all mandatory fields are read. | |||
| A decoder SHOULD also impose implementation-defined limits on: | A decoder SHOULD also impose implementation-defined limits on: | |||
| skipping to change at page 23, line 36 ¶ | skipping to change at page 25, line 28 ¶ | |||
| FUNCTION := | FUNCTION := | |||
| CUSTOM_FUNCTION | CUSTOM_FUNCTION | |||
| | PREDEFINED_FUNCTION | | PREDEFINED_FUNCTION | |||
| CUSTOM_FUNCTION := | CUSTOM_FUNCTION := | |||
| FUNCTION_LENGTH | FUNCTION_LENGTH | |||
| FUNCTION_BYTECODE | FUNCTION_BYTECODE | |||
| PREDEFINED_FUNCTION := | PREDEFINED_FUNCTION := | |||
| FUNCTION_PARAMETERS | [FUNCTION_PARAMETERS] | |||
| RESIDUAL := | RESIDUAL := | |||
| RESIDUAL_FUNCTION_TYPE | RESIDUAL_FUNCTION_TYPE | |||
| RESIDUAL_FUNCTION_DATA | RESIDUAL_FUNCTION_DATA | |||
| CUSTOM_RESIDUAL := | RESIDUAL_FUNCTION_DATA := | |||
| 0000 | CUSTOM_RESIDUAL_DATA | |||
| | PREDEFINED_RESIDUAL_DATA | ||||
| CUSTOM_RESIDUAL_DATA := | ||||
| FUNCTION_LENGTH | FUNCTION_LENGTH | |||
| FUNCTION_BYTECODE | FUNCTION_BYTECODE | |||
| PREDEFINED_RESIDUAL_DATA := | ||||
| [FUNCTION_PARAMETERS] | ||||
| PHYSICAL_EOF := | PHYSICAL_EOF := | |||
| END_OF_PHYSICAL_GCMF_STREAM | END_OF_PHYSICAL_GCMF_STREAM | |||
| The fundamental mathematical decoding equation is: | The fundamental mathematical decoding equation is: | |||
| d(i) = f(x(i)) + r(x(i)) | d(i) = f(x(i)) + r(x(i)) | |||
| where r(x(i)) = 0 for representations without residuals. For | where r(x(i)) = 0 for representations without residuals. For | |||
| LITERAL, Decode(G) = LiteralPayload(G). No additional EOF field is | LITERAL, Decode(G) = LiteralPayload(G). No additional EOF field is | |||
| encoded. | encoded. | |||
| skipping to change at page 24, line 22 ¶ | skipping to change at page 26, line 22 ¶ | |||
| registration action from IANA at this time. Should GCMF advance | registration action from IANA at this time. Should GCMF advance | |||
| toward broader adoption, a future document MAY request registration | toward broader adoption, a future document MAY request registration | |||
| of application/gcmf in accordance with the procedures of [RFC2046] | of application/gcmf in accordance with the procedures of [RFC2046] | |||
| and BCP 13. | and BCP 13. | |||
| 6. Security Considerations | 6. Security Considerations | |||
| GCMF CUSTOM function representations embed executable stack-based | GCMF CUSTOM function representations embed executable stack-based | |||
| bytecode within the file format, as described in Section 4.10. A | bytecode within the file format, as described in Section 4.10. A | |||
| decoder that evaluates untrusted GCMF input therefore executes | decoder that evaluates untrusted GCMF input therefore executes | |||
| untrusted, attacker- supplied instructions. Implementations MUST | untrusted, attacker-supplied instructions. Implementations MUST | |||
| treat GCMF decoding of CUSTOM and MODULAR functions as execution of | treat GCMF decoding of CUSTOM and MODULAR functions as execution of | |||
| untrusted code and SHOULD apply appropriate sandboxing. | untrusted code and SHOULD apply appropriate sandboxing. | |||
| Because GCMF functions are evaluated over a domain that can be | Because GCMF functions are evaluated over a domain that can be | |||
| specified independently of the resulting output size, a maliciously | specified independently of the resulting output size, a maliciously | |||
| crafted file can specify an extremely large [X_start, X_end] domain, | crafted file can specify an extremely large [X_start, X_end] domain, | |||
| or a MODULAR or POLYNOMIAL function with a large nesting or degree, | or a MODULAR or POLYNOMIAL function with a large nesting or degree, | |||
| in order to force a decoder to perform an excessive amount of | in order to force a decoder to perform an excessive amount of | |||
| computation or produce an excessive volume of output from a very | computation or produce an excessive volume of output from a very | |||
| small input file. This is a form of decompression-bomb risk | small input file. This is a form of decompression-bomb risk | |||
| skipping to change at page 25, line 22 ¶ | skipping to change at page 27, line 22 ¶ | |||
| environments SHOULD apply them at another layer. | environments SHOULD apply them at another layer. | |||
| 7. Normative References | 7. Normative References | |||
| [RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail | [RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail | |||
| Extensions (MIME) Part Two: Media Types", RFC 2046, | Extensions (MIME) Part Two: Media Types", RFC 2046, | |||
| DOI 10.17487/RFC2046, November 1996, | DOI 10.17487/RFC2046, November 1996, | |||
| <https://www.rfc-editor.org/rfc/rfc2046>. | <https://www.rfc-editor.org/rfc/rfc2046>. | |||
| [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | |||
| Requirement Levels", RFC 2119, DOI 10.17487/RFC2119, March | Requirement Levels", BCP 14, RFC 2119, | |||
| 1997, <https://www.rfc-editor.org/rfc/rfc2119>. | DOI 10.17487/RFC2119, March 1997, | |||
| <https://www.rfc-editor.org/rfc/rfc2119>. | ||||
| 8. Informative References | 8. Informative References | |||
| [J_RISSANEN_1978] | [J_RISSANEN_1978] | |||
| Rissanen, J., "Modeling by shortest data description", | Rissanen, J., "Modeling by shortest data description", | |||
| Automatica, vol. 14, no. 5, pp. 465-471, 1978. | Automatica vol. 14, no. 5, pp. 465-471, 1978. | |||
| Appendix A. Example GCMF Representation | Appendix A. Example GCMF Representation | |||
| This appendix reproduces, for reference, the worked example of | This appendix reproduces, for reference, the worked example of | |||
| Section 4.21 in a single consolidated view. | Section 4.21 in a single consolidated view. | |||
| Given the source sequence 10, 13, 16, 19, 22, 25 (six 8-bit unsigned | Given the source sequence 10, 13, 16, 19, 22, 25 (six 8-bit unsigned | |||
| values, an 6-byte source), an encoder implementing this specification | values, a 6-byte source), an encoder implementing this specification | |||
| could produce a LINEAR representation with a = 3, b = 10, X_start = | could produce a LINEAR representation with a = 3, b = 10, X_start = | |||
| 0, X_end = 5, discovering that: | 0, X_end = 5, discovering that: | |||
| f(x) = 3x + 10 | f(x) = 3x + 10 | |||
| f(0) = 10 | f(0) = 10 | |||
| f(1) = 13 | f(1) = 13 | |||
| f(2) = 16 | f(2) = 16 | |||
| f(3) = 19 | f(3) = 19 | |||
| f(4) = 22 | f(4) = 22 | |||
| skipping to change at page 25, line 49 ¶ | skipping to change at page 28, line 4 ¶ | |||
| 0, X_end = 5, discovering that: | 0, X_end = 5, discovering that: | |||
| f(x) = 3x + 10 | f(x) = 3x + 10 | |||
| f(0) = 10 | f(0) = 10 | |||
| f(1) = 13 | f(1) = 13 | |||
| f(2) = 16 | f(2) = 16 | |||
| f(3) = 19 | f(3) = 19 | |||
| f(4) = 22 | f(4) = 22 | |||
| f(5) = 25 | f(5) = 25 | |||
| which exactly reproduces the six-value source sequence with an exact | which exactly reproduces the six-value source sequence with an exact | |||
| (residual-free) representation, per Section 3. Per Section 4.22, the | (residual-free) representation, per Section 3. Per Section 4.22, the | |||
| encoder would compare the total encoded length of this LINEAR | encoder would compare the total encoded length of this LINEAR | |||
| representation, including all structural overhead, against the | representation, including all structural overhead, against the | |||
| LITERAL representation of the same six bytes, and select whichever | LITERAL representation of the same six bytes, and select whichever | |||
| candidate yields the smaller value of L(C). | candidate yields the smaller value of L(C). | |||
| +======+=============+==========================================+ | ||||
| | AAAA | Function | Formula | | ||||
| +======+=============+==========================================+ | ||||
| | 0000 | CUSTOM | User-defined | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0001 | LITERAL | N/A -- raw bytes, see Section 4.18 | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0010 | CONSTANT | F(x) = c | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0011 | LINEAR | F(x) = ax + b | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0100 | ARITHMETIC | F(x) = a + xd | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0101 | GEOMETRIC | F(x) = a r^x | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0110 | POLYNOMIAL | F(x) = sum_(i=0)^n a_i x^i | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 0111 | EXPONENTIAL | F(x) = a b^x | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1000 | LOGARITHMIC | F(x) = a log_b(x) + c | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1001 | POWER | F(x) = a x^b + c | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1010 | MODULAR | F(x) = G(x) mod m | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1011 | FIBONACCI | F(x)=F(x-1)+F(x-2), x>=2, F(0)=0, F(1)=1 | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1100 | FACTORIAL | F(x) = x! | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1101 | TRIANGULAR | F(x) = x(x+1)/2 | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1110 | SQUARE | F(x) = x^2 | | ||||
| +------+-------------+------------------------------------------+ | ||||
| | 1111 | CUBE | F(x) = x^3 | | ||||
| +------+-------------+------------------------------------------+ | ||||
| Table 1: Function Type Addresses | ||||
| NOTE: the LITERAL row was previously listed with the formula "F(x) = | ||||
| x", which incorrectly implied LITERAL evaluates a function. Per | ||||
| Section 4.18, LITERAL stores no function, domain, or base; the | ||||
| decoded output is the raw payload copied verbatim (Section 4.29). | ||||
| +========+==========+===========================+ | ||||
| | OPCODE | MNEMONIC | STACK OPERATION | | ||||
| +========+==========+===========================+ | ||||
| | 00000 | CONST | Push constant | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00001 | X | Push current x | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00010 | ADD | a + b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00011 | SUB | a - b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00100 | MUL | a * b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00101 | DIV | a / b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00110 | MOD | a mod b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 00111 | POW | a ^ b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01000 | NEG | -a | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01001 | ABS | |a| | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01010 | SQRT | sqrt(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01011 | LOG | ln(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01100 | LOG10 | log10(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01101 | EXP | e^a | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01110 | SIN | sin(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 01111 | COS | cos(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10000 | TAN | tan(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10001 | ASIN | arcsin(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10010 | ACOS | arccos(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10011 | ATAN | arctan(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10100 | FLOOR | floor(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10101 | CEIL | ceil(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10110 | ROUND | round(a) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 10111 | MIN | min(a, b) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11000 | MAX | max(a, b) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11001 | AND | a & b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11010 | OR | a | b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11011 | XOR | a xor b (bitwise) | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11100 | SHL | a << b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11101 | SHR | a >> b | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11110 | DUP | Duplicate top stack value | | ||||
| +--------+----------+---------------------------+ | ||||
| | 11111 | END | End function | | ||||
| +--------+----------+---------------------------+ | ||||
| Table 2: Function Bytecode Opcodes | ||||
| Appendix B. Changes from draft-dutta-gcmf-00 | ||||
| This appendix is non-normative and summarizes the corrections made | ||||
| relative to draft-dutta-gcmf-00; it will be removed before this | ||||
| document advances. | ||||
| * Section 4.8 (CUSTOM Function): the layout table previously marked | ||||
| Function Length and Function Bytecode as OPTIONAL, which | ||||
| contradicted the surrounding prose (Function Length "MUST be | ||||
| sufficient", bytecode "MUST contain a valid END"). These fields | ||||
| are mandatory for every CUSTOM representation; the table has been | ||||
| corrected and an explanatory sentence added. | ||||
| * Section 4.9 (Predefined Functions): the layout table previously | ||||
| showed Function Parameters as an unmarked, unconditionally present | ||||
| field, even though Section 4.9.1 defines five predefined functions | ||||
| that take no parameters. The field is now correctly marked | ||||
| OPTIONAL/zero-length for those functions, with a cross-reference | ||||
| to Section 4.9.1. | ||||
| * Section 4.8 and Section 4.9: the layout tables previously repeated | ||||
| a trailing "Magic | 4 bytes" and "EOF" field at the end of each | ||||
| representation. This contradicted Section 4.1 (one Magic field | ||||
| per file) and Section 4.19 (no in-band EOF marker is ever | ||||
| encoded). The trailing Magic/EOF rows have been removed and | ||||
| replaced with an explicit forward reference to Section 4.19. | ||||
| * Section 4.20 (Complete Canonical Layout): previously presented a | ||||
| single combined diagram with FUNCTION LENGTH, FUNCTION BYTECODE, | ||||
| and FUNCTION PARAMETERS all bracketed as independently optional, | ||||
| which incorrectly implied a single representation could mix CUSTOM | ||||
| and PREDEFINED fields, or omit them freely. The section now | ||||
| presents the CUSTOM, PREDEFINED, and LITERAL field sequences as | ||||
| three separate, mutually exclusive layouts, consistent with the | ||||
| grammar in Section 4.30. | ||||
| * Section 4.18 (Literal): the layout table previously omitted the | ||||
| 3-bit zero-padding field shown in the surrounding prose, and | ||||
| retained a trailing bare "EOF" row. The table now shows the | ||||
| padding field explicitly and labels the terminal row "Physical | ||||
| EOF" for consistency with Section 4.19. | ||||
| * Section 4.1: added an explicit statement that the file contains | ||||
| exactly one Magic field and no in-band EOF marker, to prevent the | ||||
| per-representation layout tables from being read as implying | ||||
| otherwise. | ||||
| * Section 4.4.2: clarified that AAAA selects among the sixteen | ||||
| entries of Table 1, and gave the reserved LITERAL+RESIDUAL | ||||
| combination its explicit 5-bit encoding (10001) rather than | ||||
| leaving it as a bare 4-bit function-type value inconsistent with | ||||
| the 5-bit Function Address field defined in Section 4.4. | ||||
| * Section 4.17: clarified that Residual Function Data is always | ||||
| present when R = 1 (it is never itself optional), that it may be | ||||
| zero-length only for a parameterless predefined function, and | ||||
| added an explicit prohibition on a Residual Function Type of | ||||
| LITERAL, consistent with Section 4.16's statement that a residual | ||||
| is always a mathematical representation. | ||||
| * Section 4.24 (Decoder State Machine): the LITERAL branch | ||||
| previously omitted the padding-skip step shown in Section 4.18's | ||||
| prose, and the non-LITERAL path had no explicit branch label. | ||||
| Both have been added, and an END state was added after EVALUATE to | ||||
| match the END state already present on the residual path. | ||||
| * Section 4.30 (Summary of Binary Grammar): PREDEFINED_FUNCTION | ||||
| previously expanded to an unconditional FUNCTION_PARAMETERS, | ||||
| inconsistent with Section 4.9.1. It is now bracketed as optional. | ||||
| RESIDUAL_FUNCTION_DATA was added as an explicit production | ||||
| distinguishing CUSTOM_RESIDUAL_DATA from PREDEFINED_RESIDUAL_DATA, | ||||
| since Section 4.17 defines both but the previous grammar only gave | ||||
| a production for the custom case. | ||||
| * Section 1.1 (Terminology): added a paragraph defining how the | ||||
| structural label "OPTIONAL" is used inside binary layout diagrams | ||||
| in Section 4, as distinct from the RFC 2119 keyword OPTIONAL, to | ||||
| prevent the ambiguity that produced the defects above. | ||||
| * Table 1 (Function Type Addresses): the LITERAL row previously | ||||
| listed a formula, "F(x) = x", which is incorrect: Section 4.18 | ||||
| states that LITERAL stores no function, domain, or base, and | ||||
| Decode(G) for LITERAL is defined in Section 4.29 as the raw | ||||
| payload, not a function evaluation. The row has been corrected to | ||||
| read "N/A -- raw bytes, see Section 4.18". | ||||
| * Section 3: removed a duplicated statement of the global | ||||
| optimization objective (arg min over C) that repeated, almost | ||||
| verbatim, the implementation-level objective (arg min over C_I) | ||||
| given immediately before it, and clarified in prose why the two | ||||
| objectives differ (C_I is what an implementation can search; C is | ||||
| the theoretical optimum it is not required to find). | ||||
| Author's Address | Author's Address | |||
| Srijal Dutta | Srijal Dutta | |||
| The ZSMC Co. | The ZSMC Co. | |||
| Gurugram | Gurugram | |||
| Haryana | Haryana | |||
| India | India | |||
| Email: srijal@thezsmc.co | Email: srijal@thezsmc.co | |||
| End of changes. 61 change blocks. | ||||
| 169 lines changed or deleted | 453 lines changed or added | |||
This html diff was produced by rfcdiff 1.49. The latest version is available from https://github.com/ietf-tools/rfcdiff | ||||