Skip to content

Tags

Tags

v1.7.1

Toggle v1.7.1's commit message
The into_tidb IT waits, before resuming a killed load, until TiDB run…

…s none of its statements: a piece the killed session was still committing made the resumed load repeat it and fail on its key

v1.7.0

Toggle v1.7.0's commit message
1.7.0: OxCaml 5.4.0+ox; quiver mirror copies a database at one TSO (e…

…very definition, collation, placement policy and name checked before anything is made, all copies renamed by one RENAME TABLE), global indexes of partitioned tables, rows of TiDB's first format, reads through the locks of unresolved transactions, holds the source's GC and the target's regions from PD's scheduling, carries or drops placement policies, gathers each copy's statistics, reaches PD and TiKV over TLS

v1.6.1

Toggle v1.6.1's commit message
1.6.1: quiver mirror into another cluster under the copy's ids: both …

…tables described as TiKV keeps them at the versions read and written (Info, from the meta keys; the status port no longer asked), keys and rows renumbered (Renumber: index ids in keys, column ids in rows of the second format re-sorted and in what indexes keep to restore, dropped columns' values and the row checksum left out, padding redone), keys of an index the source does not have public left behind, a row written before a column was added with a default refused (Missing) instead of read as NULL, a schema being changed refused; checked live between two playgrounds after dropped and added columns and indexes; the head of a chain that failed by itself tells the tail Leave before closing its side (it closed first, or raced the thread that took the failure, and the tail waited out the whole reconnect window); a row written before a column was added with a default gets that default written in, as TiDB writes it, instead of the copy being refused (Cell, every type checked byte for byte against rows TiDB rewrote; refused only for a JSON default other than null, a TIMESTAMP of TiDB before 3.0 and other character sets); a partitioned table copied partition by partition, each into the copy's partition of its name (Info reads the partitions and whether they are being rearranged, Renumber moves every key to its partition's id in the copy; one with a global index refused), checked live into another cluster and the same for RANGE, HASH, LIST COLUMNS and KEY tables after truncated, dropped and added partitions

v1.6.0

Toggle v1.6.0's commit message
1.6.0: TiDB 8.5 as a source and a target, and quiver mirror copying T…

…iDB to TiDB as TiKV keys, within a cluster or into another; the README's TiDB section with the mirror against IMPORT INTO FROM SELECT drawn (assets/mirror.svg), the docs site naming TiDB

v1.5.0

Toggle v1.5.0's commit message
1.5.0: a quiverfile's mask at the top reaches dump and copy instead o…

…f the mask command, a list at the top goes only to the commands that take it, and a full example quiverfile with its masking policy (examples/shop.scm)

v1.4.0

Toggle v1.4.0's commit message
1.4.0: values masked as rows are read (--mask, --mask-secret, --mask-…

…strict), a masking policy language of S-expressions with selectors, keep, strict, word lists and dates (--mask-script, expr:, quiver mask), maps in the quiverfile, the user's own statements after the load (--after-load), and documentation generated from the binary (quiver docs)

v1.3.9

Toggle v1.3.9's commit message
1.3.9: quiver builds the secondary keys it can order by default; enco…

…ding test without border seeds

- --tablespace-keys defaults to quiver: on the colleague's copy it was 15% faster than server
  on 1.3.5 and 30% on 1.3.7, every copy equal to its source. server keeps every key for ALTER.
- The encoding controller's property also takes a rung whose next rung toward the bottleneck
  gains under 15%, a step the model's noise hides: CI of the 1.3.7 tag failed on seed 9757516
  (a rung at 79.9% of the best, its neighbour 5.2% better). 300 seeds pass; no rollback, a
  step without a signal, the wrong direction and no doubling of silence are still caught.
- README: keys quiver builds, the default, and --keyless-read.

v1.3.8

Toggle v1.3.8's commit message
1.3.8: a table without a primary key is read in one pass again, built…

… on every renderer

- On the colleague's server (1.3.7) reading through another index cost the source ~7x:
  121-135 reader-seconds against 17 on 1.3.5, reading ended at 17.6-20.4 s instead of
  12-14.5 s, so the shorter tail of table_049 (11 -> 7.1 s) bought nothing. Such a table is
  read whole again by default; --keyless-read ranges keeps the 1.3.6 ranges.
- A part of a table whose runs number their own rows takes its batches into whichever run a
  renderer holds, as it renders: the lowest of 16 free slots, each with its run, closed every
  32 MiB so secondary records sort while the table is still read. One reader's pass is no
  longer built on one thread.
- After a dump or copy the CLI prints the five tables read last: parts, when their reading
  began and ended, reading and waiting.
- cw on the scratch 8.4.11, keys built by quiver, 2 pairs: reading summed 1.7 s against 10 s
  with ranges, reading ended at 1.0 s against 1.4 s, the t049 tail 1.15 s against 1.3 s.

v1.3.7

Toggle v1.3.7's commit message
1.3.7: a table with a primary key is read in its order again; seconda…

…ry merges split by whole records

- 1.3.6 read a table whose primary key cannot bound chunks (a DATE column under --zero-dates keep)
  through FORCE INDEX of another index, so rows came out of clustered order and copy --ingest
  tablespace failed with Tablespace Out_of_order (a colleague's table: PRIMARY KEY (date, int),
  KEY (int, date)). Such a table is read whole again. A primary key with a usable UNIQUE index
  failed the same way since before 1.3.5 (UNIQUE ranges without ORDER BY) and is read whole too.
  The key choice is Provider.key, under a property test against a model of the choice.
- A merge range now ends at a record of the index, not at a 14-byte key prefix, and ranges hold
  equal numbers of records. Equal prefixes could not be split before: in a text(10) key every
  https://www.… URL is the same value, told apart only by DB_ROW_ID, and all of them merged on one
  thread (1.5M rows, 16 threads: the largest range 1.39 → 1.03 of the mean).
- A part's sorted records go to the scratch file in one piece, and the merge reads them up to 32
  pages at a time, at most 64 MiB of buffers in all: 15k reads of 16 KiB → 512 on 3M rows.
- bench/ibd builds and checks a table like the colleague's table_049 without a server.
  finish on 3M rows, 64 parts: 2 threads 0.40 → 0.365 s, 8 threads 0.27–0.35 → 0.16–0.23 s,
  90% URLs on 8 threads 0.45–0.51 → 0.30–0.37 s.

v1.3.6

Toggle v1.3.6's commit message
1.3.6: a table without a primary or unique key is read in parallel th…

…rough another index

- A table with neither key was read by one query on one reader: on a colleague's copy a 10M-row
  table took ~9 s while the other readers idled. Such a table is now chunked by ranges of the
  NOT NULL first column of another visible BTREE index (whole column, integer first, then the
  highest cardinality); ranges are read with FORCE INDEX and no ORDER BY.
- The wire frame is v8: a table's key may be Indexed, so agents and collectors must both be 1.3.6.