User perfs #13134

ornicar · 2023-06-30T12:49:31Z

extract User.perfs to UserPerfs in the scala code. This reduces the User object heap size by 70%. Also the cost of mapping it from a mongodb doc.
user4.perfs to user_perf in mongodb. This reduces the user4.document size by 70%

users are obviously a very commonly used piece of data, and we can make it lighter and faster this way. It does add some code complexity to load the perfs from DB where and when we actually need it.

user4 document example:

{
  _id: 'thibault',
  count: {
    ai: 606,
    draw: 915,
    drawH: 904,
    game: 21067,
    loss: 10048,
    lossH: 9719,
    rated: 18733,
    win: 10099,
    winH: 10020
  },
  createdAt: ISODate("2010-11-22T08:48:00.000Z"),
  enabled: true,
  lang: 'en-US',
  profile: {
    country: 'UA',
    location: 'Saint-Nazaire',
    bio: 'I turn coffee into bugs.',
    firstName: 'Thibault',
    lastName: 'Duplessis',
    links: 'github.com/ornicar\r\nmas.to/@thibault'
  },
  progress: -54,
  rating: 1527,
  roles: [ ],
  seenAt: ISODate("2023-07-04T10:04:07.587Z"),
  toints: 2983,
  username: 'thibault',
  email: 'redacted',
  kid: false,
  colorIt: -2,
  len: 8,
  plan: {
    months: 59,
    active: true,
    since: ISODate("2016-07-19T06:00:13.122Z")
  },
  time: { total: 5854865, tv: 17974 },
  bpass: Binary(Buffer.from("***", "hex"), 0),
  totp: Binary(Buffer.from("b3af02512478deaedbac3c98ebd1c5f6c95aa25b", "hex"), 0),
  marks: []
  perfs: {
    blitz: {
      gl: {
        r: 1868.3251167215603,
        d: 49.93315796357045,
        v: 0.056911382900622524
      },
      nb: 9541,
      re: [
        1868, 1868, 1862,
        1856, 1850, 1843,
        1847, 1851, 1856,
        1856, 1855, 1855
      ],
      la: ISODate("2023-06-03T19:48:51.027Z")
    },
    bullet: {
      gl: {
        r: 1814.4327978754734,
        d: 71.39905018343043,
        v: 0.057798698695270656
      },
      nb: 7131,
      re: [
        1814, 1814, 1811,
        1810, 1837, 1863,
        1853, 1842, 1853,
        1865, 1873, 1879
      ],
      la: ISODate("2023-04-15T18:50:38.966Z")
    },
    chess960: {
      gl: {
        r: 1551.5861781645706,
        d: 66.54509362084751,
        v: 0.05990883146628108
      },
      nb: 348,
      re: [
        1551, 1554, 1555,
        1543, 1530, 1516,
        1501, 1484, 1494,
        1495, 1487, 1490
      ],
      la: ISODate("2018-08-11T21:17:06.170Z")
    },
    classical: {
      gl: {
        r: 1806.5844724202923,
        d: 150.91364413684076,
        v: 0.059885642887367134
      },
      nb: 24,
      re: [ 1805, 1779, 1810 ],
      la: ISODate("2021-04-09T18:55:14.873Z")
    },
    kingOfTheHill: {
      gl: {
        r: 1744.4624163352112,
        d: 67.52911394645916,
        v: 0.060029022004587006
      },
      nb: 94,
      re: [
        1744, 1749, 1728,
        1736, 1732, 1712,
        1695, 1700, 1712,
        1716, 1723, 1730
      ],
      la: ISODate("2017-05-13T15:21:14.345Z")
    },
    puzzle: {
      gl: {
        r: 1874.182944312056,
        d: 66.94134964558688,
        v: 0.0617986485249278
      },
      nb: 5066,
      la: ISODate("2023-07-03T21:17:10.814Z")
    },
    standard: {
      gl: {
        r: 1845.2741254972652,
        d: 62.139244552675784,
        v: 0.05729090838836459
      },
      nb: 16672,
      la: ISODate("2023-06-03T19:48:51.027Z")
    },
    threeCheck: {
      gl: {
        r: 1728.0990987120676,
        d: 150.37570913203353,
        v: 0.06006423835345389
      },
      nb: 66,
      re: [
        1728, 1731, 1658,
        1538, 1547, 1570,
        1568, 1559, 1566,
        1578, 1587, 1596
      ],
      la: ISODate("2021-03-27T21:25:15.000Z")
    },
    correspondence: {
      gl: {
        r: 1926.127714977465,
        d: 116.29719592482803,
        v: 0.05988556724295303
      },
      nb: 369,
      re: [
        1926, 1960, 1919,
        1954, 1920, 1947,
        1972, 1942, 1910,
        1879, 1842, 1853
      ],
      la: ISODate("2023-05-23T07:44:15.072Z")
    },
    antichess: {
      gl: {
        r: 1512.0190365475771,
        d: 66.65431411944455,
        v: 0.0600183701790729
      },
      nb: 72,
      re: [
        1512, 1514, 1520,
        1527, 1534, 1542,
        1550, 1542, 1525,
        1510, 1521, 1532
      ],
      la: ISODate("2017-07-17T14:59:53.950Z")
    },
    atomic: {
      gl: {
        r: 1633.9234301933614,
        d: 69.42874642602649,
        v: 0.059952130348281715
      },
      nb: 99,
      re: [
        1633, 1653, 1656,
        1652, 1648, 1643,
        1637, 1639, 1635,
        1621, 1612, 1615
      ],
      la: ISODate("2016-07-07T15:16:14.324Z")
    },
    horde: {
      gl: {
        r: 1592.8516937243799,
        d: 75.64937953443014,
        v: 0.060008785807215394
      },
      nb: 46,
      re: [
        1592, 1583, 1566,
        1579, 1585, 1592,
        1610, 1594, 1612,
        1613, 1597, 1612
      ],
      la: ISODate("2017-11-29T14:47:28.389Z")
    },
    racingKings: {
      gl: {
        r: 1552.2357399960738,
        d: 144.27340681083268,
        v: 0.06000237100077715
      },
      nb: 13,
      re: [ 1550, 1603, 1625 ],
      la: ISODate("2016-01-30T16:03:04.867Z")
    },
    crazyhouse: {
      gl: {
        r: 1567.0457493437748,
        d: 75.4762700612214,
        v: 0.05998371721683685
      },
      nb: 50,
      re: [
        1567, 1589, 1591,
        1611, 1607, 1600,
        1627, 1636, 1627,
        1629, 1616, 1601
      ],
      la: ISODate("2016-10-07T07:09:39.982Z")
    },
    ultraBullet: {
      gl: {
        r: 1688.4426658439488,
        d: 247.39110046551573,
        v: 0.06001175225632094
      },
      nb: 3,
      la: ISODate("2018-06-28T12:37:57.445Z")
    },
    rapid: {
      gl: {
        r: 1855.4659797122008,
        d: 105.49122208374152,
        v: 0.059827359317597535
      },
      nb: 866,
      re: [
        1855, 1884, 1864,
        1847, 1817, 1822,
        1831, 1827, 1851,
        1832, 1810, 1790
      ],
      la: ISODate("2022-03-25T10:13:59.190Z")
    },
    storm: { runs: 39, score: 33 },
    racer: { runs: 78, score: 51 },
    streak: { runs: 46, score: 33 }
  },
}

* master: Update sbt to 1.9.1

* master: fix gamebook embeds in Lpv - closes #13129 fix branchToMove function name, again rename branchToMove to please fleischfleisch let lichess-pgn-viewer read default orientation from the PGN Orientation tag don't try loading lpv css twice fix ui/round reload URL expand forum studies without iframes expand blog studies without iframes Update sbt to 1.9.1 Update google-auth-library-oauth2-http to 1.19.0 Codegolf with ByColor Scala 15.3.6 and update Crazyhouse.Pocket changes

* 'user-perfs' of github.com:lichess-org/lila:

* master: X-file, redirect to next user when auto-next report enabled fix X-file controller type use new scala3 syntax for `flatMap`

* 'user-perfs' of github.com:lichess-org/lila: X-file, redirect to next user when auto-next report enabled fix X-file controller type use new scala3 syntax for `flatMap`

* origin/master: remove board menu "NEW" nag fix ui/analysis game socket url New Crowdin updates (#13125) Update sbt-bloop to 1.5.7

* master: New Crowdin updates (#13140) fix lichobile game analysis board

* master: (39 commits) New translations: preferences.xml (Vietnamese) New translations: arena.xml (Sorani (Kurdish)) New translations: arena.xml (Sorani (Kurdish)) New translations: storm.xml (Sorani (Kurdish)) New translations: faq.xml (Sorani (Kurdish)) New translations: contact.xml (Sorani (Kurdish)) New translations: arena.xml (Sorani (Kurdish)) New translations: broadcast.xml (Sorani (Kurdish)) New translations: insight.xml (Sorani (Kurdish)) New translations: class.xml (Sorani (Kurdish)) New translations: arena.xml (Sorani (Kurdish)) New translations: puzzle.xml (Sorani (Kurdish)) New translations: ublog.xml (Uzbek) New translations: preferences.xml (Uzbek) Remove add SanStr when black moves first New translations: puzzle.xml (Sorani (Kurdish)) New translations: site.xml (Ukrainian) fix ws exception taking down dasher New translations: puzzle.xml (Sorani (Kurdish)) New translations: keyboardMove.xml (Spanish) ...

* master: allow up to 64 official broadcasts to be active at the same time (!) Hide "search note" page from the sidebar to non-admin mods Fix tests Make ImporterForm returns Either Migrate scalachess 15.4.0 with Either

* origin/master: New Crowdin updates (#13149) whitelist trusted oauth apps operated by lichess scalafmt tweak abort and error propagation for explorerXhr Fix missing negation & code golf Refactor findCorrespondingChapter Refactor RelaySync.apply Use cats for LilaFuture

* origin/master: send new clock total time to WS on clockInc scala tweaks Fix move to X-file

* origin/master: Fix puzzleis playedXTimes translation - fixes #13159

ornicar · 2023-07-05T10:45:59Z

anyone wants to review 5K LOC? No? I'll just merge then

lenguyenthanh

I only look at lila.user, and have few comments.

lenguyenthanh · 2023-07-05T16:51:12Z

modules/user/src/main/UserPerfs.scala

+
+  def bestRatedPerf: Option[Perf.Typed] =
+    val ps    = perfs.filter(_._1 != PerfType.Puzzle)
    val minNb = math.max(1, ps.foldLeft(0)(_ + _._2.nb) / 10)


suggestion: there are few places, where we use 10, 15, should we create a const variables, so it'll be easier to understand?

modules/user/src/main/UserApi.scala

* 'master' of github.com:lichess-org/lila: use union type after #13134 neater cats syntax after #13134 New Crowdin updates (#13167) use ReadPref.priTemp fix ublog page title when editing reduce some youtube streamer log statement priorities

* master: scala tweaks fix Long to Int globally import cats.syntax.all.* tweak tournament top cache scala tweaks use union type after lichess-org#13134 neater cats syntax after lichess-org#13134 New Crowdin updates (lichess-org#13167)

* master: (46 commits) globally import cats.{Eq, Show, data.NonEmptyList} scala tweaks fix Long to Int globally import cats.syntax.all.* tweak tournament top cache scala tweaks use union type after lichess-org#13134 neater cats syntax after lichess-org#13134 New Crowdin updates (lichess-org#13167) use ReadPref.priTemp Fix wrong rewrite by scalafmt fix ublog page title when editing Update git blame ignore revs Fix scalafmt crashes Run scalafmtAll again Fix compiling error Run scalafmtAll reduce some youtube streamer log statement priorities Bump scalafmt 3.7.7 and add rewrite config remove last references to ReadPreference ...

* master: (58 commits) globally import cats.{Eq, Show, data.NonEmptyList} scala tweaks fix Long to Int globally import cats.syntax.all.* tweak tournament top cache scala tweaks use union type after lichess-org#13134 neater cats syntax after lichess-org#13134 New Crowdin updates (lichess-org#13167) use ReadPref.priTemp Fix wrong rewrite by scalafmt fix ublog page title when editing Update git blame ignore revs Fix scalafmt crashes Run scalafmtAll again Fix compiling error Run scalafmtAll reduce some youtube streamer log statement priorities Bump scalafmt 3.7.7 and add rewrite config remove last references to ReadPreference ...

* master: (55 commits) globally import cats.{Eq, Show, data.NonEmptyList} scala tweaks fix Long to Int globally import cats.syntax.all.* tweak tournament top cache scala tweaks use union type after lichess-org#13134 neater cats syntax after lichess-org#13134 New Crowdin updates (lichess-org#13167) use ReadPref.priTemp Fix wrong rewrite by scalafmt fix ublog page title when editing Update git blame ignore revs Fix scalafmt crashes Run scalafmtAll again Fix compiling error Run scalafmtAll reduce some youtube streamer log statement priorities Bump scalafmt 3.7.7 and add rewrite config remove last references to ReadPreference ...

* master: (49 commits) globally import cats.{Eq, Show, data.NonEmptyList} scala tweaks fix Long to Int globally import cats.syntax.all.* tweak tournament top cache scala tweaks use union type after lichess-org#13134 neater cats syntax after lichess-org#13134 New Crowdin updates (lichess-org#13167) use ReadPref.priTemp Fix wrong rewrite by scalafmt fix ublog page title when editing Update git blame ignore revs Fix scalafmt crashes Run scalafmtAll again Fix compiling error Run scalafmtAll reduce some youtube streamer log statement priorities Bump scalafmt 3.7.7 and add rewrite config remove last references to ReadPreference ...

ornicar added 25 commits June 29, 2023 23:55

rename UserPerfs

891edb8

extract user.perfs to user_perf collection WIP

be5c3ff

Merge branch 'master' into user-perfs

a93aee1

* master: Update sbt to 1.9.1

Merge branch 'user-perfs' of github.com:lichess-org/lila into user-perfs

10c48b0

* 'user-perfs' of github.com:lichess-org/lila:

UserPerfs WIP

fc49606

Merge branch 'master' into user-perfs

335aa91

* master: X-file, redirect to next user when auto-next report enabled fix X-file controller type use new scala3 syntax for `flatMap`

UserPerfs WIP

3b81c8d

Merge branch 'user-perfs' of github.com:lichess-org/lila into user-perfs

527635e

* 'user-perfs' of github.com:lichess-org/lila: X-file, redirect to next user when auto-next report enabled fix X-file controller type use new scala3 syntax for `flatMap`

Merge remote-tracking branch 'origin/master' into user-perfs

8a74402

* origin/master: remove board menu "NEW" nag fix ui/analysis game socket url New Crowdin updates (#13125) Update sbt-bloop to 1.5.7

UserPerfs WIP

da70011

Merge branch 'master' into user-perfs

3b71b91

* master: New Crowdin updates (#13140) fix lichobile game analysis board

UserPerfs WIP

36f74e9

UserPerfs WIP, remove PerfPicker

2060630

UserPerfs WIP

306d11f

UserPerfs WIP

7958f0a

Merge branch 'master' into user-perfs

e85d0f7

* master: allow up to 64 official broadcasts to be active at the same time (!) Hide "search note" page from the sidebar to non-admin mods Fix tests Make ImporterForm returns Either Migrate scalachess 15.4.0 with Either

UserPerfs WIP, Option.soFu

8a2a1ed

use Option.soFu some more

98e6243

UserPerfs WIP

2d563ad

fix setup controller

b50237f

UserPerfs WIP

12da21d

UserPerfs WIP

11f225f

ornicar mentioned this pull request Jul 4, 2023

Scalafmt 3.7.6 and rewrite #13160

Closed

ornicar added 4 commits July 4, 2023 12:00

Merge remote-tracking branch 'origin/master' into user-perfs

ee82eeb

* origin/master: send new clock total time to WS on clockInc scala tweaks Fix move to X-file

fix bogus embedded links

8c29289

Merge remote-tracking branch 'origin/master' into user-perfs

94a3cd8

* origin/master: Fix puzzleis playedXTimes translation - fixes #13159

UserPerfs WIP

634bd28

ornicar added 4 commits July 5, 2023 12:21

UserPerfs: it compiles!

4dd53bc

tweak coach list CSS

aa8f0fb

scala tweaks

1704acf

avoid loading streamer perfs on the list page

d3b3f37

ornicar marked this pull request as ready for review July 5, 2023 10:44

ornicar merged commit 814f06c into master Jul 5, 2023

lenguyenthanh reviewed Jul 5, 2023

View reviewed changes

ornicar added a commit that referenced this pull request Jul 6, 2023

neater cats syntax after #13134

4a02200

ornicar added a commit that referenced this pull request Jul 6, 2023

use union type after #13134

6209e57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

User perfs #13134

User perfs #13134

Uh oh!

ornicar commented Jun 30, 2023 •

edited

Loading

Uh oh!

ornicar commented Jul 5, 2023

Uh oh!

lenguyenthanh left a comment

Uh oh!

lenguyenthanh Jul 5, 2023

Uh oh!

ornicar Jul 6, 2023

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

User perfs #13134

User perfs #13134

Uh oh!

Conversation

ornicar commented Jun 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ornicar commented Jul 5, 2023

Uh oh!

lenguyenthanh left a comment

Choose a reason for hiding this comment

Uh oh!

lenguyenthanh Jul 5, 2023

Choose a reason for hiding this comment

Uh oh!

ornicar Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ornicar commented Jun 30, 2023 •

edited

Loading