Skip to content

write fields to redis/clickhouse, fewer data sync submits #10093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mayberryzane merged 7 commits into from
Mar 25, 2025
Merged

write fields to redis/clickhouse, fewer data sync submits #10093

mayberryzane merged 7 commits into from
Mar 25, 2025

Conversation

@mayberryzane
Copy link
Contributor

@mayberryzane mayberryzane commented Mar 19, 2025 •
edited
Loading

Summary

  • save fields to Redis and then ClickHouse without saving to Postgres
  • add fields_by_session table and mv for looking up fields by session id
  • remove ResumedAfterProcessedTime updates as that is just for debugging and doesn't seem necessary
  • clean up some unnecessary? logic for excluded sessions
  • only submit to the data sync queue if fields have changed (this is 20-25% of push payload time)
    Screenshot 2025-03-19 at 12 57 51 PM

How did you test this change?

  • local clicktesting
  • tested places where fields are used - session CSV report, metadata panel, etc
  • tested live + resumed sessions
  • tested old user properties alert

Are there any deployment considerations?

  • fields_by_session table has already been populated in prod
  • can remove fields and session_fields tables in Postgres eventually

Does this work require review from our design team?

  • no

@changeset-bot
Copy link

changeset-bot bot commented Mar 19, 2025 •
edited
Loading

⚠️ No Changeset found

Latest commit: 5abb6d2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR 
Some errors occurred when validating the changesets config:
The package or glob expression "rrdom" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrdom-nodejs" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb-player" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb-snapshot" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/all" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/types" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/packer" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-canvas-webrtc-record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-canvas-webrtc-replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-console-record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-console-replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-sequential-id-record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-sequential-id-replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/utils" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/web-extension" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrvideo" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

@mayberryzane mayberryzane marked this pull request as ready for review March 24, 2025 00:40
@mayberryzane mayberryzane requested review from a team and ccschmitz and removed request for a team March 24, 2025 00:40
@mayberryzane mayberryzane changed the title fewer data sync submits write fields to redis/clickhouse, fewer data sync submits Mar 24, 2025
ccschmitz
ccschmitz approved these changes Mar 25, 2025
View reviewed changes
Copy link
Contributor

@ccschmitz ccschmitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code + click test LGTM!

@mayberryzane mayberryzane merged commit e17f6cc into main Mar 25, 2025
28 checks passed
@mayberryzane mayberryzane deleted the zane/fewer_syncs branch March 25, 2025 17:56
mayberryzane added a commit that referenced this pull request Mar 25, 2025
@mayberryzane
Revert "write fields to redis/clickhouse, fewer data sync submits (#1…
c27cfdb 
…0093)"

This reverts commit e17f6cc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ccschmitz ccschmitz ccschmitz approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mayberryzane @ccschmitz