✅ Deploy Preview for v3-meteor-api-docs canceled.

Cool to see this, @harryadel. Great experimantation! I agree having an advanced way to treat all types, also files, is great to have.

EJSON is used widely across the Meteor runtime, and in the past I looked for ways to make it faster, so relying on a more optimized approach is great. We need tests passing, since this core change touches many places, and we should measure its impact on Meteor 3 performance. It also looks like a good fit for Meteor 3.5, where a lot of benchmarking is happening. Once benchmarks are fixed for the next Change Stream server changes, we can measure how the new encoding strategies affect them, get insight into the gains, and use that to verify and consolidate the change.

@nachocodoner Hello Nacho, thanks for chiming in and being open to experimentation. I'm just trying different things and following curiosity/creativity. I know this touches many parts of the Meteor core so we're ought to ensure this doesn't break anything. so no worries I'm in it for the long run although I know how tiresome it'd be 😅
For now getting tests to run pass with this weird packages structure is the goal. ejson moved to deprecated, cbor added, mongo-id and mongo-decimal too were modified 🤷‍♂️

I'd like to raise a small concern regarding this PR: is Meteor Software really planning to replace the underlying data format without a transition period? This is a huge deal, since rolling out this change will mean that all existing connections will simply break, as after they connect to the new server, they won't be able to receive the information about the new version being available (it'll be sent to them in a different format).

I see there is some fallback in DDPCommon.parseDDP, but - correct me if I'm wrong - it will not work for the custom types registered via EJSON.addType (including Date).

I'm not against experimenting with new data formats, but I think there should be a distinction between the data format used internally (e.g., when cloning method parameters) and the one used on the wire (which can be replaced with patching DDPCommon.parseDDP and DDPCommon.stringifyDDP; plus a small core-level tweak to allow for binary formats if needed).

Another concern is that the cbor-x package has literally 1 (one!) maintainer. And I don't want to be a fearmonger here, but after the recent supply chain attacks, I'm more and more sceptical of relying on such (and I know for a fact that "big companies" are too). There are also quite a few unaddressed issues, including kriszyp/cbor-x#117, which will be an issue for applications with a strict CSP.

It's always a pleasure to hear from you @radekmie and I'm personally glad to get to learn from you. The Meteor community is lucky to have you around.

Is Meteor Software really planning to replace the underlying data format without a transition period?

I think @nachocodoner @italojs @Grubba27 are the authority here but if we you were to read the previous comments made by @nachocodoner and me. This is nothing but an experiment really that both of us are happy to see through. We've not committed to anything yet really. Heck there're few failing tests that I'd still have to fix before asking for a proper review really! So yeah this is more of a public experiment to gather feedback and brain storm together about Meteor problems and solutions. I explained my motives for this PR more than the nitty gritty of code. It's important to consider the current status of Meteor and how newcomers interact with it. Meteor needs to create a "pit of success".

This PR originated from a specific problem . i.e. How can we pass Files to methods? This was my way of answering this question, if you can offer a better way or suggest a way to improve the current way I'll gladly oblige.

This is a huge deal, since rolling out this change will mean that all existing connections will simply break, as after they connect to the new server, they won't be able to receive the information about the new version being available (it'll be sent to them in a different format).

This is definitely a interesting point I've not thought of, how'd existing connections utilize the new format. Would it cause any breakage or interrupting of flow? This must be addressed for sure. Though this is what partially motivated me to create a backwards compatible package with 1:1 similar API to ensure no breakage.

I see there is some fallback in DDPCommon.parseDDP, but - correct me if I'm wrong - it will not work for the custom types registered via EJSON.addType (including Date).

Actually custom types are supported in a backwards compatible way.

Another concern is that the cbor-x package has literally 1 (one!) maintainer. And I don't want to be a fearmonger here, but after the recent supply chain attacks, I'm more and more sceptical of relying on such (and I know for a fact that "big companies" are too). There are also quite a few unaddressed issues, including kriszyp/cbor-x#117, which will be an issue for applications with a strict CSP.

Thank you for pointing this out. I didn't consider this aspect when picking a library. I only hoped for a quick library that'll grant Meteor performance improvements. Maybe a copy-fork will help against it? I dunno.

All in all, thank for weighing in. Remaining tests are ought to be fixed and few preparations made then hopefully this can be revisited again.

[...] but if we you were to read the previous comments [...] This is nothing but an experiment really that both of us are happy to see through. We've not committed to anything yet really.

I read them. But then I see the labels and a milestone added, which suggests the next release (I assume 3.4 is already closed feature-wise). I know there are no "experiment-specific" labels, but the tone of the comments suggests "we have to measure the performance impact" as the only obstacle. (I guess that's not the case, but that's the tone.)

How can we pass Files to methods? This was my way of answering this question, if you can offer a better way or suggest a way to improve the current way I'll gladly oblige.

To be perfectly honest, sending files (or any binary blobs) via a WebSocket is not a good idea performance-wise. First of all, there's an inevitable memory limit on the file size. Secondly, there's no way to parallelize the upload, do a partial one, or resume it after going offline. And possible operations on the file blob on the server are also limited (usually, you will either push it to some storage or parse straight away).

Allowing the client to upload files directly to some storage (e.g., an S3-compatible one) is strictly superior in all cases except for the ease of use. (It's not limited to WebSockets; the same applies to all kinds of APIs, including REST or GraphQL.)

But assuming we really want to make it happen, I'd either accept the 33% cost (base64 transformation), or focus on supporting binary blobs in the DDP (EJSON already supports it, so it's just about transferring these more efficiently).

A non-trivial, but also non-invasive way of sending them would be sending them in dedicated follow-up binary messages. Just like the Transferable objects, we could somehow replace the binary blob with some placeholder, and send them right after their original message.

Though this is what partially motivated me to create a backwards compatible package with 1:1 similar API to ensure no breakage.

Actually custom types are supported in a backwards compatible way.

API-level incompatibility is one topic (it's not fully addressed yet, as the package name has changed), but the wire format is another. At least for the release, the server should be able to accept both formats and negotiate which one to send the messages in.

This could be a good case for a new DDP version. Whether it'll be a new wire format, or a support for binary blobs, DDP already has a version negotiation logic which could be used here with preserving backward compatibility.

Maybe a copy-fork will help against it? I dunno.

Maybe. But then it comes at a much higher cost of maintaining a whole new library.

This is definitely considered a contribution. Contributions are outside the official roadmap. We evaluate them when it's time to plan releases. If it's a big change like this one, we appreciate early feedback and extra focus when we move into the next phase. We tagged it for 3.5 so we don't lose track of this PR, and so we revisit it often, at least for the next release. But from experience, contributions often slide to later releases. We usually don't feel comfortable merging some of them right away if they involve possible breaking changes or performance risks, and we want more evidence, and community and core time put into them. This PR is in so early phase. Maybe an experimentation flag would be more suitable, but so far our process has been: keep contributions visible, revisit them, and try to eventually ship them, that is the reason to append them in release tracks. But to avoid confusion and release expectations we may add a new experimental flag. I'm going to discuss with the team.

Regarding this PR, we need to measure performance impact surely, gather feedback and deep analysis from our team, and also from the community like you're doing, which we appreciate a lot. We also need time to look for other concerns in this area. I understand this changes the foundation of how we handle data. It may keep API compatibility, but it could still break things in other ways as you stated. Because of that, I think this PR needs more work: full testing cycles, betas, possible release candidates if it gets that far, and also a fast way to revert it.

Whatever the final answer is, either replacing the current approach or extending EJSON to handle effeciently binary blobs in files, what I like here is the spark of experimentation and action. I prefer to have a PR that actually changes things instead of long discussions in forums or GitHub without action. I want this to push everyone to discuss what's best. We appreciate the feedback and experience here.

@harryadel if you want to keep exploring this approach, that's completely fine, or you can look into the alternatives mentioned. In either case, I would also suggest a way to make this opt-in behind an experimental config (like modern for bundler). I know can be tricky to achieve for a EJSON replacement. With that opt-in setting, we can already achieve easily to introduce no breaking change, and a fallback mechanism to EJSON. Also, if we find a good path, this could become a community package to expand usage and discussion, and maybe reach core later with more consensus.

Let's keep the creative and innovative process going.

But assuming we really want to make it happen, I'd either accept the 33% cost (base64 transformation), or focus on supporting binary blobs in the DDP (EJSON already supports it, so it's just about transferring these more efficiently).

On the topic itself, I believe this is the most straightforward approach to reduce the breaking changes we discussed. This is how I upload images in my apps: I convert them to base64 and send them through the method. I understand there are performance and capacity concerns, but in the cases I've tried it hasn't been that bad. Maybe this is the minimum we can ship. Having an opt-in mechanism in existing methods that can detect files and do the transformation in the method API, without pushing that work to Meteor app devs, would be great. Or we can try to orchestrate sending binary blobs for bigger scale.

This doesn't block us from brainstorming and experimenting with the other approach. It can happen that after a few iterations on what is hard, it becomes a strong candidate in the future, maybe also giving a performance boost or other wins. Just that starting with something small and achievable in the short term helps you gain experience with it and already gives people a usable path.

Anyone is welcome to contribute and open a PR to address these points. Regarding the core, work on DDP is already in progress, and we’ll also take into account everything discussed here to explore new complementary directions. Having multiple approaches to compare will give us clearer insights and more direct answers. In my opinion, the key is to always keep options open for everyone who wants to contribute instead of sticking to a single approach defined by the Core team.

Not to toot my own horn but thank you @nachocodoner <3. Talk is cheap and lots of people talk. I'd rather start a PR, discuss it and close it rather than just talk and talk. Surely, it's a bit work but I get to learn and there's work in public that others can rely on later in the future to avoid pitfalls or even improve the proposed solution. Shoot first, ask later 😅

There's a value in EJSON to CBOR conversion regardless of this PR.

PROS:

• Native support for alternative types (including Files but not exclusively)
• Potential Performance gains? whether in encoding or transmission
• Modern approach

CONS:

• Potential breaking changes to current apps
• Supply chain attacks given our reliance on external implementation of CBOR instead of our own

Since what's left to get the tests passing is very little compared to rolling everything back I'm going to continue and then see what the future holds for us.

@radekmie Thank you for chiming in, contesting ideas is great for the overall health of our community and our beloved Meteor. 🙏

@radekmie Always brings good points. I like it when he reviews the PRs. We won't consider those points if he didn't bring them. I understand Radek's point when he mentioned the labels and milestones. Maybe we should have better indicators of the real state of the PR. We never merge critical changes like this without a thorough analysis of security, DX, performance, and other factors.
At the same time, the point about experiment -> discuss experiment again is the exact behavior we love to see.

What we need here is a guide for the next steps. The PR shows a good promise; we should go ahead, but I see a few questions that need answers first:

• Ensure it solves the issue
• Ensure it has a fallback
• Discuss the cbor-x usage
  • Do we have another alternative?
  • Can we maintain it?
  • How much effort do we need to solve the main issues(and who will do it)?
• - Measure how much faster it is compared with EJSON and fallback-EJSON

Feel free to contribute to this list, guys ;)

@harryadel I have an unfinished project where I use OpenTelemetry + Prometheus + Grafana to measure a few metrics from meteorjs app, I will try to push it beween today or tomorrow, then we weill be able to mensure the performance of it, in this mean time you can work on the other questions

@harryadel Here inside /otel folder you can setup a few benchmarks

# shell1
meteor npm i
docker compose up -d
npm run start

# shell2
npx artillery run tests/artillery/add-task.yml   # it just test the server

Open Grafana at localhost:3000

You can see a few metrics inside dashabords menu session (PS: Dont trust on trace numbers from dashobards`

Original author of the "Building Meteor in 2025" here.

It's exciting to see Meteor's modernization shifting from forum discussions to active PRs great work on pushing this forward @harryadel!

From my experiments earlier this year, the optimal path is to prioritize refactoring the core DDP and EJSON logic for extensibility via abstractions before diving into specific integrations like CBOR. This keeps DDP as a protocol-agnostic communication standard, decoupling it from specific transports (e.g., the current SockJS/WebSocket setup) and serialization formats (e.g., EJSON).

Here are refined steps for setting up Meteor to integrate newer serialization libraries, drawing from the current DDP implementation.

1. Decouple DDP from Transport and Serialization

Currently, DDP on the client uses a LivedataConnection class (exposed as DDP.Connection) that tightly couples to a Stream transport (e.g., SockJS for WebSockets) and EJSON for message handling. On the server, similar logic in livedata_server.js processes messages via EJSON.parse and handles connections through a conduit. We can abstract this by defining a core DDP class with required methods for transports and serializers to implement, ensuring the core focuses on protocol semantics like methods, subs, and data updates.

Example abstraction, inspired by existing LivedataConnection structure:

// packages/ddp-core/abstract/ddp.js
import { EJSON } from 'meteor/ejson'; // For fallback or registry

export class DDP {
  constructor(options = {}) {
    this.transport = options.transport; // Injected transport instance (e.g., SockJSStream)
    this.serializer = options.serializer || EJSON; // Default to EJSON
    this.registry = {
      transports: new Map(),
      serializers: new Map()
    };
    this._methodInvokers = {}; // Track outstanding methods
    this._subs = {}; // Track subscriptions
    // ... other state from current LivedataConnection
  }

  // Abstract connect, delegated to transport
  async connect(url, retry = true) {
    if (!this.transport) throw new Error('Transport not set');
    await this.transport.connect(url);
    this.transport.on('message', (raw) => this._processMessage(raw));
    this.transport.on('reset', () => this._handleReset());
    // Handle reconnection logic if retry=true, mirroring current auto-reconnect
  }

  // Send message, using serializer and transport
  send(message) {
    const serialized = this.serializer.stringify(message);
    this.transport.send(serialized);
  }

  // Process incoming, using serializer
  _processMessage(raw) {
    let msg;
    try {
      msg = this.serializer.parse(raw);
    } catch (e) {
      console.error('Parse error:', e);
      return;
    }
    // Dispatch based on msg.msg: 'method', 'result', 'updated', 'added', etc.
    if (msg.msg === 'result') {
      this._handleMethodResult(msg);
    } else if (['added', 'changed', 'removed'].includes(msg.msg)) {
      this._handleDataUpdate(msg);
    } // ... extend for other types like 'ready', 'nosub'
  }

  // Registry methods for plug-and-play
  registerTransport(name, transportClass) {
    this.registry.transports.set(name, transportClass);
  }

  registerSerializer(name, serializer) {
    this.registry.serializers.set(name, serializer); // { stringify, parse }
  }

  // Negotiation during connect handshake
  negotiate(clientCapabilities) {
    // Example: Client sends {transports: ['websocket', 'webtransport'], serializers: ['ejson', 'cbor']}
    const preferredTransport = this._selectBest(clientCapabilities.transports, this.registry.transports);
    const preferredSerializer = this._selectBest(clientCapabilities.serializers, this.registry.serializers);
    this.transport = new (this.registry.transports.get(preferredTransport))();
    this.serializer = this.registry.serializers.get(preferredSerializer);
    return { selectedTransport: preferredTransport, selectedSerializer: preferredSerializer };
  }

  // Core handlers (agnostic)
  _handleMethodResult(msg) {
    const invoker = this._methodInvokers[msg.id];
    if (invoker) {
      if (msg.error) invoker.setError(msg.error);
      else invoker.setResult(msg.result);
      delete this._methodInvokers[msg.id];
    }
  }

  _handleDataUpdate(msg) {
    // Update local stores or trigger reactivity, as in current _livedata_data
  }
}

This mirrors the existing client-side flow (e.g., _stream.send and EJSON.parse in livedata_connection.js) but makes transport and serializer injectable, reducing coupling.

2. Implement a Registry for Transports and Serializers

Build a central registry in ddp-core for dynamic registration. Current implementations (e.g., SockJS stream on client, WebSocket on server) can auto-register as defaults.

Example:

// In ddp-websocket package
import { DDP } from 'meteor/ddp-core';
import SockJS from 'sockjs-client'; // Or native WebSocket

class WebSocketTransport {
  constructor(url) {
    this._sock = new SockJS(url); // Or WebSocket(url)
  }
  async connect() {
    return new Promise((resolve, reject) => {
      this._sock.onopen = resolve;
      this._sock.onerror = reject;
    });
  }
  send(data) {
    this._sock.send(data);
  }
  on(event, callback) {
    if (event === 'message') this._sock.onmessage = callback;
    // ... other events like 'close', 'error'
  }
}

DDP.registerTransport('websocket', WebSocketTransport);

For serializers:

// In ddp-cbor package
const CBOR = require('cbor-x');

const CborSerializer = {
  stringify: (obj) => CBOR.encode(obj),
  parse: (data) => CBOR.decode(data)
};

DDP.registerSerializer('cbor', CborSerializer);

3. Migrate Existing Implementations

Extract current logic to separate packages: ddp-websocket for transport (based on stream_client_sockjs.js), ddp-ejson for serialization (wrapping ejson.js's stringify/parse). Auto-register on import for compatibility—no changes for existing apps.

4. Add Client-Server Negotiation

Extend the 'connect' DDP message with capabilities, as in current handshake (version negotiation via 'support' field). Server responds with selected options.

Example extended message:

{ msg: 'connect', version: '1', support: ['1', 'pre2'], capabilities: { transports: ['websocket', 'webtransport'], serializers: ['ejson', 'cbor'] } }

5. Test and Integrate New Options

With the framework, adding CBOR is simple: Implement the serializer, register it, and benchmark.

This sets Meteor up for future innovations in protocol communications and data serialization formats without core rewrites.

Relevant to this discussion: JSON Serialization Formats.

@harryadel

Native support for alternative types (including Files but not exclusively)

Just for the record: it can be achieved now using EJSON.addType. You could even create a small Meteor package which would register it on both client and server with respective base64 conversion for the transfer.

Potential breaking changes to current apps

I'll keep being pedantic about it: this is, not may be, a breaking change.

@italojs

Maybe we should have better indicators of the real state of the PR.

The in-discussion seem good enough.

@mvogttech

Extend the 'connect' DDP message with capabilities, as in current handshake (version negotiation via 'support' field). Server responds with selected options.

While the DDP specification is "[...] a rough description of the protocol and not intended to be entirely definitive [...]", I'd like to highlight that there are a few DDP non-JS implementations out there, and for some of them (e.g., the one in the DDP Router, sending additional fields will be a breaking change on its own.)

An alternative would be to do something like "scoped versions", e.g., 2.cbor. Please note that the first message (connect) should stick to EJSON, at least for the time being. I also believe the new formats should be opt-in, so it's possible for every connected client to opt-in.

An alternative would be to do something like "scoped versions", e.g., 2.cbor. Please note that the first message (connect) should stick to EJSON, at least for the time being. I also believe the new formats should be opt-in, so it's possible for every connected client to opt-in.

I agree.

This is the exact design I was trying to convey with my previous message.