Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1348 Ideas

Oussama ALLALI
Seasoned ⭐️⭐️
Oussama ALLALISeasoned ⭐️⭐️

Feature request — native CDF Time Series push for metrics in PI Extractor and DB ExtractorGathering Interest

ContextWe currently run several custom Python extractors built on cognite-extractor-utils. For those, cognite-extractor-utils lets us configure a metrics.cognite block that periodically pushes the extractor's Prometheusmetrics (Counter/Gauge) directly to CDF Time Series — this gives us consistent,queryable observability (row counts, run duration, throughput, failure counts, etc.)across all our extractors, without any extra tooling.ProblemThe PI Extractor and the DB Extractor (official Cognite binaries) already exposePrometheus metrics internally (e.g. pi_extractor_timeseries_states,pi_extractor_stream_iterations, pi_extractor_pi_data_pipe_events,pi_extractor_streamer_data_points, pi_extractor_event_queue_size,pi_extractor_pi_data_loss, pi_extractor_pi_connection_time,pi_extractor_pi_connections, extractor_utils_cdf_datapoints, …), but today they canonly be exposed via a local Prometheus scrape endpoint (metrics.server) or a PushGateway (metrics.push-gateways). There is no equivalent of metrics.cognite to pushthese metrics natively to CDF Time Series, unlike our custom extractors.ImpactWithout native CDF Time Series push, we can't monitor these extractors the same way as our custom ones (no unified dashboarding/alerting in CDF/Grafana). We want to keep CDF as the single source of truth for observability, not add a separate tool just to collect metrics.RequestCould you add support for a metrics.cognite configuration block (or equivalent) in thePI Extractor and DB Extractor, consistent with the one already available viacognite-extractor-utils, so their internal Prometheus metrics can be pushedperiodically to CDF Time Series (external-id prefix, target data set, push interval)?

Gunnar AndreasSeasoned ⭐️⭐️

Full API access to inspect and update instances in ChartsGathering Interest

Today there is limited to no API access to Charts, creating several issues.One obvious issue is that we cannot automate the setup of Charts. Eg, when an anomaly detection method detects an issue, we would like to be able to auto-create the Charts with the necessary timeseries, timerange, calculations and overlay anomaly insight (CogniteActivity extension). This is not possible today (only pre-build url with instances and range).But another issue is around the maintainability of Charts. Due to the limit on the number or Views/revisions, we need to deprecate and delete old revisions on a regular basis (in particular in periods with heavy data model development). As far as I have seen, Atlas, Canvas and Chart lock the View version to the latest version (or maybe based on the location filter) during config. For Atlas agents and Canvas’ we can find the View versions via the API, allowing us to identify users depending on models that are deprecated, and also auto-duplicate with the updated versions/views. However, with Charts this is not possible. If Charts is supposed to only be a do-my-analysis-and-then-throw-it-away tool, that is ok, but if it is intended to be used for more long term analysis (it does offer functionality for this like calculations and threshold monitoring) we need a way to monitor status and update the views/instances in the Charts on an enterprise level (via API)