Migrate status service and status page to New Platform

Subtasks:

Original issue content

Status service

from lifecycle rfc:
Core should expose a global mechanism for core services and plugins to signal their status. This is equivalent to the legacy status API kibana.Plugin.status which allowed plugins to set their status to e.g. 'red' or 'green'. The exact design of this API is outside of the scope of this RFC.

What is important, is that there is a global mechanism to signal status changes which Core then makes visible to system administrators in the Kibana logs and the /status HTTP API. Plugins should be able to inspect and subscribe to status changes from any of their dependencies.

This will provide an obvious mechanism for plugins to signal that the conditions which are required for this plugin to operate are not currently present and manual intervention might be required. Status changes can happen in both setup and start lifecycles e.g.:

[setup] a required remote host is down
[start] a remote host which was up during setup, started returning connection timeout errors.

Status API and page

Kibana currently exposes a api/status endpoint and associated status_page which renders this output:

name: config.get('server.name'),
uuid: config.get('server.uuid'),
version: {
  number: config.get('pkg.version').replace(matchSnapshot, ''),
  build_hash: config.get('pkg.buildSha'),
  build_number: config.get('pkg.buildNum'),
  build_snapshot: matchSnapshot.test(config.get('pkg.version'))
},
status: kbnServer.status.toJSON(), // https://github.com/elastic/kibana/blob/2a290a14066d4da2b626bb0b4e4e9d0193853230/src/legacy/server/status/server_status.js#L111
metrics: kbnServer.metrics // https://github.com/elastic/kibana/issues/46563 https://github.com/elastic/kibana/blob/ec481861799ed8dcced9cafd8112e5b26e641c54/src/legacy/server/status/lib/metrics.js#L57-L68

The status page app is rendered as a hiddenUiApp

kibana/src/legacy/server/status/routes/page/register_status.js

Line 50 in ec48186

}));

and migrating this will be blocked by

The api/stats endpoint is currently created in the same legacy plugin. It won't be migrated to Core, but depends on an equivalent to kbnServer.metrics being exposed from Core.

kibana/src/legacy/server/status/collectors/get_ops_stats_collector.js

Line 45 in c87e881

    
           ...kbnServer.metrics // latest metrics captured from the ops event listener in src/legacy/server/status/index

Pinging @elastic/kibana-platform

Hi,

The api/stats endpoint is currently created in the same legacy plugin. It won't be migrated to Core, but depends on an equivalent to kbnServer.metrics being exposed from Core.

I created the /api/stats endpoint with the idea that the stats data isn't suitable for Monitoring since the values come directly from Hapi, and weren't suitable for Monitoring consumption.

What are the thoughts about having /api/stats preserved and be a full replacement for /api/status?

BTW, the stats abbreviation was chosen to be similar to the Elasticsearch endpoints, such as _cluster/stats

I think we should make this APIs separate and just have the status page UI consume both the status API and stats API.

Migrating the stats API is part of #46563

Here's my initial thoughts on how plugin & core service statuses should work. This is not intended to replace an RFC on the concept, but to organize and document the original thinking before investigating all use cases.

Please poke holes in this 😄

High-level design concepts

More expressive status levels

Right now we have red, yellow, and green. These don't really explain much or have a consistent semantic meaning.

We could benefit from having status levels with explicit meaning and associated behaviors:

enum ServiceStatusLevel {
  available,   // everything is working!
  degraded,    // some features may not be working
  unavailable, // the service/plugin is unavailable, but other functions should work
  fatal        // block all user functions and show status page
               // (reserved for core services?)
}

Statuses reflect dependencies between plugins

In legacy, it's common for plugins to use the "mirror plugin status" concept to inherit their status from another plugin (most commonly, the Elasticsearch plugin).

It seems beneficial for this concept to be baked into the design of the new service:

A plugin's status should default to the highest-severity status of its required dependencies
A plugin's status should default to degraded if any of it's optional dependencies are >= degraded

Kibana should always try to keep as much functionality working as possible

In the legacy system, if any plugin changes to "red", pretty much all of Kibana's UI becomes blocked by the status page.

This prevents the user from using some of the built-in management and debug tools to diagnose and correct the problem. For instance, if Machine Learning is in the unavailable state, the user should still be able to use the Console app, License management, and Elasticsearch management tools to diagnose & fix the issue.

This is the purpose of the distinction between unavailable and fatal. I suspect that very few plugins should ever need to go into a fatal state. The few exceptions:

Security may want to trigger the fatal state if authentication is broken in some way (eg. license expiration)
Some key core services may need to block (eg. Saved Object migrations)

Plugins status does not alter which plugins are enabled

Anything that cannot be recovered from without restarting Kibana completely, should be throwing exceptions during setup or start rather than setting a unavailable or fatal status.

Therefore, we should not disable plugins because they are currently unavailable since removing plugins from the dependency tree requires an entire restart of the Core lifecycles (essentially an in-place restart of Kibana).

Core services & plugins should use the same status mechanism

Pretty self explanatory. There should be a single concept that backs status of different components in the system and they should easily interop with one another.

API Design

enum ServiceStatusLevel {
  available,   // everything is working!
  degraded,    // some features may not be working
  unavailable, // the service/plugin is unavailable, but other functions should work
  fatal        // block all user functions and show status page
               // (reserved for core services?)
}

interface ServiceStatus {
  level: ServiceStatusLevel;
  summary?: string;
  detail?: string;
  meta?: object;
}

interface StatusSetup {
  // Allows a plugin to specify a custom status dependent on its own criteria.
  // See calculation section below on how this is combined with dependency statuses
  setStatus(status$: Observable<ServiceStatus>): Observable<ServiceStatus>;

  // Exposes plugin status for dependencies of current plugin.
  // Type could be inferred by the generic type arg provided by plugins.
  getPluginStatuses$(): Observable<Record<string, ServiceStatus>>;

  // Statuses for all of core's services. Can be used with `inheritStatus` utility
  // for expressing dependent statuses on core services.
  core$: Observable<{
    http: ServiceStatus;
    elasticsearch: ServiceStatus;
    savedObjects: ServiceStatus;
    uiSettings: ServiceStatus;
  }>;
}

Status calculation

// Utility for merging several statuses together and producing a single status with the 
// most severe status up to the maxLevel.
const inheritStatus:
  (
    statuses$: Array<Observable<ServiceStatus>>,
    maxLevel?: ServiceStatusLevel
  ) => Observable<ServiceStatus>;

// Pseudo-code calculuation of a plugin's status
function calculatePluginStatus(requiredDeps: string[], optionalDeps: string[], pluginCustomStatus$?: Observable<ServiceStatus>) {
  const requiredDepStatus$ = inheritStatus(
    requiredDeps.map(dep => getStatusForPlugin(dep))
  );
  const optionalDepStatus$ = inheritStatus(
    optionalDeps.map(dep => getStatusForPlugin(dep)),
    ServiceStatusLevel.degraded // Optional dependencies are capped to 'degraded'
  );
  return inheritStatus([
    requiredDepStatus$,
    optionalDepStatus$,
    // `pluginCustomStatus$` is only set if plugin called `status.setStatus()`
    ...(pluginCustomStatus$ ? [pluginCustomStatus$] : [])
  ]);
}

Open questions

Do plugins need to be able to override the default inheritance of statuses from their dependencies?
Do we need a status service on the frontend as well as the backend?

@chrisronline

To unblock Monitoring's usage of the Kibana's overall status, we need these pieces in place:

Add compatibility with legacy plugins statuses
- Core's StatusService needs to consume the statuses from legacy plugins in order to calculate the overall status
Expose overall status to plugins (required for Monitoring)

@chrisronline Do you anticipate any major problems with us deprioritizing this for 7.8? It looks like continuing to consume this from legacy should work and we can move this under-the-hood later in 7.x without any user impact.

@joshdover

@joshdover No issues at all. Sounds good.

@joshdover

Currently, status service re-calculates the status every ~2.5sec. There is no way to detect a problem when a plugin interacts with the ES cluster. That doesn't allow plugins to rely on the platform providing the actual elasticsearch status when they need to be sure that information was written to / read from ES cluster. So plugins have to add custom logic that intercepts connection errors and retry the operation https://github.com/elastic/kibana/blob/master/x-pack/plugins/spaces/server/default_space/default_space_service.ts#L91-L101
There is a risk that plugins will have to re-implement that logic again and again. Also, it's easy to do it in the wrong way and overwhelm a cluster with the number of requests.

There was a suggestion for status service to provide API allowing plugins to enforce status check. Btw, the licensing service does it.
As an alternative, we can intercept all network request to ES and detect status change (503 status in response) to update status change. Or even allow plugins to claim to perform a request to ES with retry.

@joshdover Can we add it to our roadmap?

The problem described makes sense to me 👍

I'm unclear on what the first suggestion solution means:

There was a suggestion for status service to provide API allowing plugins to enforce status check. Btw, the licensing service does it.

This would be an API to allow plugins to register their own status check that would get executed by Core? Maybe we could leverage the ServiceSetup#set mechanism proposed in the original RFC for this (possibly with some changes)

As an alternative, we can intercept all network request to ES and detect status change (503 status in response) to update status change. Or even allow plugins to claim to perform a request to ES with retry.

This may be difficult. From my understanding it's possible for some APIs in Elasticsearch to return 503 Unavailable but other APIs to be working fine. Maybe we could put the ES status to degraded when Core's check is passing but there are 503s being emitted from some API calls? I just worry that 503s on some APIs may be irrelevant to many plugins. I think we need to analyze the actual failure behavior here in ES and then determine what global behavior here may (or may not) make sense.

@azasypkin

@azasypkin do you think we need this solved in Core before #65472 can proceed?

Maybe we could leverage the ServiceSetup#set mechanism proposed in the original RFC for this (possibly with some changes)

As I understand set allows to specify own status. But here a plugins needs to enforce status service to perform status checks immediately not waiting for polling interval to be completed. Licensing plugin provides refresh method triggering license re-fetch https://github.com/elastic/kibana/blob/master/x-pack/plugins/licensing/server/types.ts#L60

Got it, a manual refresh would be easy enough to add. Only complex part would be exposing a new manualTrigger$ or similar argument to core services and as part of the API exposed to plugins (when set is introduced).

@azasypkin

@azasypkin do you think we need this solved in Core before #65472 can proceed?

Not necessarily, I'd rather wait a bit more to give you all enough time to come up with a reasonable solution. I can try to replicate something similar to what Larry did here and switch to a proper solution as soon as you provide it.

Just a note of something I discovered while working on #67979:

In the legacy platform, there was a mechanism to display the status page instead of the app when the server was not ready

kibana/src/legacy/ui/ui_render/ui_render_mixin.js

Lines 212 to 225 in fa93a81

    
           server.route({ 
        
             path: '/app/{id}/{any*}', 
        
             method: 'GET', 
        
             async handler(req, h) { 
        
               const id = req.params.id; 
        
               const app = server.getUiAppById(id); 
        
               try { 
        
                 if (kbnServer.status.isGreen()) { 
        
                   return await h.renderApp(app); 
        
                 } else { 
        
                   return await h.renderStatusPage(); 
        
                 } 
        
               } catch (err) {

Probably important to note that this is currently broken, and have been for a long time:

We now are waiting for ES to be ready before actually starting the plugins, so the real app is never displayed until the server is green (only the kibana is not ready yet body content)
Now that we are based on a SPA (for non-legacy mode), the client-side router would display the app page instead of the status page after core_system's boot.

This is not blocker for #67979, and we did not get any complains about this broken feature, but we will probably want to be able to display the status page during the server's startup, and I don't see this task in the issue task list.

Closing as this is essentially complete, and the main outstanding task (#72831) is being tracked separately.

Navigation Menu

Repository navigation

Migrate status service and status page to New Platform #41983

Original issue content

Status service

Status API and page

High-level design concepts

More expressive status levels

Statuses reflect dependencies between plugins

Kibana should always try to keep as much functionality working as possible

Plugins status does not alter which plugins are enabled

Core services & plugins should use the same status mechanism

API Design

Status calculation

Open questions

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	server.route({
	path: '/app/{id}/{any*}',
	method: 'GET',
	async handler(req, h) {
	const id = req.params.id;
	const app = server.getUiAppById(id);

	try {
	if (kbnServer.status.isGreen()) {
	return await h.renderApp(app);
	} else {
	return await h.renderStatusPage();
	}
	} catch (err) {

Migrate status service and status page to New Platform #41983

Description

Original issue content

Status service

Status API and page

Activity

elasticmachine commented on Jul 25, 2019

tsullivan commented on Jan 17, 2020

joshdover commented on Jan 17, 2020

joshdover commented on Feb 27, 2020

High-level design concepts

More expressive status levels

Statuses reflect dependencies between plugins

Kibana should always try to keep as much functionality working as possible

Plugins status does not alter which plugins are enabled

Core services & plugins should use the same status mechanism

API Design

Status calculation

Open questions

9 remaining items

joshdover commented on Apr 9, 2020

chrisronline commented on Apr 9, 2020

mshustov commented on May 20, 2020

joshdover commented on May 20, 2020

joshdover commented on May 20, 2020

mshustov commented on May 20, 2020

joshdover commented on May 20, 2020

azasypkin commented on May 20, 2020

pgayvallet commented on Jul 16, 2020

lukeelmers commented on Aug 17, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions