An Microservices-Based Openstack Monitoring Tool

M eixia Yang M ing Huang

Wireless Innovation Lab. o f Yunnan University, School o f Wireless Innovation Lab. o f Yunnan University, School o f
Information Science and Engineering o f Yunnan University Information Science and Engineering o f Yunnan University,
Kunming 650091 China Kunming 650091 China
mitchellyang@ 126.com huangming@ynu.edu.cn

Abstract—Monitoring of openstack clouds is an imperative Microservices is an architectural style where systems and
necessity for cloud providers and administrators to analyze and applications are broken down to a more granular and modular
discover what is happening in the cloud. In this paper, a level. The concept was created as a service-oriented
microservices-based openstack monitoring system, namely architecture (SOA) about 10 years ago [5-7]. It is about
openstack-reporter, is proposed, aiming at monitoring openstack decomposing complex applications into small and independent
clouds and providing a convenient tool for the administrators. process, thus improving performance. The benefit of
Openstack-reporter adopting microservices architecture consists microservices is that applications are easier to understand, test,
of three constituent components where each component is and most importantly easier to maintain over the life of the
responsible for a single aspect. The connection among these
application. It enables organizations to achieve much higher
components is implemented with the help of kubernetes DNS-
agility and be able to vastly improve the time it takes to get
based service discovery and Role-based access control (RBAC)
mechanisms. The management of these three components is working improvements to production [8-10]. This approach has
performed by kubernetes which is devoted to automate rollouts, been proven to be superior, especially for large enterprise
rollbacks. We have released the openstack-reporter, docker applications which are developed by teams of geographically
images and kubernetes configurations which can be accessed and culturally diverse developers.
publicly. One can easily build the openstack monitoring system Docker container is an emerging container technology
just by deploying the openstack-reporter in our kubernetes
where different functions are provided to make provisioning of
cluster. To validate the performance of the proposed monitoring
microservices cost effective and easy to manage [11]. The two
system, an openstack platform and openstack-reporter have been
built in our datacenter and the management center respectively,
are often paired even though they can operate autonomously of
and the results are displayed. each other. Compared to VMs where each VM has its own
guest OS, Containers reduce the overhead of having different
Keywords-component; microservices; kubernetes; service guest OS for different container process which results in high
discovery; cloud computing; monitoring performance, less memory requirement and reduced
infrastructure cost [12]. Kubernetes provides the means to
support container-based deployment, focusing specifically on
I. In t r o d u c t io n
cluster-based systems. Kubernetes enables deployment of
Cloud computing architectures have gained more and more multiple “pods” across physical machines, enabling scale out
attention in recent years and the openstack cloud is one of such of an application with dynamically changing workload. Each
cloud platform [1]. It was originally formed by Rackspace and pod can support multiple Docker containers which are able to
NASA in 2010 as a platform for allowing organizations to make use of services associated with a pod [13]. Kubernetes
provide the cloud computing services running on standard has become the standard for managing containers at scale for
commodity hardware [2]. The aims of cloud computing are to agile, secure software development which was proposed by
create abstracted pools of compute, storage, and networking Google in 2014.
resources which can be used to create virtual machines on top
of standard server hardware. OpenStack is built as a set of the In this paper, a microservices-based openstack monitoring
distributed services [3]. These services communicate with each system, namely openstack-reporter, is proposed whose
other, and carry various functions expected by the objective is to monitor and analyze the performance of
virtualization/cloud management software. The followings are openstack cloud and provide a convenient tool for the
some of the key OpenStack services: nova, neutron, cinder, administrators. Microservices architecture is adopted where
swift, keystone, glance, heat and horizon. The efficient openstack-reporter is decoupled into three small and
management of Cloud infrastructure and deployments is one of independent applications, including openstack-exporter,
the hotspots of current research. Understanding the behaviors prometheus and grafana. Specifically, openstack-exporter is
of these hierarchical systems and managing them optimally are responsible for collecting openstack API derived metrics and
challenging tasks, which can be facilitated by pervasive provides data for prometheus. Prometheus concentrates on
monitoring. Monitoring tools and techniques play an important capturing measurements and exposing them via an API. And
role in this area by gathering the desired information for the finally, grafana is used for visualization and analysis based on
informed decisions [4]. prometheus data. Mutual access among the three parts must be
taken into consideration and kubernetes DNS-based service
This work was supported in part by the National Natural Science Foundation
of China under Grant Nos. 61963037, 61863035, 11564044

978-1-7281-0945-9/19/$31.00©2019 IEEE

706

Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 10,2020 at 15:24:40 UTC from IEEE Xplore. Restrictions apply.
discovery and role-based access control (RBAC) is leveraged. B. Kubernetes DNS-based service discovery and RABC
The management of the three applications is promoted by The service discovery is the process of figuring out how to
kubernetes for the reason of the superior performance relating connect to a service. Kubernetes includes a DNS server,
to deploy, scale and management of kubernetes. There is no CoreDNS, for use in the service discovery. This DNS server
limitation on distance between the openstack platform and utilizes the libraries from SkyDNS to serve DNS requests for
openstack-reporter, and network connectivity is the only Kubernetes pods and services. RBAC is a method of regulating
requirement. To present the performance of the openstack- access to computer or network resources based on the roles of
reporter, an openstack platform and openstack-reporter have individual users within an enterprise. The RBAC ensures the
been built in our datacenter and management center control of who has access to your Information system and what
respectively. The openstack platform consists of one controller users have access to. The RBAC model in kubernetes is based
node, two compute nodes and storage nodes where six services on three elements: 1) Roles: definition of the permissions for
are included: keystone, glance, nova, neutron, cinder and each kubernetes resource type; 2) Subjects: users (human or
horizon and the openstack-reporter which is running in a two- machine users) or groups of users; 3) RoleBindings: definition
node kubernetes cluster. Importantly, the openstack-operator of what Subjects have which Roles.
can be access publicly and docker images are published in the
docker hub and kubernetes configurations can be obtained from
C. the proposed microservices-based openstack monitoring
github, using which you can deploy your own openstack
system
monitoring system in a quick and easy way.

II. THE PROPOSED MICROSERVICES-BASED

OPENSTACK MONITORING TOOL
The implementation details of openstack-reporter are
provided in this section. It comprises of three constituent
components: openstack-exporter, prometheus and grafana. The
connecting among three applications are conducted using
kubernetes DNS-based service discovery and RBAC
mechanisms. Fig. 1 overall arcitecure of openstack monitoring

A. prometheus and grafana The proposed openstack-reporter is shown in Fig. 1. It can

Generally speaking, prometheus is a monitoring solution be seen that the openstack-reporter is running in the kubernetes
for storing time series data and grafana allows to visualize the cluster and it pulls metrics from the openstack platform, and
data stored in Prometheus. Prometheus has following features: the monitoring results are visualized. In openstack-reporter,
1) a multi-dimensional data model with time series data microservices architecture is adopted where openstack-reporter
identified by metric name and key/value pairs; 2) a flexible application is broken down into three constituent components
query language to leverage this dimensionality; 3) no reliance including openstack-exporter, prometheus and grafana. The
on distributed storage; single server nodes are autonomous; 4) management of openstack-reporter is performed by Kubernetes
time series collection happens via a pull model over HTTP; 5) which is leveraged to automate rollouts, rollbacks, and
pushing time series is supported via an intermediary gateway; 6) monitors the health of openstack-reporter. It is worth
targets are discovered via service discovery or static mentioning that the openstack platform and openstack-reporter
configuration; 7) multiple modes of graphing and dashboarding can be used in different places and the network connectivity is
support. the only requirement.

Grafana is an open-source software for monitoring and The implementation details of openstack-reporter are
analyzing. One of its major characteristics is that it supports described as follows: the three components are deployed as
many different data sources, from popular CloudWatch, services independently in the kubernetes cluster. Each
Elasticsearch, Graphite, and influxDB, to OpenStack Gnocchi component focuses on a single aspect. Specifically, openstack-
or Google Calendar. Grafana equips with various panel options. exporter is responsible for collecting openstack API derived
Other than the basic text, graphs, and charts, administrators or metrics and provide data for prometheus. Prometheus focuses
operators can use panel, such as gantt chart or flow chart to on capturing measurements and exposing them via an API.
display monitored data and information, and change or adjust And finally, grafana is used for visualization and analyzing
the color they like. Another convenient characteristic is that it based on prometheus data.
allows users to save the pages as a template. Thus, when there
is a need to install Grafana on different server, simply import
the saved template.

707

Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 10,2020 at 15:24:40 UTC from IEEE Xplore. Restrictions apply.
 prometheus service shown in Fig. 3. Besides, grafana provides
beautiful and elegant dashborads for displaying various metrics
through a web browser. The highlight is the templates for
displaying monitoring results have been saved and
incorporated with our application shown in Fig. 3. We have
made docker images and kubernetes configurations publicly.
Docker images can be downloaded from Docker hub
(mitchell520/Prometheus, mitchell520/openstack-exporter and
mitchell520/grafana-grafana) and kubernetes configurations
can be founded in github
(https://github.com/Mitchell1314/openstack-reporter).

III. RESULTS AND DISCUSSION

An openstack platform has been built in our datacenter
which consists of a control node and two compute nodes, the
storage of two hard disk is respectively mounted to two
compute nodes which are used for cinder service providing
volumes to Nova virtual machines. Six components have been
installed including: keystone, glance, nova, neutron, cinder and
horizon. In order to monitor the openstack cloud, a
Fig.2 RABC for prometheus to access openstack metrics

Mutual access among the three microservices plays a key

role i n openstack-reporter. It is implemented by kubernetes
DNS-based service discovery and RABC mechanisms. RABC
allows prometheus to access resources of openstack-exporter.
In detail, in order to pull metrics from openstack-exporter,
prometheus should be granted permissions to access pods,
endpoints, and services in kubernetes cluster and it is can be
done via ClusterRole resource that defines an RBAC policy
and details are shown in Fig.2. First, a ClusterRole named
datasource.yaml
grafana
data sources:

r~j
- name: prometheus

±
1 ,
type: prometheus
url: http://prometheu5.default.svc.dijster.local:9090
[' dashboard

'c.cluster.local:9090

dasbboard.yaml

apiVersion: 1
providers:
- name: 'default1
orgld: 1
fo ld er "
type: file
update! nterva ISeconds: 10
Service(prometheus)
path: /tmp/grafana

Fig.3 the process of grafana obtaining prometheus data

prometheus is created and is permitted to get, list and watch

services in the kubernetes cluster. In the subsequent, a
ServiceAccount for prometheus to represent its identity in the
cluster is created. Finally, we bind the ServiceAccount and the
ClusterRole using the ClusterRoleBinding resource allowing
prometheus to access openstack-exporter resources.
And DNS-based service discovery mechanism allows
prometheus detecting openstack-exporter service and grafana
detecting prometheus service automatically, and configurations
are shown in Fig. 2 and Fig. 3 respectively. The configuration
of prometheus should contain a list of scrape targets and Fig. 4 monitoring results
kubernetes auto-discover settings that allow prometheus to management center is built, which is a kubernetes cluster
automatically detect openstack-exporter application. The comprising of two machines: a master node and a worker node.
Grafana targets prometheus data by defining datasource in the The openstack-reporter is deployed in the kubernetes cluster.
form of datasource URL which is the DNS record of The monitoring results about our openstack clouds using

708

Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 10,2020 at 15:24:40 UTC from IEEE Xplore. Restrictions apply.
openstack-reporter are shown in Fig. 4. Firstly, system management system where metrics from openstack-reporter
information relating to ram, disk size and CPU is presented, w ill be used to check the opesntack status.
and the total ram, total block storage and number of vcpus are
222GB, 71.4TB and 144 respectively for our openstack cloud. Re f e r en c es
Secondly, openstack services status within a period of time are
[1] OpenSatck, https://www.openstack.org/J. Clerk Maxwell, A Treatise on
shown which contributes to check the status of the cloud. Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892,
Subsequently, the amount of resources available is given, for pp.68-73.
instance, block storage of 67.2TB, ram of 158GB and 110 [2] O. Sefraoui, M. Aissaoui, and M. Eleuldj, "OpenStack: Toward an
vcpus can be access. Furthermore, the instant status of nova Open-Source Solution for Cloud Computing," International Journal of
service and neutron can be access, by which we can locate Computer Applications, vol. 55, pp. 38-42, 2012.
error components of openstack in a quick way. Other [3] T. Rosado and J. Bernardino, An overview of openstack architecture,
information is also included, such as url of each openstack 2014.
components and refresh rate of openstack-exporter. [4] K. Fatema, V. C. Emeakaroha, P. D. Healy, J. P. Morrison, and T. Lynn,
"A survey of Cloud monitoring tools: Taxonomy, capabilities and
objectives," Journal of Parallel & Distributed Computing, vol. 74, pp.
IV. CONCLUSION 2918-2933, 2014.
In this paper, we propose a microservices-based openstack [5] S. Newman, "Building microservices : designing fine-grained systems,"
2015.
monitoring system named openstack-reporter. The openstack-
[6] “ Microservices,” martinfowler.com. [Online]. Available:
reporter is broken down into three applications and each https://martinfowler.com/articles/microservices.html. [Acc.: 06-Feb-18].
component focus on single function. These three elements are [7] N. Dragoni, S. Giallorenzo, A. L. Lafuente, M. Mazzara, F. Montesi, R.
openstack-exporter, prometheus and grafana. A ll the Mustafin, et al., "Microservices: Yesterday, Today, and Tomorrow,"
applications are running in the kubernetes cluster which 2017.
manages the life circle of these applications in an easy way. [8] A. Bucchiarone, N. Dragoni, S. Dustdar, S. T. Larsen, and M. Mazzara,
The connecting among three applications are implemented by "From Monolithic to Microservices: An experience report," IEEE
using kubernetes DNS-based service discovery and RBAC Software, vol. 35, pp. 50-55, 2018.
mechanism where RBAC allows applications to access [9] A. Singleton, "The Economics of Microservices," IEEE Cloud
Computing, vol. 3, pp. 16-20, 2016.
kubernetes resources and service discovery mechanism helps to
detect service automatically. The docker images of the three [10] H. Knoche and W. Hasselbring, "Using Microservices for Legacy
Software Modernization," IEEE Software, vol. 35, pp. 44-49, 2018.
applications can be downloaded from docker hub, and
[11] H. Khazaei, C. Barna, N. Beigi-Mohammadi, and M. Litoiu, "Efficiency
kubernetes configuration can be found in github. The Analysis of Provisioning Microservices," in IEEE International
monitoring system is all based on configurations which means Conference on Cloud Computing Technology & Science, 2017.
it is easy to reconfigure to accommodate user demands. We [12] R. K. Barik, R. K. Lenka, K. R. Rao, and D. Ghose, "Performance
have applied the monitoring results to our openstack platform Analysis of Virtual Machines and Containers in Cloud Computing," in
using openstack-reporter and obviously it is the most of International Conference on Computing, 2017.
openstack metrics that can be captured. In the future, the [13] V. Medel, O. Rana, J. A. Banares, and U. Arronategui, "Modelling
openstack-reporter w ill be in incorporated to our openstack performance & resource management in kubernetes," in International
Conference on Utility & Cloud Computing, 2016.

709

Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 10,2020 at 15:24:40 UTC from IEEE Xplore. Restrictions apply.