Monitoring WebSphere Liberty JEE6 applications in Red Hat OpenShift

4 min readDec 2, 2020

WebSphere Liberty and OpenLiberty provides out-of-the-box support for Prometheus metrics scraping unless your application requires JEE6 features or conflicts with the Liberty cdi features. In this article I’ll describe how you can monitor Liberty applications without using the mpMetrics features.

Prometheus and Grafana

Prometheus and Grafana are widely used in Kubernetes to monitor applications. Prometheus is a monitoring system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. Data is visualized using Grafana.

Kubernetes Cluster Monitoring

Red Hat OpenShift monitoring is focused on cluster monitoring and provides data related to the overall cluster state such as pod state (running, pending, error etc), cluster CPU usage and cluster RAM usage. Individual pod data is also available including CPU and RAM usage. This data is useful for an overall cluster view but it doesn’t provide application specific data.

Application Monitoring

Prometheus can pull metrics data from applications using scraping. It is the responsibility of the application to make their metrics available on a /metrics endpoint which Prometheus scrapes and stores the data in its database for later visualization using Grafana.

Metrics related to connection pools, thread pools, CPU usage, memory usage and JVM heap usage are more useful for a Java application than the basic pod metrics available from cluster monitoring.

Metrics scraping with WebSphere Liberty

WebSphere Liberty provides a mpMetrics feature which publishes application server metrics on a /metrics endpoint that can be read by Prometheus. However, the mpMetrics feature requires the cdi-1.2 or newer feature which conflicts with JavaEE6 applications or applications that require the cdi-1.0 feature. This means that there are many applications that therefore can’t use the mpMetrics feature and it is these applications that are the focus of this article.

There is however a solution to this problem which utilizes the Prometheus JMX exporter. WebSphere Liberty provides a monitor-1.0 feature which enables the server Performance Monitoring Infrastructure (PMI) and makes the monitoring data available via MXBeans. Prometheus however cannot natively scrape MXBeans. This is where the Prometheus JMX exporter comes in.

Prometheus JMX exporter configuration

The Prometheus JMX exporter connects to any MXBeans on a JVM, retrieves their data and exposes the results on a /metrics endpoint so that the data can be scraped by Prometheus.

In order to use the JMX Exporter the monitor-1.0 feature should be enabled in server.xml as shown below

<server>
<featureManager>
<feature>monitor-1.0</feature>
...
</featureManager>
...
</server>

The JMX exporter is made up of a JAR file and a configuration file. The configuration file used for the Customer Order Services application is shown below. These files are typically placed in the /opt/ibm/wlp/usr/shared/resources/jmx_exporter folder on your Docker image

---
startDelaySeconds: 0
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

In order to connect the JMX exporter and configuration file to Liberty a javaagent parameter must be provided to the JVM on start up. Liberty uses the jvm.options file to provide JVM parameters. An example jvm.options file is shown below.

-javaagent:/opt/ibm/wlp/usr/shared/resources/jmx_exporter/jmx_prometheus_javaagent-0.11.0.jar=9081:/opt/ibm/wlp/usr/shared/resources/jmx_exporter/jmx-config.yml

This configuration results in the Liberty metrics being published on http://servername:9081/metrics which can then be scraped by Prometheus.

Example Red Hat OpenShift 4.5 configuration

You will need to configure Red Hat OpenShift to monitor applications using the instructions in the documentation. This will allow Prometheus to scrape your application which is disabled by default.

You will also need to install your own Grafana instance in to Red Hat OpenShift 4.5 using the instructions in this blog post. This is required as the Grafana packaged with Red Hat OpenShift is read-only and doesn’t allow you to import your own Dashboards.

With your application deployed to Red Hat OpenShift, you can then create a ServiceMonitor definition that is used to tell Prometheus how to scrape your application. The example below shows a ServiceMonitor that attaches to a Service and scrapes the default /metrics url on thehttp port every 30s

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: cos-monitor
  name: cos-monitor
  namespace: cos-liberty-tekton
spec:
  endpoints:
  - interval: 30s
    port: http
    scheme: http
  selector:
    matchLabels:
      application: cos-liberty

With the ServiceMonitor in place, after a short amount of time, Prometheus will start to collect data from your Liberty instance.

I’ve provided a sample Grafana Dashboard that you can import and visualize your Liberty application. This dashboard shows how many CPUs and how much RAM the container has access to (note that if you don’t provide limits, then your container may see the entire node). The dashboard also shows information about the JVM, Servlets and Connection Pools as well as threads and the classloader.

In this article I’ve shown you how to use the monitor-1.0 feature of WebSphere Liberty in conjunction with the Prometheus JMX Exporter to collect and visualize application metrics using Prometheus and Grafana on Red Hat OpenShift 4.5

If you are interested in seeing an end-to-end scenario that takes an existing WebSphere application and moves it to Liberty and deploys it to Red Hat OpenShift, then look no further than the Runtime Modernization scenario in the IBM Application Modernization Playbook.