One place for hosting & domains

      How to Deploy TOBS on LKE

      , by Rajakavitha Kodhandapani

      Traducciones al Español

      Estamos traduciendo nuestros guías y tutoriales al Español. Es
      posible que usted esté viendo una traducción generada
      automáticamente. Estamos trabajando con traductores profesionales
      para verificar las traducciones de nuestro sitio web. Este proyecto
      es un trabajo en curso.

      Create a Linode account
      to try this guide with a $100 credit.

      This credit will be applied to any valid services used during your first 60 days.

      In this guide, deploy
      TOBS to your Linode Kubernetes Engine (LKE) cluster using
      Helm. And use kubectl port-forward for local access to your monitoring interfaces.

      The Prometheus Operator Monitoring Stack

      TOBS, short for The Observability Stack, is a pre-packaged distribution of monitoring tools and dashboard interfaces which can be installed on any existing Kubernetes cluster. It includes many of the most popular open-source observability tools such as Prometheus, Grafana, Promlens, TimescaleDB, and others. Together, these provide a maintainable solution to analyze the traffic on the server and identify any potential problems with a deployment. You can use Helm charts to configure and update TOBS deployments.

      TOBS includes the following components:

      • OpenTelemetry collector is deployed to collect traces.
      • Alertmanager, is deployed alongside Prometheus, forms the alerting layer of the stack, and handles alerts generated by Prometheus.
      • Grafana is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data.
      • PromLens helps users build PromQL queries with ease. PromLens is a PromQL query builder that helps you build, understand, and fix your queries much more effectively.
      • TimescaleDB is for long-term storage of metric data. Long-term storage provides the ability to perform post-hoc analysis on metric data over long periods of time. Such data analysis can be used for capacity planning, identifying slow-moving regressions, trend analysis, auditing, and more. For information about connecting to the database from the cluster, see
        TimescaleDB Documentation
      • Promscale provides the translation layer between Prometheus and the database. It allows the Prometheus server to store and retrieve metrics from TimescaleDB, and allows users to use PromQL on Promscale and Prometheus.
      • Prometheus is an open-source systems monitoring and altering stack. It has become the de-facto standard in metric monitoring and is the basis of standards such as OpenMetrics. It allows you to monitor and understand how your infrastructure and applications are performing. Service discovery allows Prometheus to automagically discover components within your Kubernetes cluster that are already emitting metrics.
      • kube-state-metrics exports the metrics related to Kubernetes resources such as the status and count of Kubernetes resources, with visibility of the desired resources and the current resources, as well as the trends in your cluster.
      • Node-Exporter is deployed to export node related metrics such as CPU, memory usage, and others from the Kubernetes cluster.

      Before You Begin


      This guide was written using
      Kubernetes version 1.23.
      1. Deploy an LKE Cluster. This guide was written using an example node pool with three
        2 GB Linodes. Depending on the workloads you plan to deploy on your cluster, you may consider using Linodes with more available resources.

      2. Install
        Helm 3 to your local environment.

      3. Install
        kubectl to your local environment and
        connect to your cluster.

      4. Create the monitoring namespace on your LKE cluster:

        kubectl create namespace monitoring
      5. Add the stable Helm charts repository to your Helm repos:

        helm repo add stable
      6. Update your Helm repositories:

        helm repo update

      TOBS Minimal Deployment

      In this section, learn to deploy TOBS for individual/local access with kubectl

      Deploy The Observability Stack

      1. Install a certificate manager for your LKE cluster:

         kubectl apply -f
      2. Using Helm, deploy the TOBS release labeled lke-monitor in the monitoring namespace on your LKE cluster:

        helm repo add timescale
        helm repo update
        helm install --wait lke-monitor timescale/tobs --namespace monitoring
      3. Verify that the Prometheus Operator has been deployed to your LKE cluster and its components are running and ready by checking the pods in the monitoring namespace:

        kubectl -n monitoring get pods

        You should see a similar output to the following:

        NAME                                                        READY   STATUS      RESTARTS      AGE
        alertmanager-tobs-kube-prometheus-alertmanager-0            2/2     Running     0             2m13s
        lke-monitor-connection-secret-j4sdh                         0/1     Completed   0             2m35s
        lke-monitor-grafana-54d979dcf5-tkkgj                        3/3     Running     2 (65s ago)   2m32s
        lke-monitor-grafana-db-swm8g                                0/1     Completed   3             2m35s
        lke-monitor-kube-state-metrics-6bc5c44b9-g8r5g              1/1     Running     0             2m27s
        lke-monitor-prometheus-node-exporter-b4vvg                  1/1     Running     0             2m33s
        lke-monitor-prometheus-node-exporter-bbcnd                  1/1     Running     0             2m34s
        lke-monitor-prometheus-node-exporter-frrfp                  1/1     Running     0             2m26s
        lke-monitor-promlens-569cfbd586-bkhrr                       1/1     Running     0             2m34s
        lke-monitor-promscale-86d574986c-9wj2z                      1/1     Running     4 (64s ago)   2m27s
        lke-monitor-timescaledb-0                                   1/1     Running     0             2m30s
        opentelemetry-operator-controller-manager-8cf5c85c8-krdj5   2/2     Running     0             2m27s
        prometheus-tobs-kube-prometheus-prometheus-0                2/2     Running     0             2m13s
        tobs-kube-prometheus-operator-5b4f674986-55r4k              1/1     Running     0             2m34s

      Access Monitoring Interfaces with Port-Forward

      1. List the services running in the monitoring namespace and review their respective ports:

        kubectl -n monitoring get svc

        You should see an output similar to the following:

        NAME                                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                           AGE
        alertmanager-operated                                       ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   3m41s
        lke-monitor                                                 ClusterIP    <none>        5432/TCP                     4m3s
        lke-monitor-config                                          ClusterIP   None             <none>        8008/TCP                     4m3s
        lke-monitor-grafana                                         ClusterIP   <none>        80/TCP                       4m3s
        lke-monitor-kube-state-metrics                              ClusterIP    <none>        8080/TCP                     4m3s
        lke-monitor-prometheus-node-exporter                        ClusterIP    <none>        9100/TCP                     4m3s
        lke-monitor-promlens                                        ClusterIP     <none>        80/TCP                       4m3s
        lke-monitor-promscale-connector                             ClusterIP    <none>        9201/TCP,9202/TCP            4m3s
        lke-monitor-replica                                         ClusterIP   <none>        5432/TCP                     4m3s
        opentelemetry-operator-controller-manager-metrics-service   ClusterIP     <none>        8443/TCP                     4m3s
        opentelemetry-operator-webhook-service                      ClusterIP     <none>        443/TCP                      4m3s
        prometheus-operated                                         ClusterIP   None             <none>        9090/TCP                     3m41s
        tobs-kube-prometheus-alertmanager                           ClusterIP     <none>        9093/TCP                     4m3s
        tobs-kube-prometheus-operator                               ClusterIP    <none>        443/TCP                      4m3s
        tobs-kube-prometheus-prometheus                             ClusterIP   <none>        9090/TCP                     4m3s

        From the above output, the resource services you will access have the corresponding ports:

        Resource Service Name Port
        Prometheus tobs-kube-prometheus-prometheus 9090
        Alertmanager tobs-kube-prometheus-alertmanager 9093
        Grafana lke-monitor-grafana 80
      2. Use kubectl
        port-forward to open a connection to a service, then access the service’s interface by entering the corresponding address in your web browser:


        Press control+C on your keyboard to terminate a port-forward process after entering any of the following commands.

        • To provide access to the Prometheus interface at the address in your web browser, enter:

          kubectl -n monitoring \
          port-forward \
          svc/tobs-kube-prometheus-prometheus \
        • To provide access to the Alertmanager interface at the address in your web browser, enter:

          kubectl -n monitoring \
          port-forward \
          svc/tobs-kube-prometheus-alertmanager  \
        • To provide access to the Grafana interface at the address in your web browser, enter:

          kubectl -n monitoring \
          port-forward \
          svc/lke-monitor-grafana  \

          When accessing the Grafana interface, log in as admin. You can get the password using:

          kubectl get secret --namespace monitoring lke-monitor-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

          The Grafana dashboards are accessible at Dashboards > Manage from the left navigation bar.

      TOBS eliminates the need to maintain configuration details for each of the applications, while providing standardized monitoring for the applications running on your cluster.

      More Information

      You may wish to consult the following resources for additional information
      on this topic. While these are provided in the hope that they will be
      useful, please note that we cannot vouch for the accuracy or timeliness of
      externally hosted materials.

      This page was originally published on

      Join the conversation.
      Read other comments or post your own below. Comments must be respectful,
      constructive, and relevant to the topic of the guide. Do not post external
      links or advertisements. Before posting, consider if your comment would be
      better addressed by contacting our
      Support team or asking on
      Community Site.

      Source link