One place for hosting & domains

      Monitoring

      How to Set Up TOBS, The Observability Stack, for Kubernetes Monitoring


      Introduction

      TOBS, short for The Observability Stack, is a pre-packaged distribution of monitoring tools and dashboard interfaces which can be installed into any existing Kubernetes cluster. It includes many of the most popular open-source observability tools with Prometheus and Grafana as a baseline, including Promlens, TimescaleDB, Alertmanager, and others. Together, these provide a straightforward, maintainable solution for analyzing server traffic and identifying any potential problems with a deployment up to a very large scale.

      TOBS makes use of standard Kubernetes Helm charts in order to configure and update deployments. It can be installed into any Kubernetes cluster, but it can be demonstrated more effectively if you’re running kubectl to manage your cluster from a local machine rather than a remote node. DigitalOcean’s Managed Kubernetes will provide you with a configuration like this by default.

      In this tutorial, you will install TOBS into an existing Kubernetes cluster, and learn how to update, configure, and browse its component dashboards.

      Prerequisites

      To follow this tutorial, you will need:

      Step 1 — Verifying your Kubernetes Configuration

      In order to install TOBS, you should first have a valid Kubernetes configuration set up with kubectl from which you can ping your worker nodes. You can test this by running kubectl get nodes:

      If kubectl is able to connect to your Kubernetes cluster and it’s up and running as expected, this command will return a list of nodes with the Ready status:

      Output

      NAME STATUS ROLES AGE VERSION pool-uqv8a47h0-ul5a7 Ready <none> 22m v1.21.5 pool-uqv8a47h0-ul5am Ready <none> 21m v1.21.5 pool-uqv8a47h0-ul5aq Ready <none> 21m v1.21.5

      If this is successful, you can move on to Step 2. If not, you should review your configuration details for any issues.

      By default, kubectl will look for a file at ~/.kube/config in order to understand your environment. In order to verify that this file exists and contains valid YAML syntax, you can run head on it to view its first several lines, i:

      Output

      apiVersion: v1 clusters: - cluster: certificate-authority-data: …

      If the file does not exist, ensure that you are logged in as the same user that you configured Kubernetes with. ~/ paths reflect individual users’ home directories, and Kubernetes configurations are saved per-user by default.

      If you are using DigitalOcean’s Managed Kubernetes, ensure that you have run the doctl kubernetes cluster kubeconfig save command after setting up a cluster so that your local machine can authenticate to it. This will create a ~/.kube/config file:

      • doctl kubernetes cluster kubeconfig save your-cluster-name

      If you are using this machine to access multiple clusters, you should review the Kubernetes documentation on using environment variables and multiple configuration files in order to avoid conflicts. After configuring your kubectl environment, you can move on to installing TOBS in the next step.

      Step 2 — Installing TOBS and Testing Your Endpoints

      TOBS includes the following components:

      • Prometheus is a time series database and monitoring tool that works by polling metrics endpoints and scraping and processing the data exposed by these endpoints. It allows you to query this data using PromQL, a time series data query language.
      • Alertmanager, usually deployed alongside Prometheus, forms the alerting layer of the stack, handling alerts generated by Prometheus and deduplicating, grouping, and routing them to integrations like email or PagerDuty. To learn more about Alertmanager, consult the Prometheus documentation on alerting.
      • Grafana is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data.
      • kube-state-metrics is an add-on agent that listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects like Deployments and Pods. These metrics are served as plaintext on HTTP endpoints and consumed by Prometheus.
      • Lastly is node-exporter, a Prometheus exporter that runs on cluster nodes and provides OS and hardware metrics like CPU and memory usage to Prometheus. These metrics are also served as plaintext on HTTP endpoints and consumed by Prometheus.

      In order to install TOBS, you first need to run the TOBS installer on your control-plane. This will set up the tobs command and configuration directories. As mentioned in the prerequisites, the tobs command is only designed to work on Linux/macOS/BSD systems (like the official Kubernetes binaries), so if you have been using Windows up to now, you should be working in the Windows Subsystem for Linux environment.

      Retrieve and run the TOBS installer:

      • curl --proto '=https' --tlsv1.2 -sSLf https://tsdb.co/install-tobs-sh |sh

      Output

      tobs 0.7.0 was successfully installed 🎉 Binary is available at /root/.local/bin/tobs.

      You can now push TOBS to your Kubernetes cluster. This is done by a one-liner using your newly-provided tobs command:

      This will generate several lines of output and may take a few moments. Depending on your exact version of Kubernetes, there may be several warnings in the output, but you can ignore these as long as you eventually receive the Welcome to tobs message:

      Output

      WARNING: Using a generated self-signed certificate for TLS access to TimescaleDB. This should only be used for development and demonstration purposes. To use a signed certificate, use the "--tls-timescaledb-cert" and "--tls-timescaledb-key" flags when issuing the tobs install command. Creating TimescaleDB tobs-certificate secret Creating TimescaleDB tobs-credentials secret skipping to create TimescaleDB s3 backup secret as backup option is disabled. 2022/01/10 11:25:34 Transport: unhandled response frame type *http.http2UnknownFrame 2022/01/10 11:25:35 Transport: unhandled response frame type *http.http2UnknownFrame 2022/01/10 11:25:35 Transport: unhandled response frame type *http.http2UnknownFrame Installing The Observability Stack 2022/01/10 11:25:37 Transport: unhandled response frame type *http.http2UnknownFrame W0110 11:25:55.438728 75479 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ W0110 11:25:55.646392 75479 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ … 👋🏽 Welcome to tobs, The Observability Stack for Kubernetes …

      The output from this point onward will contain instructions for connecting to each of Prometheus, TimescaleDB, PromLens, and Grafana’s web endpoints in your browser. It is reproduced in full below for reference:

      Output

      ############################################################################### 🔥 PROMETHEUS NOTES: ############################################################################### Prometheus can be accessed via port 9090 on the following DNS name from within your cluster: tobs-kube-prometheus-prometheus.default.svc.cluster.local Get the Prometheus server URL by running these commands in the same shell: tobs prometheus port-forward The Prometheus alertmanager can be accessed via port 9093 on the following DNS name from within your cluster: tobs-kube-prometheus-alertmanager.default.svc.cluster.local Get the Alertmanager URL by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace default -l "app=alertmanager,alertmanager=tobs-kube-prometheus-alertmanager" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $POD_NAME 9093 WARNING! Persistence is disabled on AlertManager. You will lose your data when the AlertManager pod is terminated. ############################################################################### 🐯 TIMESCALEDB NOTES: ############################################################################### TimescaleDB can be accessed via port 5432 on the following DNS name from within your cluster: tobs.default.svc.cluster.local To get your password for superuser run: tobs timescaledb get-password -U <user> To connect to your database, chose one of these options: 1. Run a postgres pod and connect using the psql cli: tobs timescaledb connect -U <user> 2. Directly execute a psql session on the master node tobs timescaledb connect -m ############################################################################### 🧐 PROMLENS NOTES: ############################################################################### PromLens is a PromQL query builder, analyzer, and visualizer. You can access PromLens via a local browser by executing: tobs promlens port-forward Then you can point your browser to http://127.0.0.1:8081/. ############################################################################### 📈 GRAFANA NOTES: ############################################################################### 1. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster: tobs-grafana.default.svc.cluster.local You can access grafana locally by executing: tobs grafana port-forward Then you can point your browser to http://127.0.0.1:8080/. 2. The 'admin' user password can be retrieved by: tobs grafana get-password 3. You can reset the admin user password with grafana-cli from inside the pod. tobs grafana change-password <password-you-want-to-set>

      Each of this is provided with a DNS name internal to your cluster so that they can be accessed from any of your worker nodes, e.g. tobs-kube-prometheus-alertmanager.default.svc.cluster.local for Prometheus. In addition, there is a port forwarding command configured for each that allows you to access them from a local web browser.

      In a new terminal, run tobs prometheus port-forward:

      • tobs prometheus port-forward

      This will occupy the terminal as long as the port forwarding process is active. You can press Ctrl+C to gracefully quit a blocking process such as this one when you want to stop forwarding the port. Next, in a web browser, go to the URL http://127.0.0.1:9090/. You should see the full Prometheus interface running and producing metrics from your cluster:

      Prometheus welcome

      You can do the same for Grafana, which is accessible at http://127.0.0.1:8080/ as long as port forwarding is active in another process. First, you’ll need to use the get-password command provided by the installer output:

      • tobs grafana get-password

      Output

      your-grafana-password

      You can then use this password to log into the Grafana interface by running its port forwarding command and opening http://127.0.0.1:8080/ in your browser.

      • tobs grafana port-forward

      Grafana welcome

      You now have a working TOBS stack running in your Kubernetes cluster. You can refer to the individual components’ documentation in order to learn their respective features. In the last step of this tutorial, you’ll learn how to make updates to the TOBS configuration itself.

      Step 3 — Editing TOBS Configurations and Upgrading

      TOBS’ configuration contains some parameters for the individual applications in the stack, as well as some parameters for the TOBS deployment itself. It is generated and stored as a Kubernetes Helm chart. You can output your current configuration by running tobs helm show-values. However, this will output the entire long configuration to your terminal, which can be difficult to read. You can instead redirect the output to a file with the .yaml extension, because Helm charts are all valid YAML syntax:

      • tobs helm show-values > values.yaml

      The file contents will look like this:

      ~/values.yaml

      2022/01/10 11:56:37 Transport: unhandled response frame type *http.http2UnknownFrame
      # Values for configuring the deployment of TimescaleDB
      # The charts README is at:
      #    https://github.com/timescale/timescaledb-kubernetes/tree/master/charts/timescaledb-single
      # Check out the various configuration options (administration guide) at:
      #    https://github.com/timescale/timescaledb-kubernetes/blob/master/charts/timescaledb-single/admin-guide.md
      cli: false
      
      # Override the deployment namespace
      namespaceOverride: ""
      …
      

      You can review the additional parameters available for TOBS’ configuration by reading the TOBS documentation

      If you ever modify this file in order to update your deployment, you can re-install TOBS over itself using the updated configuration. Just pass the -f option to the tobs install command with the YAML file as an additional argument:

      • tobs install -f values.yaml

      Finally, you can upgrade TOBS with the following command:

      This performs the equivalent of a helm upgrade by fetching the newest upstream chart.

      Conclusion

      In this tutorial, you learned to deploy and configure TOBS, The Observability Stack, on an existing Kubernetes cluster. TOBS is particularly helpful because it eliminates the need to individually maintain configuration details for each of these apps, while providing standardized monitoring for the applications running on your cluster.

      Next, you might want to learn how to use Cert-Manager to handle HTTPS ingress to your Kubernetes cluster.



      Source link

      How to Deploy Percona Monitoring and Management (PMM) with One-Click Apps


      Updated by Linode

      Contributed by
      Linode

      Marquee image for How to Deploy Percona Monitoring and Management (PMM) with One-Click Apps

      Percona (PMM) One-Click App

      Percona Monitoring and Management (PMM) is an open-source tool which provides a GUI powered by Grafana for monitoring and managing MySQL, MariaDB, PostgreSQL, and MongoDB databases. You can use PMM to easily observe important metrics, logging, and statistics related to your databases and the hosts they run on. Additionally, it includes a number of tools which can help to optimize your database’s performance, manage all database instances, and track and identify potential security threats. Linode’s Percona (PMM) One-Click App deploys a Linode with PMM installed and ready for you to begin monitoring your databases.

      PMM Architecture Overview

      The PMM tool uses a client-server model which makes it easy to scale monitoring across several databases hosted on various remote hosts. See the table below to better understand how each PMM component fits into the context of Linode’s Percona (PMM) One-Click App.

      Component Description
      PMM Server The Percona (PMM) One-Click App deploys an instance of the PMM Server, which includes the Grafana web interface to visualize all the data collected from the databases it monitors.
      PMM Client You will need to install the PMM Client on any Linode that hosts a database that you would like to monitor. The PMM Client will help you connect to the PMM Server and relay host and database performance metrics to the PMM Server.

      Note

      Deploy a Percona (PMM) One-Click App

      Linode’s One-Click App Marketplace allow you to easily deploy software on a Linode using the Linode Cloud Manager. To access Linode’s One-Click App Marketplace:

      1. Log in to your Linode Cloud Manager account.

      2. From the Linode dashboard, click on the Marketplace button in the left-hand navigation menu.

      3. The Linode creation page will appear, with the One-Click and Marketplace tabs pre-selected.

      4. Under the Select App section, select the app you would like to deploy:

        Select a One-Click App to deploy

      5. Once you have selected the app, proceed to the app’s Options section and provide values for the required fields.

      Percona (PMM) Options

      Configuration Description
      Hostname The hostname to assign to your new Linode. Required.

      Linode Options

      The following configuration options are possible for your Linode server:

      Configuration Description
      Select an Image Debian 10 and Ubuntu 20.04 are currently the only images supported by the Percona (PMM) One-Click App. Required.
      Region The region where you would like your Linode to reside. In general, it’s best to choose a location that’s closest to you. For more information on choosing a DC, review the How to Choose a Data Center guide. You can also generate MTR reports for a deeper look at the network routes between you and each of our data centers. Required.
      Linode Plan Your Linode’s hardware resources. Percona recommends roughly 1 GB of storage on your PMM Server for each database node you would like to monitor. For more information on Percona’s system requirements see their official documentation. If you decide that you need more or fewer hardware resources after you deploy your app, you can always resize your Linode to a different plan. Required.
      Linode Label The name for your Linode, which must be unique between all of the Linodes on your account. This name will be how you identify your server in the Cloud Manager’s Dashboard. Required.
      Root Password The primary administrative password for your Linode instance. This password must be provided when you log in to your Linode via SSH. It must be at least 6 characters long and contain characters from two of the following categories: lowercase and uppercase case letters, numbers, and punctuation characters. Your root password can be used to perform any action on your server, so make it long, complex, and unique. Required.

      When you’ve provided all required Linode Options, click on the Create button. Your PMM app will complete installation anywhere between 1-5 minutes after your Linode has finished provisioning.

      Software Included

      The Percona (PMM) One-Click App will install the following required software on your Linode:

      Software Description
      Docker Docker is used to containerize PMM.
      Grafana Grafana is the visualization tool used by PMM to provides charts, graphs, and alerts for the databases that are being monitored.
      pmm-admin pmm-admin is a command-line tool used to manage PMM Clients.

      Note

      Currently, Linode does not manage software and systems updates for One-Click Apps. It is up to the user to perform routine maintenance on software deployed in this fashion.

      Getting Started after Deployment

      Access your PMM Server’s Grafana Dashboard

      After the PMM Server has finished installing, you will be able to access its Grafana dashboard over http:// and your Linode’s IPv4 address. To find your Linode’s IPv4 address:

      1. Click on the Linodes link in the sidebar. You will see a list of all your Linodes.

      2. Find the Linode you just created when deploying your app and select it.

      3. Navigate to the Networking tab.

      4. Your IPv4 address will be listed under the Address column in the IPv4 table.

      5. Copy and paste the IPv4 address into a browser window. Ensure you are using http://.

      6. You will see the PMM Home Dashboard actively monitoring your server:

        Percona Home Page

      Installing the PMM Client

      To begin monitoring a database node, you will need to install the PMM Client on the Linode that hosts your database and connect the node to the PMM Server.

      Note

      The PMM Server deployed with Linode’s Percona (PMM) One-Click App is compatible with PMM Client version 1.
      1. Connect to your Linode via SSH.

      2. Update your system’s software.

        Debian or Ubuntu:

        sudo apt-get update && apt-get upgrade
        

        CentOS:

        sudo yum update
        
      3. Install the PMM Client (version 1):

        Debian or Ubuntu:

        sudo apt-get install pmm-client
        

        CentOS:

        sudo yum install pmm-client
        
      4. Connect your database node to the PMM Server. Replace 192.0.2.0 with your PMM Server’s IPv4 address.

        Note

        sudo pmm-admin config --server 192.0.2.0
        

        Once complete, you should see a similar output:

          
        OK, PMM server is alive.
        
        PMM Server      | 192.0.2.0
        Client Name     | li222-111
        Client Address  | 192.0.2.1
              
        

      Monitor a Database Instance

      Once your database node and your PMM Server are communicating, the final step is to add your database instance to be monitored by your PMM Server. These steps are completed on your Database node.

      1. Connect to your Linode via SSH.

      2. Add your database instance to be monitored by your PMM Server. This command will add a MySQL database instance to be monitored. You can replace mysql with the database type that your node is hosting (i.e. postgresql or mongodb). See Percona’s Managing PMM Client documentation for details.

        pmm-admin add mysql
        

        Your Percona dashboard should now be monitoring your MySQL service.

        https://www.linode.com/perconafinal.png

        Note

      Next Steps

      The Percona Monitoring Management Tool is very powerful and can be configured to monitor and display various pieces of your database instances. Once you have completed the basic configurations outlined in this guide, you should review Percona’s documentation to learn more about the PMM tool. Here are some suggested topics:

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      Survey: Monitoring and OS Maintenance are IT Pros Biggest Infrastructure Time Wasters


      If time is considered a finite resource, IT pros are feeling a shortage.

      For INAP’s second annual State of IT Infrastructure Management report, we asked IT professionals to list the routine infrastructure activities taking too much and too little of their time. Once again, monitoring topped the “too much time” list, while designing and implementing new solutions ranked No. 1 in the “not enough time” category.

      Overall, 59 percent of IT pros are frustrated by the time spent on routine infrastructure activities and 84 percent agreed that they “could bring more value to their organization if they spent less time on routine tasks”—up 7 points from 2018.

      The survey was conducted late last year among 500 IT senior leaders and infrastructure managers in the United States and Canada. The margin of error was +/- 5 percent.

      Participants also shared how often their personal time is interrupted and how they would spend their time if they were given 16 hours back to use as they please.

      Check out the results below and download a copy of the full report here.

      How IT Pros Spend (And Don’t Spend) Their Time

      Here’s the full list of routine infrastructure activities alongside IT’s assessment of whether each is getting the attention it deserves.

      IT's Time

      Only 23 percent of participants said they don’t spend enough time monitoring infrastructure, which is less than half the rate of those who consider this routine activity to be something that is eating into time that could be spent elsewhere (48 percent).

      Operating system and hardware maintenance came in second and third, at 42 percent and 40 percent, respectively.

      Nearly half (47 percent) of IT pros want to spend more time on designing and implementing new solutions, compared to 28 percent who already spend too much time.

      IT pros remain polarized, as they were in the 2018 report, as to whether the amount of time spent securing their infrastructure is hitting the mark, with 39 percent saying it’s too much and 42 percent saying it’s not enough. Information security management/vulnerability migration is also the activity where the highest percentage of IT pros went one way or another on the issue, as only 19 percent fell into the “neither” category.

      Senior leaders were far more likely to say they spend too much time on security compared to non-senior infrastructure managers—30 percent vs. 13 percent. They were also more likely to say they don’t focus enough on OS maintenance—20 percent vs 6 percent of non-leaders.

      How IT Pros Would Like to Spend Their Time

      Survey respondents say their personal time is disrupted by work responsibilities related to server and/or cloud infrastructure an average of 6.24 times per month—up slightly from 5.9 times in 2018’s report.

      With so much time—on and off the clock—being dedicated to upkeep and maintenance, we once again asked, “What would you do if we gave you 16 hours back in your week?”

      16 hours

      Application related answers make up three of the top four activities this year, with “enhancing existing applications” and “optimizing existing environments for application performance” coming in third and fourth, respectively. Reclaiming work-life balance fell to second, after claiming the top spot in 2018.

      In the first annual State of IT Infrastructure Management report, IT pros noted that their departments are the key driver of their organization’s digital transformation initiatives, but they are spending too much time on routine tasks, focusing on functions that are “just keeping the lights on.” This sentiment continued in 2019. The list of activities noted in the chart above can be considered the opportunity costs of these routine tasks.

      Have you read checked out our second annual State of IT Infrastructure Management report yet? If not, download a free copy and get your report card for the hybrid IT and multicloud era:

      Laura Vietmeyer


      READ MORE



      Source link