One place for hosting & domains

      January 2022

      12 Clever Ways to Boost Your Income From Affiliate Marketing


      Affiliate marketing is one of the most effective ways to make money through your website. However, the way everyone’s doing it isn’t necessarily the way you should be doing it. For example, maybe you’re not a blogger, or perhaps you want to be original. Alternatively, maybe you do have a blog but want to explore new ways to expand your reach.

      Fortunately, there are many different ways to promote your affiliate products outside of your blog. One or more of these methods may be what you’ve been looking for to help you earn extra money from your efforts.

      In this article, we’ll discuss what affiliate marketing is and why you may want to try different approaches to it. Then we’ll review 12 less common (but potentially effective) ways to take your affiliate marketing strategy to the next level. Let’s get started!

      What Affiliate Marketing Is (And How It Works)

      Affiliate marketing is a digital marketing model that enables companies to reach more people at a lower cost than traditional advertising. Sellers do this by compensating third-party publishers, or affiliates, for promoting their products or services.

      If you have an active blog, a YouTube channel, or a social media page, this makes you a good candidate for becoming an affiliate marketer. As an affiliate, you get to promote the seller’s offerings on your preferred platforms. In essence, you’re giving the seller a wider audience.

      When someone makes a purchase through your channel, you’ll normally earn a commission on that sale. Alternatively, the brand may pay you for bringing in leads or driving traffic.

      The vendor will give you an affiliate link that contains an ID that is unique to you. That way, when someone purchases a product through this URL, the seller can attribute the sale to you and give you a commission.

      This model carries little risk for sellers because they don’t have to pay anything until you make a sale for them. However, it could potentially be very profitable for you as a publisher. It may take you a while to gain traction, but you can start earning a passive income without much effort once you do.

      Why You Might Want to Try Affiliate Marketing Outside of Your Blog

      affiliate marketing

      Blogging is one of the most common ways to promote products as an affiliate. Starting a blog can be a very simple process, and you can produce a lot of content without spending any money.

      However, you may find yourself in a situation that makes blogging less than optimal for you. For example, you may not enjoy writing, or you have other skills you’d like to focus on, such as video-making and photography.

      Alternatively, you may have built a following on another medium, such as a social media platform. Therefore, you may want to monetize that page instead of building another one from scratch.

      However, there may be another very good reason to promote products outside a blog. As a professional marketer, you may be looking to diversify your income. That way, if one source fails, you have another recourse.

      Setting up multiple channels can also help you grow your audience. However, if you do decide to promote products on more than one channel, it’s important to remember that focus is key in business. You may get better returns from managing one to three mediums very well, rather than five or six haphazardly.

      To be a good affiliate marketer, you’ll need to get to know your audience. You’ll have to consider what places your target users frequent and what messages they respond to. Those factors should drive the bulk of your medium choice.

      How to Boost Your Affiliate Marketing Income (12 Clever Strategies)

      Now that you know the importance of evaluating all the channels available to you, it’s time to look at some effective options. Here are 12 strategies to help you stand out from your competition and take your affiliate marketing to the next level.

      1. Promote Your Products in Email Newsletters

      Since it’s such a strong medium, email will likely never go away. You can think of a customer’s email inbox as a curated feed of mostly relevant information – and they’re in full control of it.

      Typically, if a user sees no value in an email, they delete it. If they see no value in a string of emails from the same company, they unsubscribe. On the other hand, if they like the emails they subscribed to, they’ll be more likely to engage with their content.

      A newsletter is a regular email you send to your subscribers. Newsletters help build customer loyalty over the long term. In each email (or issue), you can give your audience updates, news, or the latest tips and tricks related to your offerings.

      The technical aspects of starting a newsletter aren’t hard to handle with the right software. For example, Mailchimp comes with a solid suite of tools for email marketing.

      Mailchimp homepage

      Building a loyal audience is the challenging part but also the most rewarding. The secret is knowing your audience and talking about what matters to them.

      For example, if you share money-saving tips and strategies in your newsletters, you might recommend an accounting software that enables users to handle their finances. When readers click on an affiliate link in your newsletter, they’ll be taken straight to the product page on the vendor’s website.

      2. Review Products in Videos

      According to a December 2020 survey by Wyzowl, 84% of people report having been convinced to purchase a product or service after watching a video about it. The average amount of time people spend watching videos each week is 18 hours. What’s more, the pandemic has made 96% of viewers more likely to consume videos online.

      Video is arguably the most personal medium. People get to see how you look and sound. This transparency can create an emotional connection. In turn, that connection can make viewers more likely to buy what you’re promoting.

      YouTube is one of the most popular online video platforms. You’ll find many affiliate marketers on this channel. If you’ve ever heard someone talk about a product and tell you there’s a link in the description, there’s a good chance they’re an affiliate.

      If you don’t mind being in front of a camera, you could give this strategy a try. For example, you could review your favorite products in engaging videos and place your affiliate links in the descriptions.

      Gerald Undone reviews gear that people can use to create YouTube content in this useful review video.

      YouTube video by Geral Undone

      In the video description, you’ll see these two links:

      Affiliate links in the YouTube video’s description

      As with all marketing materials, your video should include a clear Call to Action (CTA) that encourages viewers to take the next step. For example, you could ask them to click on the provided link to purchase the product.

      3. Start a Podcast

      The podcasting industry is growing in popularity. It is estimated to reach $95 billion globally by 2028.

      As with video, podcasting allows you to tell a compelling story with your own voice. However, podcasts tend to be cheaper and easier to produce than videos. The main drawback is the missing the visual element, which may deter people who would rather watch a video.

      Before you get started, you’ll need to reflect on the theme and format of your show. You’ll also need some gear – a good microphone and a pair of headphones at a minimum – and the right software. Fortunately, you can host your podcasts on WordPress. This platform integrates well with media hosting services such as Blubrry.

      Blubrry podcast hosting

      You can recommend a relevant product or service in each episode and provide an affiliate link in the captions. In addition, we recommend establishing a consistent schedule (e.g., one episode per week). This can help keep your audience engaged in your content, which may increase your revenue from affiliate marketing.

      4. Promote Products on Social Media Sites

      In marketing, you go where your customers are. In the United States, 82% of the population is active on at least one social media profile.

      You don’t even need to have your own website to promote products on social media. Instead, your profiles and business pages can act as your main platforms. The important thing is to choose the channel that represents the best union between the products you promote and the audience you’re targeting.

      For instance, TikTok’s user base mainly consists of Generation Z. Therefore, this should be your chosen platform if you want to target the younger generation.

      Meanwhile, if the products you want to promote are visually appealing, you may want to choose an image-heavy channel, such as Pinterest or Instagram.

      Pinterest homepage

      For instance, you can create graphics for Pinterest and include an affiliate link in the pin description. You could also post your stunning images on Instagram. However, keep in mind that you cannot embed links in Instagram posts. Many affiliates get around that issue by putting a link in their bio.

      5. Write an Informative E-book

      A well-written e-book with useful, actionable advice can position you as an expert in your field. A common approach is to make the e-book available for free, often in exchange for the user’s email address.

      If you need some inspiration, you can go to Amazon Kindle and look at existing books in your industry. You may also want to check to see if they’re selling well. This will give you a hint as to the viability of your topic.

      If there are too many books on your chosen topic, the competition may be strong. Therefore, you’ll need to see what you can do better than your competition to make your e-book more appealing.

      When you’re writing your book, you’ll want to make sure to include your affiliate links in ways that are helpful to your readers. We also recommend using a soft-sell approach and only mentioning products where relevant. If you litter your book with links, you may come across as pushy and aggressive, which can put off your readers.

      It may be a while before you see results from this strategy, but it’s important to keep working on it. For instance, you can promote your e-book on your website or social media pages.

      6. Create an Online Course

      Like podcasting, online learning is growing fast. A forecast by Research and Markets estimated that the market will grow to $350 billion by 2025.

      The online learning platform Coursera saw its enrollment skyrocket by 664% in early 2020 compared to the same period in the previous year.

      Coursera homepage

      Meanwhile, Udemy saw a 425% surge during the same period. The demand for online courses remains higher than it did at pre-pandemic levels.

      While pandemic-driven lockdowns and restrictions won’t last forever, the new worldwide attitudes toward online learning may. The changes in online education, formal or informal, appear to be permanent.

      This presents a big opportunity for your business. It means that people worldwide are consuming something you can offer. While creating a high-quality online course requires a fair amount of work and time, once it’s ready, you can simply launch it on your site and start earning a passive income from it.

      To profit from this trend, you’ll need to embed product promotions seamlessly within your course material. You can do this naturally by recommending products that enable students to do what you teach them.

      However, nothing is stopping you from making your course about the product you want to promote. For instance, you could teach people how to use particular management software to run their online business more effectively.

      A high-quality online course can help you build credibility. People may be more likely to listen to your recommendations if they trust you. This will, in turn, result in higher sales.

      You could sell your course for a small fee or give it away for free. If your course is high quality, the second approach may lead to more signups.

      Skip the line and get tips right in your inbox

      Click below to sign up for more how-to’s and tutorials just like this one, delivered to your inbox.

      marketing tips

      7. Publish a Tutorial

      Online tutorials are a popular way to learn new skills quickly. They often take the form of step-by-step articles or how-to videos. In particular, video tutorials differ from product review videos in that they are intended to teach the audience a specific skill.

      Tutorials are a fantastic medium for affiliate promotion, whether in video or article form. You can teach your audience a skill that requires using a product you promote. For example, you could teach them how to do their taxes. Then, you would introduce them to the tax-accounting software you promote.

      Alternatively, you can teach users how to use the very product you’re promoting. For example, you might show them how to use a customer relationship management (CRM) platform and encourage them to purchase it by providing an affiliate link.

      To publish your tutorials, you have a few different options. For articles, you could use a platform such as Medium, where you can write and edit your articles directly while keeping track of audience statistics.

      Medium homepage

      For videos, YouTube is a great platform, as we discussed earlier. However, you could also publish your tutorial videos on your Facebook page or website.

      8. Post in Online Communities and Forums

      We tend to get together with those who share our interests and views. That’s the case with online communities. These are virtual places where people can exchange information on their common interests. They also enable users to support each other and share advice.

      Social media communities exist as subcommunities of the larger social media landscape. Here is an example of a very active Facebook community, Science and Technology:

      Science and Technology community on Facebook

      You’ll also find plenty of online forums on the internet. These communities exist outside of social media as places to exchange useful information or advice. Reddit is one of the most well-known forum sites.

      An example of a community forum on Reddit

      As a marketer, online communities are a boon for your business. You can communicate with people directly and answer any questions they may have. Often, you can keep your affiliate links as part of your profile so that users can visit them at any time.

      As you engage with online communities, you must abide by their community guidelines. Moreover, you’ll want to make sure you contribute valuable information consistently; otherwise, you may appear as though you’re only there for your own self-interest.

      9. Launch an Email Marketing Campaign

      We’ve already discussed one common form of email marketing: newsletters. However, there are other options worth looking at, such as:

      • Standard promotional campaigns: This is a one-time email to promote a specific product or service.
      • Seasonal campaigns: Typically, these emails promote special offers around holidays.
      • Triggered email series: This is a series of emails sent sequentially in response to an event, such as a product download, survey response, or new signup.
      • Post-purchase drip emails: These are sent as a follow-up to a purchase to help build trust and loyalty with a new customer.
      • Re-engagement campaigns: These are emails sent to subscribers who haven’t engaged with you for some time, with the hope to make them active again.

      When done right, these email campaigns can be very effective for your affiliate marketing strategy. After all, email remains a powerful channel of communication. In a survey by MarketingSherpa, 86% of respondents said they like receiving promotional emails at least once a month. Fifteen percent of participants even said that they enjoy daily promotional emails.

      The first step in setting up an email campaign is to sign up for a marketing tool. Mailchimp is a good option, as we discussed earlier. SendPulse is another powerful tool that gives you complete campaign automation with individual-level personalization.

      SendPulse homepageNext, you’ll want to produce amazing, share-worthy content and embed your affiliate links within it. You can also add social media buttons and an Email to a friend link to increase the recipient’s likelihood of sharing your content.

      10. Purchase Online Ads

      Purchasing online advertising may be the riskiest strategy in this post because you’ll have to put money on the line before you can see results. However, if you know what you’re doing, you may reach your target sooner than expected.

      There are different kinds of ads for different mediums. Let’s talk about two kinds: Pay-Per-Click (PPC) ads and solo ads. Both methods are intended to take your prospects to your home page, a landing page, or another place designed to convert them. However, they work a little differently.

      With PPC ads, you pay a small fee each time a user clicks on your ad to get to your target page. They are very common on the web. Here’s an example of what’s likely a PPC ad on Facebook:

      An example of a sponsored ad on Facebook

      On the other hand, solo ads are more common among affiliates and information marketers. They are generally more expensive than PPC ads, but they are ideal for affiliate marketing purposes. Moreover, when these ads do work, they can be cheaper than other forms of advertising.

      You can buy solo ads from someone who owns an email list that targets your ideal audience. Here’s how it works:

      1. First, you buy an ad from the list owner.
      2. Then, the owner sends out an email (the ad) to their subscribers about your products.
      3. Finally, subscribers click on the links and, hopefully, convert.

      Whatever type of ad you choose, you’ll want to make sure to educate yourself first to avoid monetary losses. For instance, you may want to see some statistics and data about the email list, including click-through rates and demographics. This can help you choose the right affiliate product to promote to that particular audience.

      11. Present Webinars

      A webinar is an online presentation of educational content. It can be live or delivered as recorded video. There is often a good amount of audience interaction via instant messages or two-way video if it’s live.

      Like courses and tutorials, webinars can boost your perceived trustworthiness and competence in the eyes of your audience. They are also a wonderful medium for presenting the solution to a problem in the form of the products you promote.

      Webinars are relatively easy and inexpensive to deliver, though they require you to prepare some material and research your subject. However, your audience doesn’t normally expect you to follow a strict script. In that way, these webinars can seem more informal than a normal educational setting.

       

      After you’re finished with your initial delivery, you can reuse your recorded session. It can serve as another piece of marketing content to help promote your brand or drive more traffic to your website. For example, you can provide a link to your recorded webinar in your newsletters or post it on social media.

      Webinars and email marketing reinforce one another. GoToWebinar reports that email marketing is responsible for more than half of webinar registrations. At the same time, requiring an email address when signing up for a webinar is an effective way to generate more leads.

      To get started with your own webinar, you’ll first need to choose your topic and delivery format. Then, you’ll have to select the tools and platforms you want to use. Zoom, Livestorm, and ClickMeeting are all popular options:

      ClickMeeting homepage

      You’ll need to set a date and time and promote your webinar. However, before you put the word out, it’s important to have a registration page up and running. You’ll want to provide easy signup forms and give users the option to select their preferred webinar dates.

      12. Join a Pay-Per-Call Affiliate Program

      There is one other affiliate advertising model we haven’t covered: pay-per-call. This is somewhat different from every other method we’ve discussed so far.

      In this model, the goal is for you to get people to call the company you’re promoting. You often have a unique phone number to share with prospects. That way, the company can attribute the call to you.

      Pay-per-call advertising works particularly well for companies that sell or book appointments over the phone, such as locksmiths and tow truck services. The rationale is that making a phone call shows more of a purchasing commitment than simply looking at a website.

      Consider joining a pay-per-call affiliate network if you think you can get people to make a phone call. You could also reach out to companies and ask them if they would be interested in collaborating with you.

      Time to Get Creative

      There are many effective ways to promote your affiliate marketing products that don’t require a blog. Whether you use them to complement your existing website or to forgo writing posts altogether, one or more of these methods could be your next hit.

      In this post, we looked at some clever strategies for boosting your affiliate marketing income. For instance, you could promote products in newsletters, YouTube videos, or social media posts. You could also create high-quality courses and webinars and embed affiliate links in your educational material.

      If you have a WordPress site and want to get started with affiliate marketing, you may want to consider looking at DreamPress. Our managed WordPress hosting plans handle updates and other maintenance tasks for you so you can focus on the important stuff: growing your business.

      Power Your Blog with DreamHost

      We’ll make sure your blog is fast, secure and always up so your visitors trust you. Plans start at $2.59/mo.

      shared hosting



      Source link

      How To Run Multiple Functions Concurrently in Go


      The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      One of the popular features of the Go language is its first-class support for concurrency, or the ability of a program to do multiple things at once. Being able to run code concurrently is becoming a larger part of programming as computers move from running a single stream of code faster to running more streams of code simultaneously. To run programs faster, a programmer needs to design their programs to run concurrently, so that each concurrent part of the program can be run independently of the others. Two features in Go, goroutines and channels, make concurrency easier when used together. Goroutines solve the difficulty of setting up and running concurrent code in a program, and channels solve the difficulty of safely communicating between the code running concurrently.

      In this tutorial, you will explore both goroutines and channels. First, you will create a program that uses goroutines to run multiple functions at once. Then you will add channels to that program to communicate between the running goroutines. Finally, you’ll add more goroutines to the program to simulate a program running with multiple worker goroutines.

      Prerequisites

      To follow this tutorial, you will need:

      Running Functions at the Same Time with Goroutines

      In a modern computer, the processor, or CPU, is designed to run as many streams of code as possible at the same time. These processors have one or more “cores,” each capable of running one stream of code at the same time. So, the more cores a program can use simultaneously, the faster the program will run. However, in order for programs to take advantage of the speed increase that multiple cores provide, programs need to be able to be split into multiple streams of code. Splitting a program into parts can be one of the more challenging things to do in programming, but Go was designed to make this easier.

      One way Go does this is with a feature called goroutines. A goroutine is a special type of function that can run while other goroutines are also running. When a program is designed to run multiple streams of code at once, the program is designed to run concurrently. Typically, when a function is called, it will finish running completely before the code after it continues to run. This is known as running in the “foreground” because it prevents your program from doing anything else before it finishes. With a goroutine, the function call will continue running the next code right away while the goroutine runs in the “background”. Code is considered running in the background when it doesn’t prevent other code from running before it finishes.

      The power goroutines provide is that each goroutine can run on a processor core at the same time. If your computer has four processor cores and your program has four goroutines, all four goroutines can run simultaneously. When multiple streams of code are running at the same time on different cores like this, it’s called running in parallel.

      To visualize the difference between concurrency and parallelism, consider the following diagram. When a processor runs a function, it doesn’t always run it from beginning to completion all at once. Sometimes the operating system will interleave other functions, goroutines, or other programs on a CPU core when a function is waiting for something else to happen, such as reading a file. The diagram shows how a program designed for concurrency can run on a single core as well as multiple cores. It also shows how more segments of a goroutine can fit into the same timeframe (9 vertical segments, as seen in the diagram) when running in parallel than when running on a single core.

      Diagram split into two columns, labeled Concurrency and Parallelism. The Concurrency column has a single tall rectangle, labeled CPU core, divided into stacked sections of varying colors signifying different functions. The Parallelism column has two similar tall rectangles, both labeled CPU core, with each stacked section signifying different functions, except it only shows goroutine1 running on the left core and goroutine2 running on the right core.

      The left column in the diagram, labeled “Concurrency”, shows how a program designed around concurrency could run on a single CPU core by running part of goroutine1, then another function, goroutine, or program, then goroutine2, then goroutine1 again, and so on. To a user, this would seem like the program is running all the functions or goroutines at the same time, even though they’re actually being run in small parts one after the other.

      The column on the right of the diagram, labeled “Parallelism”, shows how that same program could run in parallel on a processor with two CPU cores. The first CPU core shows goroutine1 running interspersed with other functions, goroutines, or programs, while the second CPU core shows goroutine2 running with other functions or goroutines on that core. Sometimes both goroutine1 and goroutine2 are running at the same time as each other, just on different CPU cores.

      This diagram also shows another of Go’s powerful traits, scalability. A program is scalable when it can run on anything from a small computer with a few processor cores to a large server with dozens of cores, and take advantage of those additional resources. The diagram shows that by using goroutines, your concurrent program is capable of running on a single CPU core, but as more CPU cores are added, more goroutines can be run in parallel to speed up the program.

      To get started with your new concurrent program, create a multifunc directory in the location of your choosing. You may already have a directory for your projects, but in this tutorial, you’ll create a directory called projects. You can create the projects directory either through an IDE or via the command line.

      If you’re using the command line, begin by making the projects directory and navigating to it:

      • mkdir projects
      • cd projects

      From the projects directory, use the mkdir command to create the program’s directory (multifunc) and then navigate into it:

      • mkdir multifunc
      • cd multifunc

      Once you’re in the multifunc directory, open a file named main.go using nano, or your favorite editor:

      Paste or type the following code in the main.go file to get started.

      projects/multifunc/main.go

      package main
      
      import (
          "fmt"
      )
      
      func generateNumbers(total int) {
          for idx := 1; idx <= total; idx++ {
              fmt.Printf("Generating number %dn", idx)
          }
      }
      
      func printNumbers() {
          for idx := 1; idx <= 3; idx++ {
              fmt.Printf("Printing number %dn", idx)
          }
      }
      
      func main() {
          printNumbers()
          generateNumbers(3)
      }
      

      This initial program defines two functions, generateNumbers and printNumbers, then runs those functions in the main function. The generateNumbers function takes the amount of numbers to “generate” as a parameter, in this case one through three, and then prints each of those numbers to the screen. The printNumbers function doesn’t take any parameters yet, but it will also print out the numbers one through three.

      Once you’ve saved the main.go file, run it using go run to see the output:

      The output will look similar to this:

      Output

      Printing number 1 Printing number 2 Printing number 3 Generating number 1 Generating number 2 Generating number 3

      You’ll see the functions run one after the other, with printNumbers running first and generateNumbers running second.

      Now, imagine that printNumbers and generateNumbers each takes three seconds to run. When running synchronously, or one after the other like the last example, your program would take six seconds to run. First, printNumbers would run for three seconds, and then generateNumbers would run for three seconds. In your program, however, these two functions are independent of the other because they don’t rely on data from the other to run. You can take advantage of this to speed up this hypothetical program by running the functions concurrently using goroutines. When both functions are running concurrently the program could, in theory, run in half the time. If both the printNumbers and the generateNumbers functions take three seconds to run and both start at exactly the same time, the program could finish in three seconds. (The actual speed could vary due to outside factors, though, such as how many cores the computer has or how many other programs are running on the computer at the same time.)

      Running a function concurrently as a goroutine is similar to running a function synchronously. To run a function as a goroutine (as opposed to a standard synchronous function), you only need to add the go keyword before the function call.

      However, for the program to run the goroutines concurrently, you’ll need to make one additional change. You’ll need to add a way for your program to wait until both goroutines have finished running. If you don’t wait for your goroutines to finish and your main function completes, the goroutines could potentially never run, or only part of them could run and not complete running.

      To wait for the functions to finish, you’ll use a WaitGroup from Go’s sync package. The sync package contains “synchronization primitives”, such as WaitGroup, that are designed to synchronize various parts of a program. In your case, the synchronization keeps track of when both functions have finished running so you can exit the program.

      The WaitGroup primitive works by counting how many things it needs to wait for using the Add, Done, and Wait functions. The Add function increases the count by the number provided to the function, and Done decreases the count by one. The Wait function can then be used to wait until the count reaches zero, meaning that Done has been called enough times to offset the calls to Add. Once the count reaches zero, the Wait function will return and the program will continue running.

      Next, update the code in your main.go file to run both of your functions as goroutines using the go keyword, and add a sync.WaitGroup to the program:

      projects/multifunc/main.go

      package main
      
      import (
          "fmt"
          "sync"
      )
      
      func generateNumbers(total int, wg *sync.WaitGroup) {
          defer wg.Done()
      
          for idx := 1; idx <= total; idx++ {
              fmt.Printf("Generating number %dn", idx)
          }
      }
      
      func printNumbers(wg *sync.WaitGroup) {
          defer wg.Done()
      
          for idx := 1; idx <= 3; idx++ {
              fmt.Printf("Printing number %dn", idx)
          }
      }
      
      func main() {
          var wg sync.WaitGroup
      
          wg.Add(2)
          go printNumbers(&wg)
          go generateNumbers(3, &wg)
      
          fmt.Println("Waiting for goroutines to finish...")
          wg.Wait()
          fmt.Println("Done!")
      }
      

      After declaring the WaitGroup, it will need to know how many things to wait for. Including a wg.Add(2) in the main function before starting the goroutines will tell wg to wait for two Done calls before considering the group finished. If this isn’t done before the goroutines are started, it’s possible things will happen out of order or the code may panic because the wg doesn’t know it should be waiting for any Done calls.

      Each function will then use defer to call Done to decrease the count by one after the function finishes running. The main function is also updated to include a call to Wait on the WaitGroup, so the main function will wait until both functions call Done to continue running and exit the program.

      After saving your main.go file, run it using go run like you did before:

      The output will look similar to this:

      Output

      Printing number 1 Waiting for goroutines to finish... Generating number 1 Generating number 2 Generating number 3 Printing number 2 Printing number 3 Done!

      Your output may differ from what is printed here, and it’s even likely to change every time you run the program. With both functions running concurrently, the output depends on how much time Go and your operating system give for each function to run. Sometimes there is enough time to run each function completely and you’ll see both functions print their entire sequences uninterrupted. Other times, you’ll see the text interspersed like the output above.

      An experiment you can try is removing the wg.Wait() call in the main function and running the program a few times with go run again. Depending on your computer, you may see some output from the generateNumbers and printNumbers functions, but it’s also likely you won’t see any output from them at all. When you remove the call to Wait, the program will no longer wait for both functions to finish running before it continues. Since the main function ends soon after the Wait function, there’s a good chance that your program will reach the end of the main function and exit before the goroutines finish running. When this happens, you’ll see a few numbers printed out, but you won’t see all three from each function.

      In this section, you created a program that uses the go keyword to run two goroutines concurrently and print a sequence of numbers. You also used a sync.WaitGroup to make your program wait for those goroutines to finish before exiting the program.

      You may have noticed that the generateNumbers and printNumbers functions do not have return values. In Go, goroutines aren’t able to return values like a standard function would. You can still use the go keyword to call a function that returns values, but those return values will be thrown out and you won’t be able to access them. So, what do you do when you need data from one goroutine in another goroutine if you can’t return values? The solution is to use a Go feature called “channels”, which allow you to send data from one goroutine to another.

      Communicating Safely Between Goroutines with Channels

      One of the more difficult parts of concurrent programming is communicating safely between different parts of the program that are running simultaneously. If you’re not careful, you might run into problems that are only possible with concurrent programs. For example, a data race can happen when two parts of a program are running concurrently, and one part tries to update a variable while the other part is trying to read it at the same time. When this happens, the reading or writing can happen out of order, leading to one or both parts of the program using the wrong value. The name “data race” comes from both parts of the program “racing” each other to access the data.

      While it’s still possible to run into concurrency issues like data races in Go, the language is designed to make it easier to avoid them. In addition to goroutines, channels are another feature that makes concurrency safer and easier to use. A channel can be thought of like a pipe between two or more different goroutines that data can be sent through. One goroutine puts data into one end of the pipe and another goroutine gets that same data out. The difficult part of making sure the data gets from one to the other safely is handled for you.

      Creating a channel in Go is similar to how you would create a slice, using the built-in make() function. The type declaration for a channel uses the chan keyword followed by the type of data you want to send on the channel. For example, to create a channel for sending int values, you would use the type chan int. If you wanted a channel for sending []byte vaules, it would be chan []byte, like so:

      bytesChan := make(chan []byte)
      

      Once a channel is created, you can send or receive data on the channel by using the arrow-looking <- operator. The position of the <- operator in relation to the channel variable determines whether you’re reading from or writing to the channel.

      To write to a channel, begin with the channel variable, followed by the <- operator, then the value you want to write to the channel:

      intChan := make(chan int)
      intChan <- 10
      

      To read a value from a channel, begin with the variable you want to put the value into, either = or := to assign a value to the variable, followed by the <- operator, and then the channel you want to read from:

      intChan := make(chan int)
      intVar := <- intChan
      

      To keep these two operations straight, it can be helpful to remember that the <- arrow always points to the left (as opposed to ->), and the arrow points to where the value is going. In the case of writing to a channel, the arrow points the value to the channel. When reading from a channel, the arrow points the channel to the variable.

      Like a slice, a channel can also be read using the range keyword in a for loop. When a channel is read using the range keyword, each iteration of the loop will read the next value from the channel and put it into the loop variable. It will then continue reading from the channel until the channel is closed or the for loop is exited in other ways, such as a break:

      intChan := make(chan int)
      for num := range intChan {
          // Use the value of num received from the channel
          if num < 1 {
              break
          }
      }
      

      In some cases, you may want only to allow a function to read from or write to a channel, but not both. To do this, you would add the <- operator onto the chan type declaration. Similar to reading and writing from a channel, the channel type uses the <- arrow to allow variables to constrain a channel to only reading, only writing, or both reading and writing. For example, to define a read-only channel of int values, the type declaration would be <-chan int:

      func readChannel(ch <-chan int) {
          // ch is read-only
      }
      

      If you wanted the channel to be write-only, you would declare it as chan<- int:

      func writeChannel(ch chan<- int) {
          // ch is write-only
      }
      

      Notice that the arrow is pointing out of the channel for reading, and pointing into the channel for writing. If the declaration doesn’t have an arrow, as in the case of chan int, the channel can be used for both reading and writing.

      Finally, once a channel is no longer being used it can be closed using the built-in close() function. This step is essential because when channels are created and then left unused many times in a program, it can lead to what’s known as a memory leak. A memory leak is when a program creates something that uses up memory on a computer, but doesn’t release that memory back to the computer once it’s done using it. This leads to the program slowly (or sometimes not so slowly) using up more memory over time, like a water leak. When a channel is created with make(), some of the computer’s memory is used up for the channel, then when close() is called on the channel, that memory is given back to the computer to be used for something else.

      Now, update the main.go file in your program to use a chan int channel to communicate between your goroutines. The generateNumbers function will generate numbers and write them to the channel while the printNumbers function will read those numbers from the channel and print them to the screen. In the main function, you’ll create a new channel to pass as a parameter to each of the other functions, then use close() on the channel to close it because it will no longer be used. The generateNumbers function should also not be a goroutine any more because once that function is done running, the program will have finished generating all the numbers it needs to. This way, the close() function is only called on the channel before both functions have finished running.

      projects/multifunc/main.go

      package main
      
      import (
          "fmt"
          "sync"
      )
      
      func generateNumbers(total int, ch chan<- int, wg *sync.WaitGroup) {
          defer wg.Done()
      
          for idx := 1; idx <= total; idx++ {
              fmt.Printf("sending %d to channeln", idx)
              ch <- idx
          }
      }
      
      func printNumbers(ch <-chan int, wg *sync.WaitGroup) {
          defer wg.Done()
      
          for num := range ch {
              fmt.Printf("read %d from channeln", num)
          }
      }
      
      func main() {
          var wg sync.WaitGroup
          numberChan := make(chan int)
      
          wg.Add(2)
          go printNumbers(numberChan, &wg)
      
          generateNumbers(3, numberChan, &wg)
      
          close(numberChan)
      
          fmt.Println("Waiting for goroutines to finish...")
          wg.Wait()
          fmt.Println("Done!")
      }
      

      In the parameters for generateNumbers and printNumbers, you’ll see that the chan types are using the read-only and write-only types. Since generateNumbers only needs to be able to write numbers to the channel, it’s a write-only type with the <- arrow pointing into the channel. printNumbers only needs to be able to read numbers from the channel, so it’s a read-only type with the <- arrow pointing away from the channel.

      Even though these types could be a chan int, which would allow both reading and writing, it can be helpful to constrain them to only what the function needs to avoid accidentally causing your program to stop running from something known as a deadlock. A deadlock can happen when one part of a program is waiting for another part of the program to do something, but that other part of the program is also waiting for the first part of the program to finish. Since both parts of the program are waiting on each other, the program will never continue running, almost like when two gears seize.

      The deadlock can happen due to the way channel communication works in Go. When part of a program is writing to a channel, it will wait until another part of the program reads from that channel before continuing on. Similarly, if a program is reading from a channel it will wait until another part of the program writes to that channel before it continues. A part of a program waiting on something else to happen is said to be blocking because it’s blocked from continuing until something else happens. Channels block when they are being written to or read from. So if you have a function where you’re expecting to write to a channel but accidentally read from the channel instead, your program may enter a deadlock because the channel will never be written to. Ensuring this never happens is one reason to use a chan<- int or a <-chan int instead of just a chan int.

      One other important aspect of the updated code is using close() to close the channel once it’s done being written to by generateNumbers. In this program, close() causes the for ... range loop in printNumbers to exit. Since using range to read from a channel continues until the channel it’s reading from is closed, if close isn’t called on numberChan then printNumbers will never finish. If printNumbers never finishes, the WaitGroup’s Done method will never be called by the defer when printNumbers exits. If the Done method is never called from printNumbers, the program itself will never exit because the WaitGroup’s Wait method in the main function will never continue. This is another example of a deadlock because the main function is waiting on something that will never happen.

      Now, run your updated code using the go run command on main.go again.

      Your output may vary slightly from what’s shown below, but overall it should be similar:

      Output

      sending 1 to channel sending 2 to channel read 1 from channel read 2 from channel sending 3 to channel Waiting for functions to finish... read 3 from channel Done!

      The output from the program shows that the generateNumbers function is generating the numbers one through three while writing them to the channel shared with printNumbers. Once printNumbers receives the number, it then prints it to the screen. After generateNumbers has generated all three numbers it will exit, allowing the main function to close the channel and wait until printNumbers is finished. Once printNumbers finishes printing out the last number, it calls Done on the WaitGroup and the program exits. Similar to previous outputs, the exact output you see will depend on various outside factors, such as when the operating system or Go runtime choose to run specific goroutines, but it should be relatively close.

      The benefit of designing your programs using goroutines and channels is that once you’ve designed your program to be split up, you can scale it up to more goroutines. Since generateNumbers is just writing to a channel, it doesn’t matter how many other things are reading from that channel. It will just send numbers to anything that reads the channel. You can take advantage of this by running more than one printNumbers goroutine, so each of them will read from the same channel and handle the data concurrently.

      Now that your program is using channels to communicate, open the main.go file again and update your program so it starts multiple printNumbers goroutines. You will need to tweak the call to wg.Add so it adds one for every goroutine you start. You don’t need to worry about adding one to the WaitGroup for the call to generateNumbers any more because the program won’t continue without finishing the whole function, unlike when you were running it as a goroutine. To ensure it doesn’t reduce the WaitGroup count when it finishes, you should remove the defer wg.Done() line from the function. Next, adding the number of the goroutine to printNumbers makes it easier to see how the channel is read by each of them. Increasing the amount of numbers being generated is also a good idea so that it’s easier to see the numbers being spread out:

      projects/multifunc/main.go

      ...
      
      func generateNumbers(total int, ch chan<- int, wg *sync.WaitGroup) {
          for idx := 1; idx <= total; idx++ {
              fmt.Printf("sending %d to channeln", idx)
              ch <- idx
          }
      }
      
      func printNumbers(idx int, ch <-chan int, wg *sync.WaitGroup) {
          defer wg.Done()
      
          for num := range ch {
              fmt.Printf("%d: read %d from channeln", idx, num)
          }
      }
      
      func main() {
          var wg sync.WaitGroup
          numberChan := make(chan int)
      
          for idx := 1; idx <= 3; idx++ {
              wg.Add(1)
              go printNumbers(idx, numberChan, &wg)
          }
      
          generateNumbers(5, numberChan, &wg)
      
          close(numberChan)
      
          fmt.Println("Waiting for goroutines to finish...")
          wg.Wait()
          fmt.Println("Done!")
      }
      

      Once your main.go is updated, you can run your program again using go run with main.go. Your program should start three printNumbers goroutines before continuing on to generating numbers. Your program should also now generate five numbers instead of three to make it easier to see the numbers spread out among each of the three printNumbers goroutines:

      The ouput may look similar to this (although your output might vary quite a bit):

      Output

      sending 1 to channel sending 2 to channel sending 3 to channel 3: read 2 from channel 1: read 1 from channel sending 4 to channel sending 5 to channel 3: read 4 from channel 1: read 5 from channel Waiting for goroutines to finish... 2: read 3 from channel Done!

      When you look at your program output this time, there’s a good chance it will vary greatly from the output you see above. Since there are three printNumbers goroutines running, there’s an element of chance determining which one receives a specific number. When one printNumbers goroutine receives a number, it spends a small amount of time printing that number to the screen, while another goroutine reads the next number from the channel and does the same thing. When a goroutine has finished its work of printing the number and is ready to read another number, it will go back and read the channel again to print the next one. If there are no more numbers to be read from the channel, it will start to block until the next number can be read. Once generateNumbers has finished and close() is called on the channel, all three of the printNumbers goroutines will finish their range loops and exit. When all three goroutines have exited and called Done on the WaitGroup, the WaitGroup’s count will reach zero and the program will exit. You can also experiment with increasing or decreasing the amount of goroutines or numbers being generated to see how that affects the output.

      When using goroutines, avoid starting too many. In theory, a program could have hundreds or even thousands of goroutines. However, depending on the computer the program is running on, it could actually be slower to have a higher number of goroutines. With a high number of goroutines, there’s a chance it could run into resource starvation. Every time Go runs part of a goroutine, it requires a little bit of extra time to start running again, in addition to the time needed to run the code in the next function. Due to the extra time it takes, it’s possible for the computer to take longer to switch between running each goroutine than to actually run the goroutine itself. When this happens, it’s called resource starvation because the program and its goroutines aren’t getting the resources they need to run, or are getting very few. In these cases, it may be faster to lower the number of parts in the program running concurrently because it will lower the time it takes to switch between them, and give more time to running the program itself. Remembering how many cores the program is running on can be a good starting point for deciding how many goroutines you want to use.

      Using a combination of goroutines and channels makes it possible to create very powerful programs capable of scaling from running on small desktop computers up to massive servers. As you saw in this section, channels can be used to communicate between as few as a couple of goroutines to potentially thousands of goroutines with minimal changes. If you take this into consideration when writing your programs, you’ll be able to take advantage of the concurrency available in Go to provide your users a better overall experience.

      Conclusion

      In this tutorial, you created a program using the go keyword to start concurrently-running goroutines that printed out numbers as they run. Once that program was running, you created a new channel of int values using make(chan int), then used the channel to generate numbers in one goroutine and send them to another goroutine to be printed to the screen. Finally, you started multiple “printing” goroutines at the same time as an example of how channels and goroutines can be used to speed up your programs on multi-core computers.

      If you’re interested in learning more about concurrency in Go, the Effective Go document created by the Go team goes into much more detail. The Concurrency is not parallelism Go blog post is also an interesting follow-up about the relationship between concurrency and parallelism, two terms that are sometimes mistakenly conflated to mean the same thing.

      This tutorial is also part of the DigitalOcean How to Code in Go series. The series covers a number of Go topics, from installing Go for the first time to how to use the language itself.



      Source link

      How To Build A Security Information and Event Management (SIEM) System with Suricata and the Elastic Stack on Ubuntu 20.04


      Introduction

      The previous tutorials in this series guided you through installing, configuring, and running Suricata as an Intrusion Detection (IDS) and Intrusion Prevention (IPS) system. You also learned about Suricata rules and how to create your own.

      In this tutorial you will explore how to integrate Suricata with Elasticsearch, Kibana, and Filebeat to begin creating your own Security Information and Event Management (SIEM) system. SIEM tools are used to collect, aggregate, store, and analyze event data to search for security threats and suspicious activity on your networks and servers.

      The components that you will use to build your own SIEM tool are:

      • Elasticsearch to store, index, correlate, and search the security events that come from your Suricata server.
      • Kibana to display and navigate around the security event logs that are stored in Elasticsearch.
      • Filebeat to parse Suricata’s eve.json log file and send each event to Elasticsearch for processing.
      • Suricata to scan your network traffic for suspicious events, and either log or drop invalid packets.

      First you’ll install and configure Elasticsearch and Kibana with some specific authentication settings. Then you’ll add Filebeat to your Suricata system to send its eve.json logs to Elasticsearch.

      Finally, you’ll learn how to connect to Kibana using SSH and your web browser, and then load and interact with Kibana dashboards that show Suricata’s events and alerts.

      Prerequisites

      If you have been following this tutorial series then you should already have Suricata running on an Ubuntu 20.04 server. This server will be referred to as your Suricata server.

      You will also need a second server to host Elasticsearch and Kibana. This server will be referred to as your Elasticsearch server. It should be an Ubuntu 20.04 server with:

      For the purposes of this tutorial, both servers should be able to communicate using private IP addresses. You can use a VPN like WireGuard to connect your servers, or use a cloud-provider that has private networking between hosts. You can also choose to run Elasticsearch, Kibana, Filebeat, and Suricata on the same server for experimenting.

      Step 1 — Installing Elasticsearch and Kibana

      The first step in this tutorial is to install Elasticsearch and Kibana on your Elasticsearch server. To get started, add the Elastic GPG key to your server with the following command:

      • curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      Next, add the Elastic source list to the sources.list.d directory, where apt will search for new sources:

      • echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

      Now update your server’s package index and install Elasticsearch and Kibana:

      • sudo apt update
      • sudo apt install elasticsearch kibana

      Once you are done installing the packages, find and record your server’s private IP address using the ip address show command:

      You will receive output like the following:

      Output

      lo UNKNOWN 127.0.0.1/8 ::1/128 eth0 UP 159.89.122.115/20 10.20.0.8/16 2604:a880:cad:d0::e56:8001/64 fe80::b832:69ff:fe46:7e5d/64 eth1 UP 10.137.0.5/16 fe80::b883:5bff:fe19:43f3/64

      The private network interface in this output is the highlighted eth1 device, with the IPv4 address 10.137.0.5/16. Your device name, and IP addresses will be different. However, the address will be from the following reserved blocks of addresses:

      • 10.0.0.0 to 10.255.255.255 (10/8 prefix)
      • 172.16.0.0 to 172.31.255.255 (172.16/12 prefix)
      • 192.168.0.0 to 192.168.255.255 (192.168/16 prefix)

      If you would like to learn more about how these blocks are allocated visit the RFC 1918 specification)

      Record the private IP address for your Elasticsearch server (in this case 10.137.0.5). This address will be referred to as your_private_ip in the remainder of this tutorial. Also note the name of the network interface, in this case eth1. In the next part of this tutorial you will configure Elasticsearch and Kibana to listen for connections on the private IP address coming from your Suricata server.

      Step 2 — Configuring Elasticsearch

      Elasticsearch is configured to only accept local connections by default. Additionally, it does not have any authentication enabled, so tools like Filebeat will not be able to send logs to it. In this section of the tutorial you will configure the network settings for Elasticsearch and then enable Elasticsearch’s built-in xpack security module.

      Configuring Elasticsearch Networking

      Since Your Elasticsearch and Suricata servers are separate, you will need to configure Elasticsearch to listen for connections on its private network interface. You will also need to configure your firewall rules to allow access to Elasticsearch on your private network interface.

      Open the /etc/elasticsearch/elasticsearch.yml file using nano or your preferred editor:

      • sudo nano /etc/elasticsearch/elasticsearch.yml

      Find the commented out #network.host: 192.168.0.1 line between lines 50–60 and add a new line after it that configures the network.bind_host setting, as highlighted below:

      /etc/elasticsearch/elasticsearch.yml

      # By default Elasticsearch is only accessible on localhost. Set a different
      # address here to expose this node on the network:
      #
      #network.host: 192.168.0.1
      network.bind_host: ["127.0.0.1", "your_private_ip"]
      #
      # By default Elasticsearch listens for HTTP traffic on the first free port it
      # finds starting at 9200. Set a specific HTTP port here:
      

      Substitute your private IP in place of the your_private_ip address. This line will ensure that Elasticsearch is still available on its local address so that Kibana can reach it, as well as on the private IP address for your server.

      Next, go to the end of the file using the nano shortcut CTRL+v until you reach the end.

      Add the following highlighted lines to the end of the file:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      discovery.type: single-node
      xpack.security.enabled: true
      

      The discovery.type setting allows Elasticsearch to run as a single node, as opposed to in a cluster of other Elasticsearch servers. The xpack.security.enabled setting turns on some of the security features that are included with Elasticsearch.

      Save and close the file when you are done editing it. If you are using nano, you can do so with CTRL+X, then Y and ENTER to confirm.

      Finally, add firewall rules to ensure your Elasticsearch server is reachable on its private network interface. If you followed the prerequisite tutorials and are using the Uncomplicated Firewall (ufw), run the following commands:

      • sudo ufw allow in on eth1
      • sudo ufw allow out on eth1

      Substitute your private network interface in place of eth1 if it uses a different name.

      Next you will start the Elasticsearch daemon and then configure passwords for use with the xpack security module.

      Starting Elasticsearch

      Now that you have configured networking and the xpack security settings for Elasticsearch, you need to start it for the changes to take effect.

      Run the following systemctl command to start Elasticsearch:

      • sudo systemctl start elasticsearch.service

      Once Elasticsearch finishes starting, you can continue to the next section of this tutorial where you will generate passwords for the default users that are built-in to Elasticsearch.

      Configuring Elasticsearch Passwords

      Now that you have enabled the xpack.security.enabled setting, you need to generate passwords for the default Elasticsearch users. Elasticsearch includes a utility in the /usr/share/elasticsearch/bin directory that can automatically generate random passwords for these users.

      Run the following command to cd to the directory and then generate random passwords for all the default users:

      • cd /usr/share/elasticsearch/bin
      • sudo ./elasticsearch-setup-passwords auto

      You will receive output like the following. When prompted to continue, press y and then RETURN or ENTER:

      Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
      The passwords will be randomly generated and printed to the console.
      Please confirm that you would like to continue [y/N]y
      
      
      Changed password for user apm_system
      PASSWORD apm_system = eWqzd0asAmxZ0gcJpOvn
      
      Changed password for user kibana_system
      PASSWORD kibana_system = 1HLVxfqZMd7aFQS6Uabl
      
      Changed password for user kibana
      PASSWORD kibana = 1HLVxfqZMd7aFQS6Uabl
      
      Changed password for user logstash_system
      PASSWORD logstash_system = wUjY59H91WGvGaN8uFLc
      
      Changed password for user beats_system
      PASSWORD beats_system = 2p81hIdAzWKknhzA992m
      
      Changed password for user remote_monitoring_user
      PASSWORD remote_monitoring_user = 85HF85Fl6cPslJlA8wPG
      
      Changed password for user elastic
      PASSWORD elastic = 6kNbsxQGYZ2EQJiqJpgl
      

      You will not be able to run the utility again, so make sure to record these passwords somewhere secure. You will need to use the kibana_system user’s password in the next section of this tutorial, and the elastic user’s password in the Configuring Filebeat step of this tutorial.

      At this point in the tutorial you are finished configuring Elasticsearch. The next section explains how to configure Kibana’s network settings and its xpack security module.

      Step 3 — Configuring Kibana

      In the previous section of this tutorial, you configured Elasticsearch to listen for connections on your Elasticsearch server’s private IP address. You will need to do the same for Kibana so that Filebeats on your Suricata server can reach it.

      First you’ll enable Kibana’s xpack security functionality by generating some secrets that Kibana will use to store data in Elasticsearch. Then you’ll configure Kibana’s network setting and authentication details to connect to Elasticsearch.

      Enabling xpack.security in Kibana

      To get started with xpack security settings in Kibana, you need to generate some encryption keys. Kibana uses these keys to store session data (like cookies), as well as various saved dashboards and views of data in Elasticsearch.

      You can generate the required encryption keys using the kibana-encryption-keys utility that is included in the /usr/share/kibana/bin directory. Run the following to cd to the directory and then generate the keys:

      • cd /usr/share/kibana/bin/
      • sudo ./kibana-encryption-keys generate -q

      The -q flag suppresses the tool’s instructions so that you only receive output like the following:

      Output

      xpack.encryptedSavedObjects.encryptionKey: 66fbd85ceb3cba51c0e939fb2526f585 xpack.reporting.encryptionKey: 9358f4bc7189ae0ade1b8deeec7f38ef xpack.security.encryptionKey: 8f847a594e4a813c4187fa93c884e92b

      Copy your output somewhere secure. You will now add them to Kibana’s /etc/kibana/kibana.yml configuration file.

      Open the file using nano or your preferred editor:

      • sudo nano /etc/kibana/kibana.yml

      Go to the end of the file using the nano shortcut CTRL+v until you reach the end. Paste the three xpack lines that you copied to the end of the file:

      /etc/kibana/kibana.yml

      . . .
      
      # Specifies locale to be used for all localizable strings, dates and number formats.
      # Supported languages are the following: English - en , by default , Chinese - zh-CN .
      #i18n.locale: "en"
      
      xpack.encryptedSavedObjects.encryptionKey: 66fbd85ceb3cba51c0e939fb2526f585
      xpack.reporting.encryptionKey: 9358f4bc7189ae0ade1b8deeec7f38ef
      xpack.security.encryptionKey: 8f847a594e4a813c4187fa93c884e92b
      

      Keep the file open and proceed to the next section where you will configure Kibana’s network settings.

      Configuring Kibana Networking

      To configure Kibana’s networking so that it is available on your Elasticsearch server’s private IP address, find the commented out #server.host: "localhost" line in /etc/kibana/kibana.yml. The line is near the beginning of the file. Add a new line after it with your server’s private IP address, as highlighted below:

      /etc/kibana/kibana.yml

      # Kibana is served by a back end server. This setting specifies the port to use.
      #server.port: 5601
      
      # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
      # The default is 'localhost', which usually means remote machines will not be able to connect.
      # To allow connections from remote users, set this parameter to a non-loopback address.
      #server.host: "localhost"
      server.host: "your_private_ip"
      

      Substitute your private IP in place of the your_private_ip address.

      Save and close the file when you are done editing it. If you are using nano, you can do so with CTRL+X, then Y and ENTER to confirm.

      Next, you’ll need to configure the username and password that Kibana uses to connect to Elasticsearch.

      Configuring Kibana Credentials

      There are two ways to set the username and password that Kibana uses to authenticate to Elasticsearch. The first is to edit the /etc/kibana/kibana.yml configuration file and add the values there. The second method is to store the values in Kibana’s keystore, which is an obfuscated file that Kibana can use to store secrets.

      We’ll use the keystore method in this tutorial since it avoids editing Kibana’s configuration file directly

      If you prefer to edit the file instead, the settings to configure in it are elasticsearch.username and elasticsearch.password.

      If you choose to edit the configuration file, skip the rest of the steps in this section.

      To add a secret to the keystore using the kibana-keystore utility, first cd to the /usr/share/kibana/bin directory. Next, run the following command to set the username for Kibana:

      • sudo ./kibana-keystore add elasticsearch.username

      You will receive a prompt like the following:

      Username Entry

      Enter value for elasticsearch.username: *************
      

      Enter kibana_system when prompted, either by copying and pasting, or typing the username carefully. Each character that you type will be masked with an * asterisk character. Press ENTER or RETURN when you are done entering the username.

      Now repeat the same command for the password. Be sure to copy the password for the kibana_system user that you generated in the previous section of this tutorial. For reference, in this tutorial the example password is 1HLVxfqZMd7aFQS6Uabl.

      Run the following command to set the password:

      • sudo ./kibana-keystore add elasticsearch.password

      When prompted, paste the password to avoid any transcription errors:

      Password Entry

      Enter value for elasticsearch.password: ********************
      

      Starting Kibana

      Now that you have configured networking and the xpack security settings for Kibana, as well as added credentials to the keystore, you need to start it for the changes to take effect.

      Run the following systemctl command to restart Kibana:

      • sudo systemctl start kibana.service

      Once Kibana starts, you can continue to the next section of this tutorial where you will configure Filebeat on your Suricata server to send its logs to Elasticsearch.

      Step 4 — Installing Filebeat

      Now that your Elasticsearch and Kibana processes are configured with the correct network and authentication settings, the next step is to install and set up Filebeat on your Suricata server.

      To get started installing Filebeat, add the Elastic GPG key to your Suricata server with the following command:

      • curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      Next, add the Elastic source list to the sources.list.d directory, where apt will search for new sources:

      • echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

      Now update the server’s package index and install the Filebeat package:

      • sudo apt update
      • sudo apt install filebeat

      Next you’ll need to configure Filebeat to connect to both Elasticsearch and Kibana. Open the /etc/filebeat/filebeat.yml configuration file using nano or your preferred editor:

      • sudo nano /etc/filebeat/filebeat.yml

      Find the Kibana section of the file around line 100. Add a line after the commented out #host: "localhost:5601" line that points to your Kibana instance’s private IP address and port:

      /etc/filebeat/filebeat.yml

      . . .
      # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
      # This requires a Kibana endpoint configuration.
      setup.kibana:
      
        # Kibana Host
        # Scheme and port can be left out and will be set to the default (http and 5601)
        # In case you specify and additional path, the scheme is required: http://localhost:5601/path
        # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
        #host: "localhost:5601"
        host: "your_private_ip:5601"
      
      . . .
      

      This change will ensure that Filebeat can connect to Kibana in order to create the various SIEM indices, dashboards, and processing pipelines in Elasticsearch to handle your Suricata logs.

      Next, find the Elasticsearch Output section of the file around line 130 and edit the hosts, username, and password settings to match the values for your Elasticsearch server:

      output.elasticsearch:
        # Array of hosts to connect to.
        hosts: ["your_private_ip:9200"]
      
        # Protocol - either `http` (default) or `https`.
        #protocol: "https"
      
        # Authentication credentials - either API key or username/password.
        #api_key: "id:api_key"
        username: "elastic"
        password: "6kNbsxQGYZ2EQJiqJpgl"
      
      . . .
      

      Substitute in your Elasticsearch server’s private IP address on the hosts line in place of the your_private_ip value. Uncomment the username field and leave it set to the elastic user. Change the password field from changeme to the password for the elastic user that you generated in the Configuring Elasticsearch Passwords section of this tutorial.

      Save and close the file when you are done editing it. If you are using nano, you can do so with CTRL+X, then Y and ENTER to confirm.

      Next, enable Filebeats’ built-in Suricata module with the following command:

      • sudo filebeat modules enable suricata

      Now that Filebeat is configured to connect to Elasticsearch and Kibana, with the Suricata module enabled, the next step is to load the SIEM dashboards and pipelines into Elasticsearch.

      Run the filebeat setup command. It may take a few minutes to load everything:

      Once the command finishes you should receive output like the following:

      Output

      Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling. Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead. See more: https://www.elastic.co/guide/en/machine-learning/current/index.html It is not possible to load ML jobs into an Elasticsearch 8.0.0 or newer using the Beat. Loaded machine learning job configurations Loaded Ingest pipelines

      If there are no errors, use the systemctl command to start Filebeat. It will begin sending events from Suricata’s eve.json log to Elasticsearch once it is running.

      • sudo systemctl start filebeat.service

      Now that you have Filebeat, Kibana, and Elasticsearch configured to process your Suricata logs, the last step in this tutorial is to connect to Kibana and explore the SIEM dashboards.

      Step 5 — Navigating Kibana’s SIEM Dashboards

      Kibana is the graphical component of the Elastic stack. You will use Kibana with your browser to explore Suricata’s event and alert data. Since you configured Kibana to only be available via your Elasticsearch server’s private IP address, you will need to use an SSH tunnel to connect to Kibana.

      Connecting to Kibana with SSH

      SSH has an option -L that lets you forward network traffic on a local port over its connection to a remote IP address and port on a server. You will use this option to forward traffic from your browser to your Kibana instance.

      On Linux, macOS, and updated versions of Windows 10 and higher, you can use the built-in SSH client to create the tunnel. You will use this command each time you want to connect to Kibana. You can close this connection at any time and then run the SSH command again to re-establish the tunnel.

      Run the following command in a terminal on your local desktop or laptop computer to create the SSH tunnel to Kibana:

      • ssh -L 5601:your_private_ip:5601 sammy@203.0.113.5 -N

      The various arguments to SSH are:

      • The -L flag forwards traffic to your local system on port 5601 to the remote server.
      • The your_private_ip:5601 portion of the command specifies the service on your Elasticsearch server where your traffic will be fowarded to. In this case that service is Kibana. Be sure to substitute your Elasticsearch server’s private IP address in place of your_private_ip
      • The 203.11.0.5 address is the public IP address that you use to connect to and administer your server. Substitute your Elasticsearch server’s public IP address in its place.
      • The -N flag instructs SSH to not run a command like an interactive /bin/bash shell, and instead just hold the connection open. It is generally used when forwarding ports like in this example.

      If you would like to close the tunnel at any time, press CTRL+C.

      On Windows your terminal should resemble the following screenshot:

      Note: You may be prompted to enter a password if you are not using an SSH key. Type or paste it into the prompt and press ENTER or RETURN.

      Screenshot of Windows Command Prompt Showing SSH Command to Port Forward to Kibana

      On macOS and Linux your terminal will be similar to the following screenshot:

      Screenshot of Windows Command Prompt Showing SSH Command to Port Forward to Kibana

      Once you have connected to your Elasticsearch server over SSH with the port forward in place, open your browser and visit http://127.0.0.1:5601. You will be redirected to Kibana’s login page:

      Screenshot of a Browser on Kibana's Login Page

      If your browser cannot connect to Kibana you will receive a message like the following in your terminal:

      Output

      channel 3: open failed: connect failed: No route to host

      This error indicates that your SSH tunnel is unable to reach the Kibana service on your server. Ensure that you have specified the correct private IP address for your Elasticsearch server and reload the page in your browser.

      Log in to your Kibana server using elastic for the Username, and the password that you copied earlier in this tutorial for the user.

      Browsing Kibana SIEM Dashboards

      Once you are logged into Kibana you can explore the Suricata dashboards that Filebeat configured for you.

      In the search field at the top of the Kibana Welcome page, input the search terms type:dashboard suricata. This search will return two results: the Suricata Events and Suricata Alerts dashboards per the following screenshot:

      Screenshot of a Browser Using Kibana's Global Search Box to Locate Suricata Dashboards

      Click the [Filebeat Suricata] Events Overview result to visit the Kibana dashboard that shows an overview of all logged Suricata events:

      Screenshot of a Browser on Kibana's Suricata Events Dashboard

      To visit the Suricata Alerts dashboard, repeat the search or click the Alerts link that is included in the Events dashboard. Your page should resemble the following screenshot:

      Screenshot of a Browser on Kibana's Suricata Alerts Dashboard

      If you would like to inspect the events and alerts that each dashboard displays, scroll to the bottom of the page where you will find a table that lists each event and alert. You can expand each entry to view the original log entry from Suricata, and examine in detail the various fields like source and destination IPs for an alert, the attack type, Suricata signature ID, and others.

      Kibana also has a built-in set of Security dashboards that you can access using the menu on the left side of the browser window. Navigate to the Network dashboard for an overview of events displayed on a map, as well as aggregate data about events on your network. Your dashboard should resemble the following screenshot:

      Screenshot of a Browser on Kibana's Security -> Network Dashboard

      You can scroll to the bottom of the Network dashboard for a table that lists all of the events that match your specified search timeframe. You can also examine each event in detail, or select an event to generate a Kibana timeline, that you can then use to investigate specific traffic flows, alerts, or community IDs.

      Conclusion

      In this tutorial you installed and configured Elasticsearch and Kibana on a standalone server. You configured both tools to be available on a private IP address. You also configured Elasticsearch and Kibana’s authentication settings using the xpack security module that is included with each tool.

      After completing the Elasticsearch and Kibana configuration steps, you also installed and configured Filebeat on your Suricata server. You used Filebeat to populate Kibana’s dashboards and start sending Suricata logs to Elasticsearch.

      Finally, you created an SSH tunnel to your Elasticsearch server and logged into Kibana. You located the new Suricata Events and Alerts dashboards, as well as the Network dashboard.

      The last tutorial in this series will guide you through using Kibana’s SIEM functionality to process your Suricata alerts. In it you will explore how to create cases to track specific alerts, timelines to correlate network flows, and rules to match specific Suricata events that you would like to track or analyze in more detail.



      Source link