One place for hosting & domains

      Static

      How To Use Static Files in Gatsby


      The author selected the Internet Archive to receive a donation as part of the Write for DOnations program.

      Introduction

      Like many popular Static Site Generators, Gatsby embraces the use of dynamic web frameworks, using React on the frontend and Node.js on the backend. But Gatsby can also pull in static files and assets, like images, CSS files, and JavaScript files.

      This tutorial covers the situations in which you might want to use static files with your Gatsby site. It will show you how to best go about adding images, stylesheets globally and as modules, JavaScript files, and arbitrary files like PDFs for your users to download.

      Prerequisites

      Before starting, here are a few things you will need:

      • A local installation of Node.js for running Gatsby and building your site. The installation procedure varies by operating system, but DigitalOcean has guides for Ubuntu 20.04 and macOS, and you can always find the latest release on the official Node.js download page.
      • A new Gatsby project, scaffolded from gatsby-starter-default. For satisfying this requirement and building a new Gatsby project from scratch, you can refer to Step 1 of the How To Set Up Your First Gatsby Website tutorial.
      • Some familiarity with React and JSX, as well as with HTML elements, if you want to customize the user interface (UI) of your posts beyond what is covered in this tutorial.
      • A program to unzip a zip archive file. On most operating systems, unzip is the command of choice, which you can download on Linux with your local package manager.
      • Access to the demo files repository used to provide sample files for this tutorial. You can access it at the DigitalOcean Community GitHub repository, and Step 1 will instruct you on how to download it.

      This tutorial was tested on Node.js v14.16.1, npm v6.14.12, Gatsby v3.13.1, and flexboxgrid v6.3.1.

      Step 1 — Preparing Example Files

      For the purposes of this tutorial, you will be working with a pre-arranged collection of static assets, which will be used throughout the following steps. The collection of files is available as a GitHub repository, and the first step of this tutorial is to download them and place them within your Gatsby project.

      First, you will extract the sample files to src/sample-assets, which you can either do manually by downloading the zip file and using an unzipping tool of your choice, or by running the following commands in your terminal at the root of your Gatsby project:

      • wget -O ../sample-assets.zip https://github.com/do-community/gatsby-static-files-tutorial-assets/archive/refs/heads/main.zip
      • unzip ../sample-assets.zip -d ./src

      The above command downloads an archive of the entire repo as a single zip archive file with wget, and then unzips the contents to the source directory.

      Once the files are unzipped, the next step is to create an empty Gatsby page component that will serve as the demo page for this tutorial. Create an empty file at src/pages/static-files-demo.js, then open the file in your editor of choice and add the following code:

      src/pages/static-files-demo.js

      import * as React from "react"
      
      import Layout from "../components/layout"
      import Seo from "../components/seo"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1>Static Files Demo</h1>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      This code serves as a Gatsby page component file that you will use to generate a public page at https://localhost/static-files-demo/. The StaticFilesDemo function is a React component that returns JSX, which becomes the page content. You use export default StaticFilesDemo as the final line, since Gatsby’s build system expects the default export of page components to be the React component responsible for rendering the page.

      After adding the page code, save the file, but keep it open as the following steps will add to it.

      In this first step you downloaded the static asset files that will be used throughout the tutorial and set up a demo page to build inside of. In the next step, you will add one of the most common forms of static assets: image files.

      Step 2 — Adding Images

      A common need for websites is to embed image files in a way that doesn’t impact the loading experience of the site. In this step, you will find out how to do this with Gatsby, using gatsby-plugin-image as well as some HTML to embed images into your Gatsby pages, while also optimizing for load time and bandwidth usage.

      Since gatsby-plugin-image is included in the gatsby-starter-default prerequisite, it is already installed as a dependency and ready for use. If you did not start your project from the gatsby-starter-default template, you can learn about installing and configuring gatsby-plugin-image in the official Gatsby docs.

      Open up the demo page component file that you created in the previous step and add the following highlighted code:

      src/pages/static-files-demo.js

      import * as React from "react"
      import { StaticImage } from "gatsby-plugin-image"
      
      import Layout from "../components/layout"
      import Seo from "../components/seo"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1>Static Files Demo</h1>
      
          <section className="demo">
            <h2>Static Image Files Demo</h2>
      
            <figure className="image-demo">
              <StaticImage
                src="https://www.digitalocean.com/community/tutorials/gatsby-static-files-tutorial-assets-main/images/water.jpg"
                width={1000}
                quality={90}
                alt="Underwater view of clear, blue body of water"
              />
              <figcaption>
                Photo by{" "}
                <a target="_blank" rel="noreferrer noopener" href="https://unsplash.com/@cristianpalmer">
                  Cristian Palmer
                </a>
              </figcaption>
            </figure>
      
            <figure className="image-demo">
              <StaticImage
                src="https://www.digitalocean.com/community/tutorials/gatsby-static-files-tutorial-assets-main/images/turtle.jpg"
                width={1000}
                quality={90}
                alt="Overhead view of a turtle floating over blue water"
              />
              <figcaption>
                Photo by{" "}
                <a target="_blank" rel="noreferrer noopener" href="https://unsplash.com/@ruizra">
                  Randall Ruiz
                </a>
              </figcaption>
            </figure>
          </section>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      Instead of using the standard HTML img tag directly and pointing it to a public image URL, in this code you are using the StaticImage component from gatsby-plugin-image and passing in the path to your local static images. This is the best practice approach, as StaticImage will generate multiple resized versions of your source images (using gatsby-plugin-sharp under the hood) and deliver the closest match to the visitor of your webpage (using the srcset feature), resulting in a faster page load and smaller download size.

      For the images passed to StaticImage, you used a quality of 90 to override the default value of 50, showing how gatsby-plugin-image can still offer improvements in filesize while preserving quality. You also specified a width of 1000, which serves as a cap on the maximum width, since both of the source images have an original width that far exceeds it. For the purposes of the demo and many web pages, 1000 pixels in width is more than enough. These two options have a substantial impact on performance, but there are many other options for gatsby-plugin-image that are worth exploring, which you can find in the Gatsby docs.

      In this page, the original versions of the two images take up roughly 4 MB combined, no matter what size screen they are viewed on. But with StaticImage, and on a small mobile phone, they will be compressed down to only 100kB, or roughly 2.5% of the original size. The loading time drops from almost 2 minutes on a 3G internet connection down to seconds. The gatsby-plugin-image plugin also takes advantage of modern image formats that are better suited to compressed web delivery, such as webp and avif.

      Save the file before moving on.

      Note: If you ever need to bypass gatsby-plugin-image and load images completely as-is, Gatsby offers a way to do this via the static folder, but this is generally advised against since this would not include the image optimizations mentioned earlier.

      You have now added several new images to your site, using best practices within Gatsby to provide an optimal user experience to visitors of your web pages. In the next step, you will focus on the styling component of web pages by adding static CSS files to your site.

      Step 3 — Adding CSS Stylesheets

      As with embedding images, there is more than one way to add CSS-based styling to a Gatsby site. Although inline styling and CSS-in-JS are always an option with Gatsby, for site-wide or component styling it is often a better practice to use dedicated static stylesheets. In this step, you will add your own custom CSS file to your Gatsby project, as well as a third-party CSS file, using an approach that follows Gatsby best practices.

      Start by creating a CSS file in the same folder as your demo page, at src/pages/static-files-demo.module.css. You are using the .module.css suffix to mark that this file is meant to be used as a CSS Module, which ensures that the styling will end up scoped to the component it is imported into and not other areas of the user interface (UI).

      After opening the newly created file, add the following code:

      src/pages/static-files-demo.module.css

      .headerText {
        width: 100%;
        text-align: center;
      }
      .container img {
        border-radius: 8px;
      }
      

      In this CSS, you are center-aligning the text in any element with the class of .headerText and making it full width, as well as giving a rounded edge to any img element inside an element with a .container class.

      Save the CSS file and close it. Now open back up the demo page component and add the following highlighted code:

      src/pages/static-files-demo.js

      import * as React from "react"
      import { StaticImage } from "gatsby-plugin-image"
      
      import Layout from "../components/layout"
      import Seo from "../components/seo"
      import * as DemoStyles from "./static-files-demo.module.css"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1 className={DemoStyles.headerText}>Static Files Demo</h1>
      
          <section className={'demo ' + DemoStyles.container}>
            <h2>Static Image Files Demo</h2>
      
            <figure className="image-demo">
              ...
            </figure>
            <figure className="image-demo">
              ...
            </figure>
          </section>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      The first change you made in this file is to add an import statement that imports the CSS file you just created. Instead of using a CSS @import statement, your are using the standard ES Module import statement, assigning the value of this import to the DemoStyles variable. Under the hood, Gatsby processes this CSS file with webpack, treating it as a CSS Module.

      You also updated the JSX in the component to use the classes from the CSS module file. You did this by updating the className attributes in strategic locations with the precise scoped class names from the DemoStyles import.

      Save your file.

      The next step involves the opposite scenario: adding static CSS code that you want to affect the entire site. An example of the kind of CSS you might want to load in this way is a small set of utility classes that help when designing flexbox-based layouts: flexboxgrid.

      Install this third-party CSS as a dependency by running this command in the root of your Gatsby project:

      Next, instead of importing the CSS that this library provides within the same demo page as before, you will import it at the highest level of your project so that it gets applied globally. In the starter template, this level is in the layout.js file, as every other component wraps its content inside it.

      Note: Another option for global CSS imports is in the gatsby-browser.js file, but Gatsby does not recommend this as the primary approach in most situations, as mentioned in the Gatsby docs about global styling. You can also use traditional <link> elements to import internal or external stylesheets, but that is also not recommended as it bypasses webpack.

      Open src/components/layout.js, and make the following change:

      src/components/layout.js

      /**
       * Layout component that queries for data
       * with Gatsby's useStaticQuery component
       *
       * See: https://www.gatsbyjs.com/docs/use-static-query/
       */
      
      import * as React from "react"
      import PropTypes from "prop-types"
      import { useStaticQuery, graphql } from "gatsby"
      
      import Header from "./header"
      import "./layout.css"
      import "../../node_modules/flexboxgrid/dist/flexboxgrid.min.css"
      
      const Layout = ({ children }) => {
        const data = useStaticQuery(graphql`
          query SiteTitleQuery {
            site {
              siteMetadata {
                title
              }
            }
          }
        `)
      
        return (
          <>
            ...
          </>
        )
      }
      
      Layout.propTypes = {
        children: PropTypes.node.isRequired,
      }
      
      export default Layout
      

      You have just added an import statement that directly imports the CSS file from the flexboxgrid library under node_modules. Since the CSS file is meant to be applied to the entire site, you are not assigning it to a variable, and because you don’t want to use it as a module, the filename does not end in .module.css.

      Save and close layout.js to complete the process of globally importing the CSS across your Gatsby site.

      With the CSS globally imported, you will now use the classes from flexboxgrid in your demo page, without having to import the CSS file again. Open the demo page component file and update the code:

      src/pages/static-files-demo.js

      import * as React from "react"
      import { StaticImage } from "gatsby-plugin-image"
      
      import Layout from "../components/layout"
      import Seo from "../components/seo"
      import * as DemoStyles from "./static-files-demo.module.css"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1 className={DemoStyles.headerText}>Static Files Demo</h1>
      
          <section className={'demo row around-xs ' + DemoStyles.container}>
            <h2 className="col-xs-12">Static Image Files Demo</h2>
      
            <figure className="image-demo col-xs-10 col-sm-5">
              ...
            </figure>
      
            <figure className="image-demo col-xs-10 col-sm-5">
              ...
            </figure>
          </section>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      You have just added some classes to your demo page that use rulesets from the flexboxgrid CSS file. The row and around-xs class on the section element turn it into a wrapped flex element with justify-content set to space-around, and the col-* classes are used to control how much of the row each element takes up.

      Make sure to save your changes to the demo file before closing it. With this change, the page has become more responsive and the images will appear side-by-side on a large enough display.

      To preview your changes so far, run this command:

      This will start a local development server for your Gatsby site at http://localhost:8000/static-files-demo. Navigate to this URL and you will find your site rendered with your new styling:

      Screenshot showing that the demo images are now side-by-side, in a row, and with a space between them

      Note: Since Gatsby ships with React out of the box, in addition to the static CSS options outlined here, you also have the option of applying styling through React and React-based frameworks.

      In this step, you used static CSS files to add additional styling to your site. In the next step, you will use a similar approach to add static JavaScript files for added functionality.

      Step 4 — Adding JavaScript Files

      Gatsby already uses JavaScript in both the backend and frontend, but this is used as either Node.js code that only runs during the build process or React components for generating the static HTML output. In this step, you will include JavaScript files that are neither Node.js- or React-based, but still get pulled into every page generated by Gatsby.

      For this tutorial, you are adding a file that prints a message to any visitor to your site that opens their developer console. You can inspect the JavaScript that will run by opening the file at src/gatsby-static-files-tutorial-assets-main/js/every-page.js.

      Rather than importing this file directly into a Gatsby file via an ES Module import (as you did with the CSS files), you will add it to the page via a Gatsby server-side rendering (SSR) API. This approach gives you fine-grained control over where in the DOM the JavaScript file is pulled in, and also prevents your code from being executed as part of Gatsby’s build process.

      However, before using the Gatsby SSR API, you need to make the static JavaScript file accessible to the public. To do this, use the special static folder that Gatsby supports by making a folder named static in the root of your Gatsby project. Then, copy the static JavaScript file to this folder. You can do both of these actions manually in your file browser, or with the following commands ran from the root of your project:

      • mkdir static
      • cp src/gatsby-static-files-tutorial-assets-main/js/every-page.js static/every-page.js

      With this action, the JavaScript file is now publicly accessible at https://localhost:8000/every-page.js. The next part of this step is to trigger it to load via HTML.

      Open up gatsby-ssr.js in the root of your Gatsby project, as that is where Gatsby will allow you to hook into the server-rendering APIs. Now add the following code:

      gatsby-ssr.js

      /**
       * Implement Gatsby's SSR (Server Side Rendering) APIs in this file.
       *
       * See: https://www.gatsbyjs.com/docs/ssr-apis/
       */
      
      import * as React from "react"
      
      export const onRenderBody = ({ setPostBodyComponents }) => {
        setPostBodyComponents([
          <script type="text/javascript" src="https://www.digitalocean.com/every-page.js" key="every-page-js" defer></script>,
        ])
      }
      

      The first line of code you added is a statement that imports React, which is necessary to enable the use of JSX in the file. Next, you export a function called onRenderBody, which takes an object as an argument with a nested function of setPostBodyComponents, which you call from within your own function. You call this function with a script tag that will load your static JS file, while using the best practices of adding a unique key property, and defer, since in this case it does not matter when the JavaScript executes.

      The setPostBodyComponents will take any React components inside the array passed as its first argument, in this case a single script tag, and render it as part of the body, thereby triggering the loading of your script file in the browser. Save the file, but keep it open for the next step.

      Now navigate to your https://localhost:8000/static-files-demo URL and open up a JavaScript console. You will find the message created by the JavaScript file, as shown in the following image:

      A browser with the demo page loaded, with the console message from `every-page.js` showing up in the JavaScript console.

      Note: If you are using the live development feature of Gatsby, you might need to halt and restart npm run develop before changes to this file take effect.

      You have now added a local static JavaScript file using the Gatsby SSR API, but the same strategy can also be used for loading external JavaScript from other domains, also known as 3rd-party scripts. To make the images in your demo easy to zoom in on, you will add a third-party lightbox library called Fancybox. In the same gatsby-ssr.js file, add the following lines:

      gatsby-ssr.js

      /**
       * Implement Gatsby's SSR (Server Side Rendering) APIs in this file.
       *
       * See: https://www.gatsbyjs.com/docs/ssr-apis/
       */
      
      import * as React from "react"
      
      export const onRenderBody = ({ setPostBodyComponents }) => {
        setPostBodyComponents([
          <script type="text/javascript" src="https://www.digitalocean.com/every-page.js" key="every-page-js" defer></script>,
          <script
            src="https://cdn.jsdelivr.net/npm/@fancyapps/[email protected]/dist/fancybox.umd.js"
            integrity="sha256-B34QrPZs5i0CQ3eqywkXHKIWw8msfAVH30RWj/i+dMo="
            crossOrigin="anonymous"
            key="fancybox-js"
            defer
          ></script>,
          <link
            rel="stylesheet"
            href="https://cdn.jsdelivr.net/npm/@fancyapps/[email protected]/dist/fancybox.css"
            integrity="sha256-WIieo0WFPkV7kcA2lQ4ZCO5gTg1Bs/SBX5YzEB4JkyM="
            crossOrigin="anonymous"
            key="fancybox-css"
          ></link>,
        ])
      }
      

      In this code, you have added both the JavaScript and CSS for Fancybox, the third-party library, through the same onRenderBody Gatsby SSR API as you are using for the local every-page.js file. Two extra attributes are used this time, both of which help with security. The crossOrigin="anonymous" explicitly tells the browser not to share credentials with a domain that does not match your own, and the integrity attribute is used to enforce Subresource Integrity (SRI), which protects against sudden changes to a file after it has been added.

      Warning: As a rule of thumb, treat third-party scripts and styles as untrusted. In addition to inspecting them before use, always use SRI. In general, loading third-party assets via URL instead of bundling with imports is something that should be avoided when possible, but is sometimes necessary for analytics, embed widgets, and error-logging services.

      This completes the task of getting the third-party code to load, but for this specific library, there is another step to trigger the new UI features.

      Save and close gatsby-ssr.js, then open src/pages/static-files-demo.js back up and make the following edits:

      src/pages/static-files-demo.js

      import * as React from "react"
      import { StaticImage } from "gatsby-plugin-image"
      
      import Layout from "../components/layout"
      import Seo from "../components/seo"
      import * as DemoStyles from "./static-files-demo.module.css"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1 className={DemoStyles.headerText}>Static Files Demo</h1>
      
          <section className={"demo row around-xs " + DemoStyles.container}>
            <h2 className="col-xs-12">Static Image Files Demo</h2>
      
            <figure className="image-demo col-xs-10 col-sm-5">
              <StaticImage
                data-fancybox
                src="https://www.digitalocean.com/community/tutorials/gatsby-static-files-tutorial-assets-main/images/water.jpg"
                ...
              />
              <figcaption>
                ...
            </figure>
      
            <figure className="image-demo col-xs-10 col-sm-5">
              <StaticImage
                data-fancybox
                src="https://www.digitalocean.com/community/tutorials/gatsby-static-files-tutorial-assets-main/images/turtle.jpg"
                ...
      
              />
              <figcaption>
                ...
              </figcaption>
            </figure>
          </section>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      Adding data-fancybox tells the fancybox library where to find images to trigger the lightbox effect on, and with that, your users can start using the lightbox viewer by clicking on a demo image, as shown in the following GIF:

      Screen recording showing that clicking a single image on the demo page launches a full-page lightbox viewer, complete with controls, which is closed after switching between the images

      Note: For loading scripts or styles in the <head> of the page, both your own and third-party, the recommended approach is with the gatsby-plugin-react-helmet plugin. This plugin is bundled with the starter template, and you can add to an existing usage of it in src/components/seo.js.

      You just used two different ways to pull local and remote JavaScript into your Gatsby site, each with its own use cases and trade-offs. For file types not covered by this step or previous ones, the next section in this tutorial will address how to include arbitrary files in your Gatsby site.

      Step 5 — Adding Arbitrary Static Files

      You have now implemented three common types of static assets in web development: images, CSS, and JavaScript. But that still leaves lots of other file types that might be part of a website. In this step, you will explore adding arbitrary static files to your Gatsby site, so that they can be embedded or offered as downloads to your visitors.

      The first way to add arbitrary static files so that users can access them is to import them inside of JavaScript files and use webpack to generate a public link for you. This strategy is recommended by the official Gatsby docs on importing assets, since webpack will help prevent typos in paths, avoid unnecessary disk space usage for files that are never imported, and in some cases, even inline the contents of the file as a data URI.

      Open the demo page component file and add the following edit:

      src/pages/static-files-demo.js

      import * as React from "react"
      import { StaticImage } from "gatsby-plugin-image"
      
      import Layout from "../components/layout"
      import Seo from "../components/seo"
      import * as DemoStyles from "./static-files-demo.module.css"
      import helloWorldPdf from "../gatsby-static-files-tutorial-assets-main/hello-world.pdf"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1 className={DemoStyles.headerText}>Static Files Demo</h1>
      
          <section>
            <h2>Arbitrary Static Files</h2>
            <a href={helloWorldPdf} title="Download the Hello World file">
              Access the Hello World file by clicking here.
            </a>
          </section>
      
          <section className={"demo row around-xs " + DemoStyles.container}>
            <h2 className="col-xs-12">Static Image Files Demo</h2>
            ...
          </section>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      In this code, you are importing the static file (a PDF file) directly in JavaScript, which will get parsed by webpack and will generate a unique link. That link will populate as the value of helloWorldPdf, which is what your a tag is pointing to.

      Due to the way assets are handled with this method, the final link will look rather random, like /static/hello-world-2f669160afa9b953cbe496f2d6ccb046.pdf. This works for most scenarios, but if you need a permanent readable link, Gatsby offers another option in the form of the special static folder. Pretending that you are an employer that wants your employees to be able to bookmark the link of your_domain.com/time-off-form.pdf, you will add a new file using this static folder.

      First, copy the static file time-off-form.pdf from the demo files directory to the root static folder. You can do this manually, or with the following command:

      • cp src/gatsby-static-files-tutorial-assets-main/time-off-form.pdf static/time-off-form.pdf

      Next, add a link to it in the page file:

      src/pages/static-files-demo.js

      import * as React from "react"
      
      ...
      
      import helloWorldPdf from "../gatsby-static-files-tutorial-assets-main/hello-world.pdf"
      
      const StaticFilesDemo = () => (
        <Layout>
          <Seo title="Static Files Demo" />
          <h1 className={DemoStyles.headerText}>Static Files Demo</h1>
      
          <section>
            <h2>Arbitrary Static Files</h2>
            <a href={helloWorldPdf} title="Download the Hello World file">
              Access the Hello World file by clicking here.
            </a>
            <br />
            <a href="https://www.digitalocean.com/time-off-form.pdf" title="Time Off Form">
              Request Time Off - Form to fill out and submit.
            </a>
          </section>
      
          <section className={"demo row around-xs " + DemoStyles.container}>
            ...
          </section>
        </Layout>
      )
      
      export default StaticFilesDemo
      

      Save the changes to this file and close it.

      With the static folder approach, you get to provide your users with a permanent path of /time-off-form.pdf, but you lose the benefits of cache-busting for file changes and having webpack’s compilation step catching typos in filepaths.

      Navigate to https://localhost:8000/time-off-form.pdf to view the following PDF:

      A sample PDF that has instructions for employees to log their time off.

      Thanks to your efforts in this step, visitors to your site can now access extra static file types enabled by your new code. This is convenient for your site visitors, as they don’t have to leave your domain to access these files, and beneficial for you, as it makes them less likely to leave the site entirely.

      Note: If you are using Markdown as a source within Gatsby and want to automatically copy files that are linked inside Markdown links to the public part of your site so that users can download them, please take a look at the gatsby-remark-copy-linked-files plugin.

      Conclusion

      Through following the steps in this tutorial, you have added new types of static files and assets to your website, integrating them into the Gatsby system and your existing content. These approaches can be applied to almost any new or existing Gatsby site and across hundreds of use-cases and file types, so you are now well-equipped to handle static files in Gatsby for the future.

      Static files are involved with many parts of web development, and impact or are impacted by related design decisions. Some topics that are worth further consideration are:

      If you would like to read more on Gatsby, check out the rest of the How To Create Static Web Sites with Gatsby.js series.



      Source link

      How To Speed Up Static Web Pages with Varnish Cache Server on Ubuntu 20.04


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Varnish is a versatile reverse HTTP proxy that caches responses from backend servers in memory so they are served quickly when requested again. It uses HTTP headers to determine whether to cache the responses to a particular request. By default, it does not cache responses with cookies because those are considered client-specific requests; however you can change this setting in the configuration file.

      Besides acting as a caching server, Varnish can be used as a:

      • Web application firewall
      • DDoS attack defender
      • Load balancer
      • Quick fix for unstable backends
      • HTTP router

      There are three locations where the HTTP cache can be saved:

      • Browser: This cache is saved on users’ browsers. It is user-specific and can be used to serve content instead of sending requests to web sites.
      • Proxy: A proxy is an intermediate server that sits between users and servers. It is usually deployed by ISPs and can be used to cache responses that will be requested by multiple users.
      • Reverse Proxy: This kind of proxy is created by the web site’s administrator and can be used to serve content from the network’s edge instead of sending requests to back end servers. This is the kind of cache you will create in this tutorial.

      Note: For more information about HTTP caching, see this tutorial on HTTP headers and caching strategies.

      In this tutorial, you will set up Varnish as a caching reverse proxy server. You’ll then test the setup with Varnish against a non-caching configuration using wrk.

      Prerequisites

      To complete this tutorial, you will need:

      Step 1 — Installing Varnish And Apache

      To start, you’ll install Apache and Varnish. First update apt-get, and then install Apache with these commands:

      • sudo apt-get update
      • sudo apt-get install apache2 -y

      You’ll see output indicating that Apache is being installed.

      After the Apache installation process is complete, install Varnish with this command:

      • sudo apt-get install varnish -y

      You’ll see output indicating that Varnish is being installed.

      Next, make sure both packages installed correctly. First, use this command to check the status of Apache:

      • sudo systemctl status apache2

      The output will look similar to this:

      Output

      root@ubuntu-s-1vcpu-2gb-fra1-01:~# sudo systemctl status apache2 ● apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-08-04 18:58:39 UTC; 4min 10s ago Docs: https://httpd.apache.org/docs/2.4/ Main PID: 2279 (apache2) Tasks: 55 (limit: 2344) Memory: 5.0M CGroup: /system.slice/apache2.service ├─2279 /usr/sbin/apache2 -k start ├─2281 /usr/sbin/apache2 -k start └─2282 /usr/sbin/apache2 -k start Aug 04 18:58:39 ubuntu-s-1vcpu-2gb-fra1-01 systemd[1]: Starting The Apache HTTP Server... Aug 04 18:58:39 ubuntu-s-1vcpu-2gb-fra1-01 apachectl[2278]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' di> Aug 04 18:58:39 ubuntu-s-1vcpu-2gb-fra1-01 systemd[1]: Started The Apache HTTP Server.

      Press the Q key to exit the status command.

      Next, check the status of Varnish with this command:

      • sudo systemctl status varnish

      The output will look similar to this:

      Output

      root@ubuntu-s-1vcpu-2gb-fra1-01:~# sudo systemctl status varnish ● varnish.service - Varnish HTTP accelerator Loaded: loaded (/lib/systemd/system/varnish.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-08-04 18:59:09 UTC; 4min 41s ago Docs: https://www.varnish-cache.org/docs/6.1/ man:varnishd Main PID: 3423 (varnishd) Tasks: 217 (limit: 2344) Memory: 10.7M CGroup: /system.slice/varnish.service ├─3423 /usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m └─3447 /usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m Aug 04 18:59:09 ubuntu-s-1vcpu-2gb-fra1-01 systemd[1]: Started Varnish HTTP accelerator. Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Debug: Version: varnish-6.2.1 revision 9f8588e4ab785244e06c3446fe09bf9db5dd8753 Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Version: varnish-6.2.1 revision 9f8588e4ab785244e06c3446fe09bf9db5dd8753 Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Debug: Platform: Linux,5.4.0-73-generic,x86_64,-junix,-smalloc,-sdefault,-hcritbit Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Platform: Linux,5.4.0-73-generic,x86_64,-junix,-smalloc,-sdefault,-hcritbit Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Debug: Child (3447) Started Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Child (3447) Started Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Info: Child (3447) said Child starts Aug 04 18:59:10 ubuntu-s-1vcpu-2gb-fra1-01 varnishd[3423]: Child (3447) said Child starts

      If you do not see both services up and running, wait a few minutes until they are fully loaded, and keep both of them running.

      Now that you have Apache2 Varnish, installed, you’ll give Varnish something to serve, in this case Apache’s static web page.

      Step 2 — Configuring Varnish To Serve Apache’s Static Web Page

      In the previous step, you installed Varnish, and next you’ll need to configure it. By default, Varnish listens on port 6081 and connects to a local web server on port 8080. You’ll change that to serve the Apache static site from Apache server.

      First, you’ll change Varnish’s listening port to 8080. Usually you would want the listening port to be 80, but because you are running Apache and Varnish on the same server, you’ll use port 8080 for Varnish and port 80 for Apache.

      There is no configuration option to change the listening port for Varnish, so you’ll do it using the command line. You’ll create a file called customexec.conf in a new directory called varnish.service.d in /etc/systemd/system/ that will change the default ports.

      Use the mkdir command to create the new directory:

      • sudo mkdir /etc/systemd/system/varnish.service.d

      Use your favorite text editor to create a new file called customexec.conf :

      • sudo nano /etc/systemd/system/varnish.service.d/customexec.conf

      In customexec.conf, add the following content:

      /etc/systemd/system/varnish.service.d/customexec.conf file

      [Service] ExecStart= ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :8080 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

      In this file you are changing the Service section of the Varnish configuration. First you remove the old value for the ExecStart option, and then you assign a new value for it.

      The new value specifies the binary file used to run Varnish with the following options:

      • -j: Specifies the jailing mechanism to use. Varnish jails are used to reduce the permissions for the varnish process over various platform-specific methods. Here you’re using the unix mechanism and the user vcache to limit the permissions. This is default for varnish on Ubuntu systems.

      • -F: Indicates that the server should run in the foreground, because systemd expects the main process to keep running so it can track it, and not fork a new process and die.

      • -a: This flag is used to specify the IP address and port for accepting client connections. The IP in this case is empty, which means the server will accept all IPs. The port is set to 8080.

      • -T: This flag specifies the IP address and port for management interface, in this case localhost and port 6082.

      • -f: This flag specifies the default VCL file for Varnish configuration. You will edit this file later in this tutorial to configure Varnish to connect to the Apache server.

      • -S: This flag specifies a shared secret file for authorizing access to the management interface. The /etc/varnish/secret value is the default for Varnish on Ubuntu. You will not use the secret file in this tutorial.

      • -s: This flag indicates where and how to store objects. The value malloc,256m is the default one for Vanish. It means to store various Varnish objects in memory using the malloc system call and a maximum size of 256 megabytes. Other possible values are default, which uses umem when malloc is not available, or file, which stores objects in a file on the disk.

      Save and close the customexec.conf file. Then execute this command to reload the systemd services file from disk:

      • sudo systemctl daemon-reload

      Then restart Varnish for changes to take effect.

      • sudo systemctl restart varnish

      You won’t see any output from these last two commands. To make sure that Varnish is now listening on port 8080, use the netstat command to display all listening TCP sockets on the server.

      • sudo netstat -ltnp | grep 8080

      You’ll see output that looks like this:

      Output

      tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 18689/varnishd tcp6 0 0 :::8080 :::* LISTEN 18689/varnishd

      Now that Varnish is running and listening to port 8080, you need to edit the default configuration file located at /etc/varnish/default.vcl:

      • sudo nano /etc/varnish/default.vcl

      Navigate to the backend default block, and then change .port to 80, as shown here:

      default.vcl file

      # Default backend definition. Set this to point to your content server. backend default { .host = "127.0.0.1"; .port = "80"; }

      Save and close the default.vcl file, then restart Varnish with this command:

      • sudo systemctl restart varnish

      If everything is fine, there won’t be any output. Open http://your_server_ip:8080 in your browser, and you’ll see the Apache static site, opened using Varnish.

      You now have Apache and Varnish running together on the same Droplet, with Apache listening to port 80 and Varnish to port 8080. Next, you’ll compare the response times of both servers using the wrk tool.

      Step 3 — Testing Varnish Using wrk

      wrk is a modern HTTP benchmarking tool. It is written in C, and can be used to load test web servers with many requests per second. In this step, you’ll use wrk to run tests against Apache and Varnish and then compare the results.

      First you’ll need to install wrk by building it from source. Start by installing some build tools for C and git, which are required for building wrk from source:

      • sudo apt-get install build-essential libssl-dev git unzip -y

      Then clone the git repository for wrk into the wrk directory:

      • git clone https://github.com/wg/wrk.git wrk

      Change to that new directory:

      Build the wrk executable with the make command:

      Copy wrk to the /usr/local/bin directory so you can access it from anywhere in your directory structure:

      • sudo cp wrk /usr/local/bin

      Now that you have wrk installed, use it to test the responsiveness of Apache with this command:

      • wrk -t2 -c1000 -d30s --latency http://server_ip/

      This command uses the following arguments:

      • -t2: This means run two threads.
      • -c1000: Keep 1000 HTTP connections open.
      • -d30s: Run the test for 30 seconds.
      • --latency: Print latency statistics.

      Wait 30 seconds until the test is done, and you’ll see output similar to this:

      output

      Running 30s test @ http://68.183.115.151/ 2 threads and 1000 connections Thread Stats Avg Stdev Max +/- Stdev Latency 44.45ms 104.50ms 1.74s 91.20% Req/Sec 8.29k 1.07k 12.40k 71.00% Latency Distribution 50% 11.59ms 75% 22.73ms 90% 116.16ms 99% 494.90ms 494677 requests in 30.04s, 5.15GB read Socket errors: connect 0, read 8369, write 0, timeout 69 Requests/sec: 16465.85 Transfer/sec: 175.45MB

      In this test, the average latency is 44.45ms, there were 494,677 total requests, 8,369 read errors, and 69 timeout errors. The exact numbers will vary in your installation.

      Now run the same test again for the Varnish server using this command:

      • wrk -t2 -c1000 -d30s --latency http://server_ip:8080/

      Wait 30 seconds until the test is done, and you’ll see output similar to this:

      output

      Running 30s test @ http://68.183.115.151:8080/ 2 threads and 1000 connections Thread Stats Avg Stdev Max +/- Stdev Latency 14.41ms 13.70ms 602.49ms 90.05% Req/Sec 6.67k 401.10 8.74k 83.33% Latency Distribution 50% 13.03ms 75% 17.69ms 90% 24.72ms 99% 58.22ms 398346 requests in 30.06s, 4.18GB read Socket errors: connect 0, read 19, write 0, timeout 0 Requests/sec: 13253.60 Transfer/sec: 142.48MB

      The output you see will likely be somewhat different, but the latency will be lower for Varnish than for Apache. In this case, the average latency is 14.41ms, there were 398,346 total requests, and no errors.

      In these tests, with Apache the average response time was 44.45ms with 8,438 errors, while Varnish achieved an increase in speed to 14.41ms, and also had no errors. This is because Varnish cached the response in memory and served it for later requests, unlike Apache, which needs to read from disk almost every time the resource is requested.

      Conclusion

      In this tutorial, you configured Varnish as a reverse proxy caching server for a static web site. You saw how to use basic HTTP caching to improve performance, and you used wrk to run load tests for the Apache and Varnish servers to compare the results.

      You’ve seen that the Varnish cache server speeds up your static site by serving content from main memory and not requesting it from the back end Apache server every time a new request arrives. For more information about other uses of Varnish, see the official documentation.



      Source link

      How To Deploy a Static HTML Website with Ansible on Ubuntu 20.04 (Nginx)



      Part of the Series:
      How To Write Ansible Playbooks

      Ansible is a modern configuration management tool that doesn’t require the use of an agent software on remote nodes, using only SSH and Python to communicate and execute commands on managed servers. This series will walk you through the main Ansible features that you can use to write playbooks for server automation. At the end, we’ll see a practical example of how to create a playbook to automate setting up a remote Nginx web server and deploy a static HTML website to it.

      If you were following along with all parts of this series, at this point you should be familiar with installing system packages, applying templates, and using handlers in Ansible playbooks. In this part of the series, you’ll use what you’ve seen so far to create a playbook that automates setting up a remote Nginx server to host a static HTML website on Ubuntu 20.04.

      Start by creating a new directory on your Ansible control node where you’ll set up the Ansible files and a demo static HTML website to be deployed to your remote server. This could be in any location of your choice within your home folder. In this example we’ll use ~/ansible-nginx-demo.

      • mkdir ~/ansible-nginx-demo
      • cd ~/ansible-nginx-demo

      Next, copy your existing inventory file into the new directory. In this example, we’ll use the same inventory you set up at the beginning of this series:

      • cp ~/ansible-practice/inventory .

      This will copy a file named inventory from a folder named ansible-practice in your home directory, and save it to the current directory.

      Obtaining the Demo Website

      For this demonstration, we’ll use a static HTML website that is the subject of our How To Code in HTML series. Start by downloading the demo website files by running the following command:

      • curl -L https://github.com/do-community/html_demo_site/archive/refs/heads/main.zip -o html_demo.zip

      You’ll need unzip to unpack the contents of this download. To make sure you have this tool installed, run:

      Then, unpack the demo website files with:

      This will create a new directory called html_demo_site-main on your current working directory. You can check the contents of the directory with an ls -la command:

      • ls -la html_demo_site-main

      Output

      total 28 drwxrwxr-x 3 sammy sammy 4096 sep 18 2020 . drwxrwxr-x 5 sammy sammy 4096 mrt 25 15:03 .. -rw-rw-r-- 1 sammy sammy 1289 sep 18 2020 about.html drwxrwxr-x 2 sammy sammy 4096 sep 18 2020 images -rw-rw-r-- 1 sammy sammy 2455 sep 18 2020 index.html -rw-rw-r-- 1 sammy sammy 1079 sep 18 2020 LICENSE -rw-rw-r-- 1 sammy sammy 675 sep 18 2020 README.md

      Creating a Template for Nginx’s Configuration

      You’ll now set up the Nginx template that is necessary to configure the remote web server. Create a new folder within your ansible-demo directory to hold non-playbook files:

      Then, open a new file called nginx.conf.j2:

      This template file contains an Nginx server block configuration for a static HTML website. It uses three variables: document_root, app_root, and server_name. We’ll define these variables later on when creating the playbook. Copy the following content to your template file:

      ~/ansible-nginx-demo/files/nginx.conf.j2

      server {
        listen 80;
      
        root {{ document_root }}/{{ app_root }};
        index index.html index.htm;
      
        server_name {{ server_name }};
      
        location / {
         default_type "text/html";
         try_files $uri.html $uri $uri/ =404;
        }
      }
      

      Save and close the file when you’re done.

      Creating a New Ansible Playbook

      Next, we’ll create a new Ansible playbook and set up the variables that we’ve used in the previous section of this guide. Open a new file named playbook.yml:

      This playbook starts with the hosts definition set to all and a become directive that tells Ansible to run all tasks as the root user by default (the same as manually running commands with sudo). Within this playbook’s var section, we’ll create three variables: server_name, document_root, and app_root. These variables are used in the Nginx configuration template to set up the domain name or IP address that this web server will respond to, and the full path to where the website files are located on the server. For this demo, we’ll use the ansible_default_ipv4.address fact variable because it contains the remote server’s public IP address, but you can replace this value with your server’s hostname in case it has a domain name properly configured within a DNS service to point to this server:

      ~/ansible-nginx-demo/playbook.yml

      ---
      - hosts: all
        become: yes
        vars:
          server_name: "{{ ansible_default_ipv4.address }}"
          document_root: /var/www/html
          app_root: html_demo_site-main
        tasks:
      

      You can keep this file open for now. The next sections will walk you through all tasks that you’ll need to include in this playbook to make it fully functional.

      Installing Required Packages

      The following task will update the apt cache and then install the nginx package on remote nodes:

      ~/ansible-nginx-demo/playbook.yml

      . . .
          - name: Update apt cache and install Nginx
            apt:
              name: nginx
              state: latest
              update_cache: yes
      

      Uploading Website Files to Remote Nodes

      The next task will use the copy built-in module to upload the website files to the remote document root. We’ll use the document_root variable to set the destination on the server where the application folder should be created.

      ~/ansible-nginx-demo/playbook.yml

      . . .
          - name: Copy website files to the server's document root
            copy:
              src: "{{ app_root }}"
              dest: "{{ document_root }}"
              mode: preserve
      

      Applying and Enabling the Custom Nginx Configuration

      We’ll now apply the Nginx template that will configure the web server to host your static HTML file. After the configuration file is set at /etc/nginx/sites-available, we’ll create a symbolic link to that file inside /etc/nginx-sites-enabled and notify the Nginx service for a posterior restart. The entire process will require two separate tasks:

      ~/ansible-nginx-demo/playbook.yml

      . . .
          - name: Apply Nginx template
            template:
              src: files/nginx.conf.j2
              dest: /etc/nginx/sites-available/default
            notify: Restart Nginx
      
          - name: Enable new site
            file:
              src: /etc/nginx/sites-available/default
              dest: /etc/nginx/sites-enabled/default
              state: link
            notify: Restart Nginx
      

      Allowing Port 80 on UFW

      Next, include the task that enables tcp access on port 80:

      ~/ansible-nginx-demo/playbook.yml

      . . .
          - name: Allow all access to tcp port 80
            ufw:
              rule: allow
              port: '80'
              proto: tcp
      . . .
      

      Creating a Handler for the Nginx Service

      To finish this playbook, the only thing left to do is to set up the Restart Nginx handler:

      ~/ansible-nginx-demo/playbook.yml

      . . .
        handlers:
          - name: Restart Nginx
            service:
              name: nginx
              state: restarted  
      

      Running the Finished Playbook

      Once you’re finished including all the required tasks in your playbook file, it will look like this:

      ~/ansible-nginx-demo/playbook.yml

      ---
      - hosts: all
        become: yes
        vars:
          server_name: "{{ ansible_default_ipv4.address }}"
          document_root: /var/www
          app_root: html_demo_site-main
        tasks:
          - name: Update apt cache and install Nginx
            apt:
              name: nginx
              state: latest
              update_cache: yes
      
          - name: Copy website files to the server's document root
            copy:
              src: "{{ app_root }}"
              dest: "{{ document_root }}"
              mode: preserve
      
          - name: Apply Nginx template
            template:
              src: files/nginx.conf.j2
              dest: /etc/nginx/sites-available/default
            notify: Restart Nginx
      
          - name: Enable new site
            file:
              src: /etc/nginx/sites-available/default
              dest: /etc/nginx/sites-enabled/default
              state: link
            notify: Restart Nginx
      
          - name: Allow all access to tcp port 80
            ufw:
              rule: allow
              port: '80'
              proto: tcp
      
        handlers:
          - name: Restart Nginx
            service:
              name: nginx
              state: restarted
      

      To execute this playbook on the server(s) that you set up in your inventory file, run ansible-playbook with the same connection arguments you’ve used when running a connection test within the introduction of this series. Here, we’ll be using an inventory file named inventory and the sammy user to connect to the remote server. Because the playbook requires sudo to run, we’re also including the -K argument to provide the remote user’s sudo password when prompted by Ansible:

      • ansible-playbook -i inventory playbook.yml -u sammy -K

      You’ll see output like this:

      Output

      BECOME password: PLAY [all] ********************************************************************************************** TASK [Gathering Facts] ********************************************************************************** ok: [203.0.113.10] TASK [Update apt cache and install Nginx] *************************************************************** ok: [203.0.113.10] TASK [Copy website files to the server's document root] ************************************************* changed: [203.0.113.10] TASK [Apply Nginx template] ***************************************************************************** changed: [203.0.113.10] TASK [Enable new site] ********************************************************************************** ok: [203.0.113.10] TASK [Allow all access to tcp port 80] ****************************************************************** ok: [203.0.113.10] RUNNING HANDLER [Restart Nginx] ************************************************************************* changed: [203.0.113.10] PLAY RECAP ********************************************************************************************** 203.0.113.10 : ok=7 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

      Once the playbook is finished, if you go to your browser and access your server’s hostname or IP address you should now see the following page:

      HTML Demo Site Deployed by Ansible

      Congratulations, you have successfully automated the deployment of a static HTML website to a remote Nginx server, using Ansible.

      If you make changes to any of the files in the demo website, you can run the playbook again and the copy task will make sure any file changes are reflected in the remote host. Because Ansible has an idempotent behavior, running the playbook multiple times will not trigger changes that were already made to the system.



      Source link