One place for hosting & domains

      Policies

      Working with CORS Policies on Linode Object Storage


      Linode Object Storage offers a globally-available, S3-compatible storage solution. Whether you are storing critical backup files or data for a static website, S3 object storage can efficiently answer the call.

      To make the most of object storage, you may need to access the data from other domains. For instance, your dynamic applications may opt to use S3 for static file storage.

      This leaves you dealing with Cross-Origin Resource Sharing, or CORS. However, it’s often not clear how to effectively navigate CORS policies or deal with issues as they come up.

      This tutorial aims to clarify how to work with CORS and S3. It covers tools and approaches for effectively reviewing and managing CORS policies for Linode Object Storage or most other S3-compatible storage solutions.

      CORS and S3 Storage – What you Need to Know

      Linode Object Storage is an S3, which stands for simple storage service. With S3, data gets stored as objects in “buckets.” This gives S3s a flat approach to storage, in contrast to the hierarchical and logistically more complicated storage structures like traditional file systems. Objects stored in S3 can also be given rich metadata.

      CORS defines how clients and servers from different domains may share resources. Generally, CORS policies restrict access to resources to requests from the same domain. By managing your CORS policies, you can open up services to requests from specified origin domains, or from any domains whatsoever.

      An S3 like Linode Object Storage can provide excellent storage for applications. However, you also want to keep your data as secure as possible while also allowing your applications the access they need.

      This is where managing CORS policies on your object storage service becomes imperative. Applications and other tools often need to access stored resources from particular domains. Implementing specific CORS policies controls what kinds of requests, and responses, each origin domain is allowed.

      Working with CORS Policies on Linode Object Storage

      One of the best tools for managing policies on your S3, including Linode Object Storage, is s3cmd. Follow along with our guide
      Using S3cmd with Object Storage to:

      1. Install s3cmd on your system. The installation takes place on the system from which you intend to manage your S3 instance.

      2. Configure s3cmd for your Linode Object Storage instance. This includes indicating the instance’s access key, endpoint, etc.

      You can verify the connection to your object storage instance with the command to list your buckets. This example lists the one bucket used for this tutorial, example-cors-bucket:

      s3cmd ls
      
      2022-09-24 16:13  s3://example-cors-bucket

      Once you have s3cmd set up for your S3 instance, use it to follow along with the upcoming sections of this tutorial. These show you how to use the tool to review and deploy CORS policies.

      Reviewing CORS Policies for Linode Object Storage

      You can get the current CORS policies for your S3 bucket using the info flag for s3cmd. The command provides general information on the designated bucket, including its policies:

      s3cmd info s3://example-cors-bucket
      
      s3://example-cors-bucket/ (bucket):
         Location:  default
         Payer:     BucketOwner
         Expiration Rule: none
         Policy:    none
         CORS:      <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CORSRule><AllowedMethod>GET</AllowedMethod><AllowedMethod>PUT</AllowedMethod><AllowedMethod>DELETE</AllowedMethod><AllowedMethod>HEAD</AllowedMethod><AllowedMethod>POST</AllowedMethod><AllowedOrigin>*</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>
         ACL:       31ffbc26-d6ed-4bc3-8a14-ad78fe8f95b6: FULL_CONTROL

      This bucket already has a CORS policy in place. This is because it was set up with the CORS Enabled setting using the Linode Cloud Manager web interface.

      The basic CORS policy above is fairly permissive, allowing access for any request method from any domain. Keep reading to see how you can fine-tune such policies to better fit your particular needs.

      Deploying CORS Policies on Linode Object Storage

      As you can see above, the Linode Cloud Manager can set up a general CORS policy for your bucket. However, if you need more fine-grained control, you need to deploy custom CORS policies.

      Creating CORS policies follows a similar methodology to the one outlined in our
      Define Access and Permissions using Bucket Policies tutorial.

      These next sections break down the particular fields needed for CORS policies and how each affects your bucket’s availability.

      Configuring Policies

      The overall structure for CORS policies on S3 looks like the following. While policies on your object storage instance can generally be set with JSON or XML, CORS policies must use the XML format:

      File: cors_policies.xml
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      
      <CORSConfiguration>
        <CORSRule>
          <AllowedHeader>*</AllowedHeader>
      
          <AllowedMethod>GET</AllowedMethod>
          <AllowedMethod>PUT</AllowedMethod>
          <AllowedMethod>POST</AllowedMethod>
          <AllowedMethod>DELETE</AllowedMethod>
          <AllowedMethod>HEAD</AllowedMethod>
      
          <AllowedOrigin>*</AllowedOrigin>
      
          <ExposeHeader>*</ExposeHeader>
      
          <MaxAgeSeconds>3000</MaxAgeSeconds>
        </CORSRule>
      </CORSConfiguration>

      To break this structure down:

      • The policy introduces a list of one or more <CORSRule> elements within a <CORSConfiguration> element. Each <CORSRule> element contains policy details.

      • Policies tend to have some combination of the five types of elements shown in the example above.

        The <AllowedHeader>, <AllowedMethod>, and <AllowedOrigin> elements are almost always present. Further, there may be multiple of these elements within a single <CORSRule>.

        The other two elements, <ExposeHeader> and <MaxAgeSeconds>, are optional. There can be multiple <ExposeHeader> elements, but only one <MaxAgeSeconds>.

      • <AllowedHeader> lets you specify request headers allowed for the given policy. You can find a list of commonly used request headers in AWS’s
        Common Request Headers documentation.

      • <AllowedMethod> lets you specify request methods that the given policy applies to. The full range of supported HTTP request methods is shown in the example above.

      • <AllowedOrigin> lets you specify request origins for the policy. These are the domains from which cross-origin requests can be made.

      • <ExposeHeader> can specify which response headers the policy allows to be exposed. You can find a list of commonly used response headers in AWS’s
        Common Response Headers documentation.

      • <MaxAgeSeconds> can specify the amount of time, in seconds, that browsers are allowed to cache the response to preflight requests. Having this cache allows the browser to repeat the original requests without having to send another preflight request.

      Example CORS Policies

      To give more concrete ideas of how you can work with CORS policies, the following are two additional example policies. One provides another simple, but more limited, policy, while the other presents a more complicated set of two policies.

      • First, a public access read-only policy. This lets any origin, with any request headers, make GET and HEAD requests to the bucket. However, the policy does not expose custom response headers.

        File: cors_policies.xml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        
        <CORSConfiguration>
          <CORSRule>
            <AllowedHeader>*</AllowedHeader>
        
            <AllowedMethod>GET</AllowedMethod>
            <AllowedMethod>HEAD</AllowedMethod>
        
            <AllowedOrigin>*</AllowedOrigin>
          </CORSRule>
        </CORSConfiguration>
            
      • Next, a set of policies for fine control over requests from example.com. The <AllowedOrigin> elements specify the range of possible example.com domains. The two policies distinguish the kinds of headers allowed based on the kinds of request methods.

        File: cors_policies.xml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        36
        
        <CORSConfiguration>
          <CORSRule>
            <AllowedHeader>Authorization</AllowedHeader>
        
            <AllowedMethod>GET</AllowedMethod>
            <AllowedMethod>HEAD</AllowedMethod>
        
            <AllowedOrigin>http://example.com</AllowedOrigin>
            <AllowedOrigin>http://*.example.com</AllowedOrigin>
            <AllowedOrigin>https://example.com</AllowedOrigin>
            <AllowedOrigin>https://*.example.com</AllowedOrigin>
        
            <ExposeHeader>Access-Control-Allow-Origin</ExposeHeader>
        
            <MaxAgeSeconds>3000</MaxAgeSeconds>
          </CORSRule>
          <CORSRule>
            <AllowedHeader>Authorization</AllowedHeader>
            <AllowedHeader>Origin</AllowedHeader>
            <AllowedHeader>Content-*</AllowedHeader>
        
            <AllowedMethod>PUT</AllowedMethod>
            <AllowedMethod>POST</AllowedMethod>
            <AllowedMethod>DELETE</AllowedMethod>
        
            <AllowedOrigin>http://example.com</AllowedOrigin>
            <AllowedOrigin>http://*.example.com</AllowedOrigin>
            <AllowedOrigin>https://example.com</AllowedOrigin>
            <AllowedOrigin>https://*.example.com</AllowedOrigin>
        
            <ExposeHeader>ETag</ExposeHeader>
        
            <MaxAgeSeconds>3000</MaxAgeSeconds>
          </CORSRule>
        </CORSConfiguration>
            

      Deploying Policies

      The next step is to actually deploy your CORS policies. Once you do, your S3 bucket starts following them to determine what origins to allow and what request and response information to permit.

      Follow these steps to put your CORS policies into practice on your S3 instance.

      1. Save your CORS policy into a XML file. This example uses a file named cors_policies.xml which contains the second example policy XML above.

      2. Use s3cmd’s setcors commands to deploy the CORS policies to the bucket. This command takes the policy XML file and the bucket identifier as arguments:

        s3cmd setcors cors_policies.xml s3://example-cors-bucket
        
      3. Verify the new CORS policies using the info command as shown earlier in this tutorial:

        s3cmd info s3://example-cors-bucket
        
        s3://example-cors-bucket/ (bucket):
           Location:  default
           Payer:     BucketOwner
           Expiration Rule: none
           Policy:    none
           CORS:      <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CORSRule><AllowedMethod>GET</AllowedMethod><AllowedMethod>HEAD</AllowedMethod><AllowedOrigin>http://*.example.com</AllowedOrigin><AllowedOrigin>http://example.com</AllowedOrigin><AllowedOrigin>https://*.example.com</AllowedOrigin><AllowedOrigin>https://example.com</AllowedOrigin><AllowedHeader>Authorization</AllowedHeader><MaxAgeSeconds>3000</MaxAgeSeconds><ExposeHeader>Access-Control-Allow-Origin</ExposeHeader></CORSRule><CORSRule><AllowedMethod>PUT</AllowedMethod><AllowedMethod>DELETE</AllowedMethod><AllowedMethod>POST</AllowedMethod><AllowedOrigin>http://*.example.com</AllowedOrigin><AllowedOrigin>http://example.com</AllowedOrigin><AllowedOrigin>https://*.example.com</AllowedOrigin><AllowedOrigin>https://example.com</AllowedOrigin><AllowedHeader>Authorization</AllowedHeader><AllowedHeader>Content-*</AllowedHeader><AllowedHeader>Origin</AllowedHeader><MaxAgeSeconds>3000</MaxAgeSeconds><ExposeHeader>ETag</ExposeHeader></CORSRule></CORSConfiguration>
           ACL:       31ffbc26-d6ed-4bc3-8a14-ad78fe8f95b6: FULL_CONTROL

      Troubleshooting Common CORS Errors

      Having CORS-related issues on your S3 instance? Take these steps to help narrow down the issue and figure out the kind of policy change needed to resolve it.

      1. Review your instance’s CORS policies using s3cmd:

        s3cmd info s3://example-cors-bucket
        

        This can give you a concrete reference for what policies are in place and the specific details of each, like header and origin information.

      2. Review the request and response data. This can give you insights on any possible inconsistencies between existing CORS policies and the actual requests and responses.

        You can use a tool like cURL for this. First, use s3cmd to create a signed URL to an object on your storage instance. This example command creates a URL for an example.txt object and makes the URL last 300 seconds:

        s3cmd signurl s3://example-cors-bucket/example.txt +300
        

        Now, until the URL expires, you can use a cURL command like this one to send a request for the object:

        curl -v "http://example-cors-bucket.us-southeast-1.linodeobjects.com/index.md?AWSAccessKeyId=example-access-key&Expires=1664121793&Signature=example-signature"
        

        The -v option gives you verbose results, outputting more details to help you dissect any request and response issues.

      3. Compare the results of the cURL request to the CORS policy on your instance.

      Conclusion

      This covers the tools and approaches you need to start managing CORS for your Linode Object Storage or other S3 instance. Once you have these, addressing CORS issues is a matter of reviewing and adjusting policies against desired origins and request types.

      Keep improving your resources for managing your S3 through our collection of
      object storage guides. These cover a range of topics to help you with S3 generally, and Linode Object Storage in particular.

      Have more questions or want some help getting started? Feel free to reach out to our
      Support team.

      More Information

      You may wish to consult the following resources for additional information
      on this topic. While these are provided in the hope that they will be
      useful, please note that we cannot vouch for the accuracy or timeliness of
      externally hosted materials.



      Source link

      How to Enact Access Control Lists (ACLs) and Bucket Policies with Linode Object Storage


      Updated by Linode

      Contributed by
      Linode

      Linode Object Storage allows users to share access to objects and buckets with other Object Storage users. There are two mechanisms for setting up sharing: Access Control Lists (ACLs), and bucket policies. These mechanisms perform similar functions: both can be used to restrict and grant access to Object Storage resources.

      In this guide you will learn:

      Before You Begin

      • This guide will use the s3cmd command line utility to interact with Object Storage. For s3cmd installation and configuration instructions, visit our How to Use Object Storage guide.

      • You’ll also need the canonical ID of every user you wish to grant additional permissions to.

      Retrieve a User’s Canonical ID

      Follow these steps to determine the canonical ID of the Object Storage users you want to share with:

      1. The following command will return the canonical ID of a user, given any of the user’s buckets:

        s3cmd info s3://other-users-bucket
        

        Note

        The bucket referred to in this section is an arbitrary bucket on the target user’s account. It is not related to the bucket on your account that you would like to set ACLs or bucket policies on.

        There are two options for running this command:

        • The users you’re granting or restricting access to can run this command on one of their buckets and share their canonical ID with you, or:

        • You can run this command yourself if you have use of their access tokens (you will need to configure s3cmd to use their access tokens instead of your own).

      2. Run the above command, replacing other-users-bucket with the name of the bucket. You’ll see output similar to the following:

          
        s3://other-users-bucket/ (bucket):
        Location:  default
        Payer:     BucketOwner
        Expiration Rule: none
        Policy:    none
        CORS:      none
        ACL:       a0000000-000a-0000-0000-00d0ff0f0000: FULL_CONTROL
        
        
      3. The canonical ID of the owner of the bucket is the long string of letters, dashes, and numbers found in the line labeled ACL, which in this case is a0000000-000a-0000-0000-00d0ff0f0000.

      4. Alternatively, you may be able to retrieve the canonical ID by curling a bucket and retrieving the Owner ID field from the returned XML. This method is an option when both of these conditions are true:

        • The bucket has objects within it and has already been set to public (with a command like s3cmd setacl s3://other-users-bucket --acl-public).
        • The bucket has not been set to serve static websites.
      5. Run the curl command, replacing the bucket name and cluster URL with the relevant values:

        curl other-users-bucket.us-east-1.linodeobjects.com
        
      6. This will result in the following output:

        <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
            <Name>acl-bucket-example</Name>
            <Prefix/>
            <Marker/>
            <MaxKeys>1000</MaxKeys>
            <IsTruncated>false</IsTruncated>
            <Contents>
            <Key>cpanel_one-click.gif</Key>
            <LastModified>2019-11-20T16:52:49.946Z</LastModified>
            <ETag>"9aeafcb192a8e540e7be5b51f7249e2e"</ETag>
            <Size>961023</Size>
            <StorageClass>STANDARD</StorageClass>
            <Owner>
                <ID>a0000000-000a-0000-0000-00d0ff0f0000</ID>
                <DisplayName>a0000000-000a-0000-0000-00d0ff0f0000</DisplayName>
            </Owner>
            <Type>Normal</Type>
            </Contents>
        </ListBucketResult>
        

        In the above output, the canonical ID is a0000000-000a-0000-0000-00d0ff0f0000.

      ACLs vs Bucket Policies

      ACLs and bucket policies perform similar functions: both can restrict or grant access to buckets. ACLs can also restrict or grant access to individual objects, but they don’t offer as many fine-grained access modes as bucket policies.

      How to Choose Between ACLs and Bucket Policies

      If you can organize objects with similar permission needs into their own buckets, then it’s strongly suggested that you use bucket policies. However, if you cannot organize your objects in this fashion, ACLs are still a good option.

      ACLs offer permissions with less fine-grained control than the permissions available through bucket policies. If you are looking for more granular permissions beyond read and write access, choose bucket policies over ACLs.

      Additionally, bucket policies are created by applying a written bucket policy file to the bucket. This file cannot exceed 20KB in size. If you have a policy with a lengthy list of policy rules, you may want to look into ACLs instead.

      Note

      ACLs and bucket policies can be used at the same time. When this happens, any rule that limits access to an Object Storage resource will override a rule that grants access. For instance, if an ACL allows a user access to a bucket, but a bucket policy denies that user access, the user will not be able to access that bucket.

      ACLs

      Access Control Lists (ACLs) are a legacy method of defining access to Object Storage resources. You can apply an ACL to a bucket or to a specific object. There are two generalized modes of access: setting buckets and/or objects to be private or public. A few other more granular settings are also available.

      With s3cmd, you can set a bucket to be public with the setacl command and the --acl-public flag:

      s3cmd setacl s3://acl-example --acl-public
      

      This will cause the bucket and its contents to be downloadable over the general Internet.

      To set an object or bucket to private, you can use the setacl command and the --acl-private flag:

      s3cmd setacl s3://acl-example --acl-private
      

      This will prevent users from accessing the bucket’ contents over the general Internet.

      Other ACL Permissions

      The more granular permissions are:

      PermissionDescription
      readUsers with can list objects within a bucket
      writeUsers can upload objects to a bucket and delete objects from a bucket.
      read_acpUsers can read the ACL currently applied to a bucket.
      write_acpUsers can change the ACL applied to the bucket.
      full_controlUsers have read and write access over both objects and ACLs.
      • Setting a permission: To apply these more granular permissions for a specific user with s3cmd, use the following setacl command with the --acl-grant flag:

        s3cmd setacl s3://acl-example --acl-grant=PERMISSION:CANONICAL_ID
        

        Substitute acl-example with the name of the bucket (and the object, if necessary), PERMISSION with a permission from the above table, and CANONICAL_ID with the canonical ID of the user to which you would like to grant permissions.

      • Revoking a permission: To revoke a specific permission, you can use the setacl command with the acl-revoke flag:

        s3cmd setacl s3://acl-example --acl-revoke=PERMISSION:CANONICAL_ID
        

        Substitute the bucket name (and optional object), PERMISSION, and CANONICAL_ID with your relevant values.

      • View current ACLs: To view the current ACLs applied to a bucket or object, use the info command, replacing acl-example with the name of your bucket (and object, if necessary):

        s3cmd info s3://acl-example
        

        You should see output like the following:

          
        s3://acl-bucket-example/ (bucket):
           Location:  default
           Payer:     BucketOwner
           Expiration Rule: none
           Policy:    none
           CORS:      b'<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CORSRule><AllowedMethod>GET</AllowedMethod><AllowedMethod>PUT</AllowedMethod><AllowedMethod>DELETE</AllowedMethod><AllowedMethod>HEAD</AllowedMethod><AllowedMethod>POST</AllowedMethod><AllowedOrigin>*</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>'
           ACL:       *anon*: READ
           ACL:       a0000000-000a-0000-0000-00d0ff0f0000: FULL_CONTROL
           URL:       http://us-east-1.linodeobjects.com/acl-example/
        
        

        Note

        The owner of the bucket will always have the full_control permission.

      Bucket Policies

      Bucket policies can offer finer control over the types of permissions you can grant to a user. Below is an example bucket policy written in JSON:

      bucket_policy_example.txt
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      
      {
        "Version": "2012-10-17",
        "Statement": [{
          "Effect": "Allow",
          "Principal": {
            "AWS": [
              "arn:aws:iam:::a0000000-000a-0000-0000-00d0ff0f0000"
            ]
          },
          "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::bucket-policy-example/*"
          ]
        }]
      }

      This policy allows the user with the canonical ID a0000000-000a-0000-0000-00d0ff0f0000, known here as the “principal”, to interact with the bucket, known as the “resource”. The “resource” that is listed (bucket-policy-example) is the only bucket the user will have access to.

      Note

      The principal (a.k.a. the user) must have the prefix of arn:aws:iam:::, and the resource (a.k.a. the bucket) must have the prefix of arn:aws:s3:::.

      The permissions are specified in the Action array. For the current example, these are:

      The Action and Principal.AWS fields of the bucket policy are arrays, so you can easily add additional users and permissions to the bucket policy, separating them by a comma. To grant permissions to all users, you can supply a wildcard (*) to the Principal.AWS field.

      If you instead wanted to deny access to the user, you could change the Effect field to Deny.

      Enable a Bucket Policy

      To enable the bucket policy, use the setpolicy s3cmd command, supplying the file name of the bucket policy as the first argument, and the S3 bucket address as the second argument:

      s3cmd setpolicy bucket_policy_example.txt s3://bucket-policy-example
      

      To ensure that it has been applied correctly, you can use the info command:

      s3cmd info s3://bucket-policy-example
      

      You should see output like the following:

        
      s3://bucket-policy-example/ (bucket):
         Location:  default
         Payer:     BucketOwner
         Expiration Rule: none
         Policy:    b'{n  "Version": "2012-10-17",n  "Statement": [{n    "Effect": "Allow",n    "Principal": {"AWS": ["arn:aws:iam:::a0000000-000a-0000-0000-00d0ff0f0000"]},n    "Action": ["s3:PutObject","s3:GetObject","s3:ListBucket"],n    "Resource": [n      "arn:aws:s3:::bucket-policy-example/*"n    ]n  }]n}'
         CORS:      none
         ACL:       a0000000-000a-0000-0000-00d0ff0f0000: FULL_CONTROL
      
      

      Note

      The policy is visible in the output.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      How To Manage Objects with Lifecycle Policies


      Updated by Linode

      Contributed by
      Linode

      Note

      While deleting a few objects in an Object Storage bucket might not take that long, when the objects number in the thousands or even millions the time required to complete the delete operations can easily become unmanageable. When deleting a substantial amount of objects, it’s best to use lifecycle policies. These policies can be represented in XML; here’s an (incomplete) snippet of an action that will delete objects after 1 day:

      1
      2
      3
      
      <Expiration>
          <Days>1</Days>
      </Expiration>

      A lifecycle policy is applied to a bucket. Policies are sets of rules that govern the management of objects after they have aged for a certain amount of time. For instance, you can create a lifecycle policy that deletes objects every thirty days, or once a week. This is useful for cases where the data in a bucket becomes outdated, such as when collecting activity logs.

      In This Guide

      This guide will first describe when policies are enforced and will then explain how to create and delete lifecycle policies with two tools:

      • s3cmd command line interface (CLI): In addition to deleting objects, more complicated policies can be managed with s3cmd, including deleting old versions of objects that have been retained, and failed multipart uploads.

      • Cyberduck desktop application (GUI): Cyberduck does not feature as many policy options, but they can be managed through a point-and-click interface.

      Before You Begin

      • Familiarize yourself with Linode Object Storage by reading the How to Use Object Storage guide.
      • For demonstration purposes, you can create an Object Storage bucket with a few objects that you will later delete.

      When Policies are Enforced

      Lifecycle policies are triggered starting at midnight of the Object Storage cluster’s local time. This means that if you set a lifecycle policy of one day, the objects will be deleted the midnight after they become 24 hours old.

      For example, if an object is created at 5PM on January 1, it will reach 24 hours in age at 5PM on January 2. The policy will then be enforced on the object at 12AM on January 3.

      Note

      There is a chance that a lifecycle policy will not delete all of the files in a bucket the first time the lifecycle policy is triggered. This is especially true for buckets with upwards of a million objects. In cases like these, most of the objects are deleted, and any remaining objects are typically deleted during the next iteration of the lifecycle policy’s rules.

      Create and Delete Lifecycle Policies

      s3cmd

      s3cmd allows users to set and manage lifecycle policies from the command line. In this section, you will find instructions on how to create and manage lifecycle policies to delete objects, previous versions of objects, and failed multipart uploads using s3cmd.

      Note

      If you don’t have s3cmd set up on your computer, visit the Install and Configure s3cmd section of the How to Use Linode Object Storage guide.

      Creating a Lifecycle Policy File

      In S3-compatible Object Storage, a lifecycle policy is represented by an XML file. You can use your preferred text editor to create this XML file. Consider the following lifecycle policy file:

      lifecycle_policy.xml
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      
      <LifecycleConfiguration>
          <Rule>
              <ID>delete-all-objects</ID>
              <Prefix></Prefix>
              <Status>Enabled</Status>
              <Expiration>
                  <Days>1</Days>
              </Expiration>
          </Rule>
      </LifecycleConfiguration>

      The above lifecycle policy deletes all objects in the bucket after one day. Each lifecycle policy file needs a LifecycleConfiguration block and a nested Rule block. The Rule block must contain Prefix and Status, and at least one action, like the Expiration block. It’s also a good idea to include an ID block:

      BlockDescription
      IDDefines a name for the lifecycle policy rule. If your lifecycle policy contains multiple rules, then the ID for each should be unique. If one is not specified in your policy file, then a random alphanumeric ID will be assigned to your policy when the policy is applied to a bucket.
      PrefixThis string is used to select objects for deletion with the same matching prefix. For example, objects that begin with error_report- could be targeted for deletion by providing this prefix. This Prefix can be empty if you want a rule to apply to all files in a bucket.
      StatusA string value describing the status of the lifecycle policy. To enable the policy, set this value to Enabled. To disable the policy set the value to Disabled.
      ExpirationContains the Days block. The Days block is the number of days before this rule will be enforced. In the above example, the Days is set to 1, meaning that the objects in the bucket will be deleted after one day.

      Additional Actions

      Other actions can also be specified in a rule:

      • NoncurrentVersionExpiration block, and its child, NoncurrentDays. These are used to control the lifecycle of objects with multiple older versions, and should only be used with buckets that have bucket versioning enabled. Using this option will delete objects that are not the newest, most current version. Below is an example of how to use NoncurrentVersionExpiration:

        lifecycle_policy_noncurrent_versions.xml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        
        <LifecycleConfiguration>
            <Rule>
                <ID>delete-prior-versions</ID>
                <Prefix></Prefix>
                <Status>Enabled</Status>
                <NoncurrentVersionExpiration>
                    <NoncurrentDays>1</NoncurrentDays>
                </NoncurrentVersionExpiration>
            </Rule>
        </LifecycleConfiguration>
      • AbortIncompleteMultipartUpload, and its child, DaysAfterInitiation. These work similarly to NoncurrentVersionExpiration, but instead of deleting previous versions of objects, they will delete failed multipart uploads. The following will delete failed multipart uploads three days after they were initiated:

        lifecycle_policy_multipart_upload.xml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        
        <LifecycleConfiguration>
            <Rule>
                <ID>delete-incomplete-multipart-uploads</ID>
                <Prefix></Prefix>
                <Status>Enabled</Status>
                <AbortIncompleteMultipartUpload>
                    <DaysAfterInitiation>3</DaysAfterInitiation>
                </AbortIncompleteMultipartUpload>
            </Rule>
        </LifecycleConfiguration>



        About multipart uploads

        Objects that are part of failed multipart uploads (the mechanism by which large files are uploaded) stay within Object Storage buckets, counting towards your total Object Storage costs. s3cmd will automatically initiate a multipart upload when a file is larger than 15MB. Lifecycle policies are a great way to clear out stale multipart uploads.

      Multiple Actions in One Rule

      More than one action can be specified in a single rule. For example, you may want to both expire the current version of an object after a set number of days and also remove old versions of it after another period of time. The following policy will delete the current version of an object after 10 days and remove any noncurrent versions of an object 3 days after they are demoted from the current version:

      lifecycle_policy_multipart_upload.xml
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      
      <LifecycleConfiguration>
          <Rule>
              <ID>delete-prior-versions</ID>
              <Prefix></Prefix>
              <Status>Enabled</Status>
              <Expiration>
                  <Days>10</Days>
              </Expiration>
              <NoncurrentVersionExpiration>
                  <NoncurrentDays>3</NoncurrentDays>
              </NoncurrentVersionExpiration>
          </Rule>
      </LifecycleConfiguration>

      Note

      As a reminder, if a versioned object is deleted, only the current version of the object will be deleted and all older versions will be preserved in the bucket. For this reason, the above rule has the effect of deleting any objects if they are not updated within 10 days, and then removing the remaining object versions after 3 days.

      Multiple Rules

      A lifecycle policy file can only contain one LifecycleConfiguration block, but the LifecycleConfiguration block can contain more than one Rule. For instance, if you had a bucket that contained both error and general output logs, you could set a lifecycle policy that saves error logs for a week but deletes standard logs at the end of every day:

      lifecycle_policy_error_and_standard_logs.xml
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      
      <LifecycleConfiguration>
          <Rule>
              <ID>delete-error-logs</ID>
              <Prefix>error</Prefix>
              <Status>Enabled</Status>
              <Expiration>
                  <Days>7</Days>
              </Expiration>
          </Rule>
          <Rule>
              <ID>delete-standard-logs</ID>
              <Prefix>logs</Prefix>
              <Status>Enabled</Status>
              <Expiration>
                  <Days>1</Days>
              </Expiration>
          </Rule>
      </LifecycleConfiguration>

      Uploading the Lifecycle Policy to a Bucket

      In order to apply a lifecycle policy to a bucket with s3cmd, you need to upload the lifecycle file to the bucket. This operation is not a normal PUT operation. Instead, the command to use is setlifecycle, followed by the name of the lifecycle policy file, and the name of bucket:

      s3cmd setlifecycle lifecycle_policy.xml s3://lifecycle-policy-example
      

      You should see output like the following:

      s3://lifecycle-policy-example/: Lifecycle Policy updated
      

      Once the lifecycle policy has been uploaded, objects will be deleted according to the policy set in place.

      Viewing a Bucket’s Lifecycle Policy

      To view a lifecycle policy after it has been uploaded to a bucket, use the getlifecycle command and provide the bucket name:

      s3cmd getlifecycle s3://lifecycle-policy-example
      

      You should see the contents of the XML file that was uploaded:

      <?xml version="1.0" ?>
      <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
        <Rule>
          <ID>delete-all</ID>
          <Prefix/>
          <Status>Enabled</Status>
          <Expiration>
            <Days>1</Days>
          </Expiration>
        </Rule>
      </LifecycleConfiguration>
      

      Deleting a Lifecycle Policy

      To delete a lifecycle policy that you’ve uploaded, effectively disabling it, use the dellifecycle command and provide the bucket name:

      s3cmd dellifecycle s3://lifecycle-policy-example
      

      You’ll see a confirmation that the lifecycle policy was deleted:

      s3://lifecycle-example/: Lifecycle Policy deleted
      

      Cyberduck

      Cyberduck allows less control over lifecycle polices than the s3cmd CLI. In particular, Cyberduck does not allow you to set a lifecycle policy that removes outdated versions of objects stored in buckets where versioning is enabled, nor does it allow you to delete multipart uploads. Cyberduck also limits the length of a lifecycle policy to commonly used time spans. Below you will learn how to set a lifecycle policy using Cyberduck.

      Note

      Enable a Lifecycle Policy

      1. Right click or control + click on the bucket for which you would like to set a lifecycle policy. This will bring up the bucket info menu.

      2. Click on the S3 tab to open the S3 bucket settings.

        Open the Cyberduck bucket settings menu.

      3. Click on the checkbox labeled Delete files and select a time interval from the drop-down menu below it.

        Click on the "S3" tab and then check the box labeled "Delete files."

      This will enable the lifecycle policy and the objects within the bucket will be deleted after the designated time.

      Disable a Lifecycle Policy

      To disable a lifecycle policy, uncheck the box entitled Delete Files that you checked in the previous section.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link