Why Did My S3 Costs Go Up?

If you've experienced higher S3 costs and are trying to figure out why, the Versioning feature for your S3 bucket may have been activated. But there's no need to worry — you can manage it and take complete control of your S3 bucket and its versioning costs. The following article provides valuable insights into the costs associated with a versioned S3 bucket and how to manage those costs effectively. With this knowledge, you can optimize your S3 bucket and ensure you only pay for the required storage.

S3 Versioning is a valuable feature that allows you to keep multiple versions of the same object in one bucket. This ensures you always have access to previous versions of objects accidentally deleted or overwritten. With S3 Versioning, you can store multiple versions of an object in one bucket and restore any version you need.

For example, if you enable S3 Versioning for a bucket, any object you delete will not be permanently removed. Instead, Amazon S3 inserts a delete marker, which becomes the current object version. You can then restore the previous version if you need to.

Similarly, if you overwrite an object, Amazon S3 adds a new object version to the bucket. The previous version remains in the bucket and becomes a noncurrent version. You can restore the previous version at any time. To learn more, read about deleting object versions from a versioning-enabled bucket.

Deleting object versions from a versioning-enabled bucket

Enabling Versioning for your S3 bucket can happen for various reasons, but it's vital to note that once it's enabled, you can not turn it off. However, you can suspend it and find a sustainable way to manage your S3 bucket's Versioning. Don't let unforeseen S3 costs hold you back. You can take control of your S3 bucket and manage your versioning costs more efficiently.

If you have multiple buckets with millions of objects, you might feel overwhelmed and need to know what you already have. AWS has provided Storage Lens, which can let you know which buckets have versioned objects, how many, and what percentage of your objects are noncurrent. It will also show what storage tier the noncurrent objects are in. For more information, proceed to the following article on how to use AWS Storage Lens.

Assessing your storage activity and usage with Amazon S3 Storage Lens

The data provided will help you quickly understand your versioning costs and determine how to manage storage better by looking at the amount of storage for noncurrent versions and their tiers. The pricing is the same as for the current versioned copy. The following illustrates how noncurrent versions can build up quickly over time, adding to the cost.

You have an object that is 4GiB in size and gets updated five times daily. That means you would have five noncurrent versions daily and one current version for six. If we take the five-time daily changes and times them by an average of 30 days, we will get five changes daily x 30 average number of days in a month = 150 noncurrent version copies. This can add up quickly, and whatever storage tier the nonversioned objects are in, that is the price you are charged.

Another example is if you have a 1MB file and 200 copies (noncurrent versions), you pay for 200MB of storage.

Once you've identified the buckets that have Versioning enabled and which objects already have noncurrent versions, let's take a moment to consider the value of having copies of the object. It's important to ask yourself, "Do I need or want copies of the object?" If the answer is yes, then the question is how long you need to keep a copy and how many copies are required. These answers will help you determine the cost of maintaining the copies and the automation necessary to manage the object's transitions. So, take your time to decide what's best for you and your needs. Not all buckets are equal.

If you do not need versioning on a bucket, follow the process below to suspend Versioning, remove nonversioned copies, and delete markers that are not required. This will save costs and provide a pseudo-disabled versioned bucket.

NOTE: Before you proceed with the procedure below, once you delete the noncurrent files, all noncurrent copies of the object will be gone forever, and there's no way to recover them. So, please ensure you're in the right S3 bucket and consider whether you need the noncurrent versions before you delete anything. You can test first on a test bucket.

  1. In the AWS S3 Console, find the bucket on which you would like to suspend Versioning and select it.
  2. Go to the Properties tab.
  3. In the Properties tab, find the section regarding Versioning and select Edit.


Properties of S3 bucket

4. Select the radio button Suspend.

5. Select the box I acknowledge the outcomes of suspending Bucket Versioning

6. Click on Save changes

7. The Versioning on a bucket is now suspended.

  • Suspending Versioning stops any new versions of the objects from being created.
  • Suspending Versioning has no impact on existing objects in the bucket.
Bucket Versioning Suspend or Enable
Bucket Versioning Suspend or Enable

The next step is to remove the noncurrent versions and expired delete markers. One of the most common ways, especially with millions of files, is to use Lifecycle Configuration. We will use that in our example as recommended by AWS, but there are also other ways. See the following for options.

Options to delete millions of objects from a versioning-enabled AWS S3 bucket

Still, in the S3 bucket, select the Management tab.

In the Management window, there is a section called Lifecycle Rules. Click on the Create lifecycle rule.

S3 Management Lifecycle Rules
  1. In the Create lifecycle rule, fill in the following sections
    Lifecycle rule name: Use a name that describes what the rule does.
    For example DeleteNV-1days-DeleteMarker-IMU-7days
    In the rule's name, we say to delete nonversioned copies after one day, DeleteMarkers, and incomplete multipart uploads after seven days.
  2. For Choose a rule scope, select the radio button, "Apply to all objects in the bucket." A box will appear with a warning message. Read the warning message and select "I acknowledge that this rule will apply to all objects in the bucket." Later in the article, the option to Limit the scope of this rule using one or more filters is discussed.
  3. Under Lifecycle rule actions, select the boxes Permanently delete noncurrent versions of objects and Delete expired object delete markers or incomplete multipart uploads.
  4. Two new sections will appear below: Permanently delete noncurrent versions of objects and Delete expired object delete markers or incomplete multipart upload.
  5. Under Permanently delete noncurrent versions of objects, in the text boxes, Days after objects become noncurrent, enter 1. The number has to be greater than zero. This is one of the decisions discussed previously on how long to keep noncurrent objects. In the text box right of it, Number of newer version to retain—Optional, leave blank. Remember the question, "How many versions do I need to retain?". This is where you would set that number. It will always retain that number of copies or noncurrent versions.
  6. Next is the Delete expired object delete marker and Delete incomplete multipart uploads section. Select the boxes Delete expired object delete markers, and Delete incomplete multipart uploads. A new selection will appear called Number of days. In the text box, enter 7. This is how many days before deleting any incomplete multipart uploads. This should give time for an incomplete multipart upload to finish if it needs to recover.
Lifecycle rule configuration


Lifecycle Rule Actions

You may be curious about why we selected the option to delete incomplete multipart uploads. This is a simple and effective way to cut down on costs that can be set up automatically on your S3 buckets Lifecycle rule. Remember that you will be charged for any incomplete multipart upload parts based on the storage class specified when the parts were uploaded. You can again utilize the S3 Storage Lens to analyze incomplete multipart uploads. For more information on incomplete multipart uploads and analyzing them with Storage Lens, please refer to the following article.

Discovering and Deleting Incomplete Multipart Uploads to Lower Amazon S3 Costs | Amazon Web…

The following is further information from AWS regarding deleting noncurrent versions of objects.

A versioning-enabled bucket has one current version and zero or more noncurrent versions for each object. When you delete an object, note the following:

If you don’t specify a version ID in your delete request, Amazon S3 adds a delete marker instead of deleting the object. The current object version becomes noncurrent, and the delete marker becomes the current version.

If you specify a version ID in your delete request, Amazon S3 deletes the object version permanently (a delete marker is not created).

A delete marker with zero noncurrent versions is referred to as an expired object delete marker. (source).

Congratulations! You have successfully taken control of a versioned S3 bucket. Depending on the lifecycle configuration you have chosen for the use case, you should see a reduced cost after a week. However, it's important to note that Lifecycle Rules run asynchronously, so removing the noncurrent version objects may take some time. But don't worry; AWS won't charge you for the additional days it may take to complete the process.

You can enable versioning again now that a Lifecycle Management rule will handle noncurrent versions automatically. Alternatively, you can keep the versioning suspended if you don't want versions of the objects. Please note that any new objects placed in the S3 bucket while versioning is suspended will have a null ID marker and will not be eligible for versioning.

Sometimes, you need versioning enabled on an S3 bucket but only need Lifecycle Management on particular objects. Lifecycle Management allows you to apply the rule for specific prefixes. It is called "Limit the scope of this rule using one or more filters." Using filters, you can decide which noncurrent objects are subject to deletion. You can also use the filter feature to have different Lifecycle rules for other objects.

You can filter where the objects are stored. For example, if you have files in a logs folder, you can set the filter as logs/. The Lifecycle rule applies to logs/log.txt, logs/temp3.txt, and logs/test1.txt. The rule does not apply to an object in the root called example.jpg. You can also filter file extensions, tags, and the size of the objects. See the following article for more information regarding filtering and different examples.

Subscribe to updates, news and more.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related blogs

Connect With Us