Home  >  Blog  >   OpenStack

Uploading Large Objects In OpenStack

Rating: 5
  
 
3150
  1. Share:
OpenStack Articles

When using cloud storage, such as OpenStack Swift, at some point you might face a situation when you have to upload relatively large files (i.e. a couple of gigabytes or more). For a number of reasons, this brings in difficulties for both client and server. From a client's point of view, uploading such a large file is quite cumbersome, especially if the connection is not very good. From a storage point of view, handling very large objects is also not trivial.

For that reason, Object Storage (swift) uses segmentation to support the upload of large objects. Using segmentation, uploading a single object is virtually unlimited. The segmentation process works by fragmenting the object, and automatically creating a file that sends the segments together as a single object. This option offers greater upload speed with the possibility of parallel uploads.
Individual objects up to 5 GB in size can be uploaded to OpenStack Object Storage. However, by splitting the objects into segments, the download size of a single object is virtually unlimited. Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object. By splitting objects into smaller chunks, you also gain efficiency by allowing parallel uploads.

To gain in-depth knowledge and be on par with practical experience, then explore the OpenStack Training Course.
  • Log in to a computer or server that has the swift client package installed.
  • Carry out the following steps to upload large objects, split into smaller segments:

MindMajix YouTube Channel

How to Upload Large objects in Open Stack

Uploading objects

You will see output similar to the following screenshot showing the status of each upload:

Output
 

Related Article: Open Stack Tutorial

Large objects in Open Stack Object Storage

OpenStack Object Storage is very good at storing and retrieving large objects. To efficiently do this in our OpenStack Object Storage environment, we have the ability to split large objects into smaller objects with OpenStack Object Storage, maintaining this relationship between the segments and the objects that appear as a single file. This allows us to upload large objects in parallel, rather than stream a single large file. To achieve this, we use the following syntax:

How it works
                                                                                                           

 

Join our newsletter
inbox

Stay updated with our newsletter, packed with Tutorials, Interview Questions, How-to's, Tips & Tricks, Latest Trends & Updates, and more ➤ Straight to your inbox!

Course Schedule
NameDates
OpenStack TrainingMar 23 to Apr 07View Details
OpenStack TrainingMar 26 to Apr 10View Details
OpenStack TrainingMar 30 to Apr 14View Details
OpenStack TrainingApr 02 to Apr 17View Details
Last updated: 03 Apr 2023
About Author

Ravindra Savaram is a Technical Lead at Mindmajix.com. His passion lies in writing articles on the most popular IT platforms including Machine learning, DevOps, Data Science, Artificial Intelligence, RPA, Deep Learning, and so on. You can stay up to date on all these technologies by following him on LinkedIn and Twitter.

read more
Recommended Courses

1 / 15