Dealing with Spaces in S3 Files Stored within Folders in a Bucket

Files and folders within a bucket can create problems when you access them. In this article, we will see some code to deal with this problem. To list all files in a given bucket in S3:

def self.all(bucket_name)
  Aws.config[:s3] = {
            region: 'us-west-1',
            credentials: Aws::Credentials.new(
              Credential.amazon_access_key_id,
              Credential.amazon_secret_access_key),
            retry_limit: 0
          }

  s3 = Aws::S3::Client.new
  resp = s3.list_objects(bucket: bucket_name)
  resp.contents.each do |object|
    puts "#{object.key}"
  end
end

You can now run this in rails console to see all the files. If you have created folders inside your bucket, you need to modify the above program:

def self.all(bucket_name)
  Aws.config[:s3] = {'same as above'}

  s3 = Aws::S3::Resource.new
  # reference an existing bucket by name
  bucket = s3.bucket(bucket_name)

  # enumerate every object in a bucket
  bucket.objects(prefix: 'Audio Files  Book  1/').each do |obj|
    puts "#{obj.key}"
  end
end

In this case I have a folder called Audio Files Book 1/ within the bucket name. This prints:

Audio Files  Book  1/
Audio Files  Book  1/Audio 1-1.mp3
Audio Files  Book  1/Audio 1-10.mp3
Audio Files  Book  1/Audio 1-11.mp3
Audio Files  Book  1/Audio 1-12.mp3
... more like this omitted here

Now can use the S3 API to copy these files to another folder without spaces in the file names.

def self.copy
  configure
  source_bucket_name = 'your-source-s3-bucket-name'
  target_bucket_name = 'your-target-s3-bucket-name'
  source_key = 'Audio Files  Book  1/Audio 1-1.mp3'
  target_key = 'audio/Audio1-1.mp3'

  s3 = Aws::S3::Client.new
  arguments = {bucket: target_bucket_name, 
               copy_source: source_bucket_name + '/' + source_key, 
               key: target_key}
  s3.copy_object(arguments)

  puts "Copying file #{source_key} to #{target_key}."
end

The configure is a private method.

private

def self.configure
  Aws.config[:s3] = {region: 'us-west-1',
                       credentials: Aws::Credentials.new(Credential.amazon_access_key_id,
                                                                                    Credential.amazon_secret_access_key),
                       retry_limit: 0}
end

To move a list of files, we can iterate over the files and copy to the desired folder.

  def self.move_videos(bucket_name)
    # configure S3 credentials

    s3 = Aws::S3::Resource.new
    bucket = s3.bucket(bucket_name)
    bucket.objects.each do |object|
      key = object.key

      if key.start_with?('Video')
        clean_file_name = key.gsub(' ', '')
        copy(key, "video/#{clean_file_name}")
      end
    end
    puts 'Done'    
    true
  end

This code copies the file with a file name that does not contain any spaces. Spaces can cause problems when accessing the files. The versions of aws gems I am using in this project:

aws-eventstream (1.0.1)
aws-partitions (1.95.0)
aws-sdk-core (3.22.1)
aws-sdk-kms (1.6.0)
aws-sdk-s3 (1.16.0)
aws-ses (0.6.0)
aws-sigv4 (1.0.3)

You can download a file from S3 and save it on your local computer.

  def self.download_file
   #  configure S3 not shown

    s3 = Aws::S3::Resource.new

    bucket = s3.bucket('your-s3-bucket-name')

    bucket.objects.each do |object|
      key = object.key

      if key.start_with?('Video') && key.end_with?('avi')
        clean_file_name = key.gsub(' ', '')
        object.get(response_target: "./#{clean_file_name}")
        puts "Downloaded #{clean_file_name}"
      end
    end

    puts "Done"    
  end

In this case I am downloading only video files that has avi extension. Installed ffmpeg to convert .avi file to .mov so that it can be played on a browser. You can do:

brew install ffmpeg

When opening the converted .mov file using Quicktime, it had issues. If you open the .avi file using Quicktime, it converts to .mov and this has no problem.

Note

You need to be careful with the gem version because it can be confusing, since they use AWS or Aws and other nasty surprises that I encountered like these error messages:

NameError (uninitialized constant AWS::S3)
NoMethodError (undefined method `bucket' for #<Aws::S3::Client>)
ArgumentError (:bucket option must not contain a forward-slash (/))

I spent a few hours going over the AWS docs, source code of the gems and reading the tests to fix the issues. I hope this helps you a few hours of your time.

References

S3 File Move Github Source
S3 Docs
AWS S3 version 3 docs


Related Articles