S3cmd: Amazon S3 command-line interface

Amazon web servicesS3cmd is command line tool for uploading, downloading and managing data in Amazon S3. It’s written in Python, and ideal for scripts, automated backups triggered from cron, etc. S3cmd is an open source project and is free for both commercial and private use. You will only have to pay Amazon for using their storage. The installation is quite simple. If you’re running debian, you just can install it simply via apt-get. Before to use s3cmd, we need to config it. This can be done via this command,

~/demo$ s3cmd --configure

It asks for two keys – access key and secret key, and you can optionally enter a GPG encryption key that will be used for encrypting your files before sending them to Amazon. When the configuration finishes, you can run with –help to get a list of general operations.

For advanced tutorials, here’s a quick list.

~/demo$ s3cmd put --acl-public --guess-mime-type storage.jpg s3://admon-test/storage.jpg
File 'storage.jpg' stored as s3://admon-test/storage.jpg (33045 bytes)

The file is uploaded to Amazon S3 bucket can either be private, that is readable only by you, possessor of the access and secret keys, or public, readable by anyone. Each file uploaded as public is not only accessible using s3cmd but also has a HTTP address, URL, that can be used just like any other URL and accessed for instance by web browsers.

To upload a directory and keep its name on the remote side specify the source without the trailing slash:

~/demo$ s3cmd put -r dir1 s3://admon-demo/some/path/
dir1/file1-1.txt -> s3://admon-demo/some/path/dir1/file1-1.txt  [1 of 2]
dir1/file1-2.txt -> s3://admon-demo/some/path/dir1/file1-2.txt  [2 of 2]

On the other hand to upload just the contents, specify the directory it with a trailing slash:

~/demo$ s3cmd put -r dir1/ s3://admon-demo/some/path/
dir1/file1-1.txt -> s3://admon-demo/some/path/file1-1.txt  [1 of 2]
dir1/file1-2.txt -> s3://s3tools-demo/some/path/file1-2.txt  [2 of 2]

sync two directory. It works like what rsync do, generating a list of file names, checksums from both local and remote side, then making a decision which of them will be processed.

~/demo$ s3cmd sync  ./  s3://admon-demo/some/path/
dir2/file2-1.log -> s3://admon-demo/some/path/dir2/file2-1.log  [1 of 2]
dir2/file2-2.txt -> s3://admon-demo/some/path/dir2/file2-2.txt  [2 of 2]
...

Here’s a more advanced example,

s3cmd sync --dry-run --exclude '*.txt' --include 'directory/*' . s3://admon-demo/demo/

How can I use s3cmd to setup ACLs for my S3 files?

S3cmd partially (as in version 0.9.9.91) implements the access control feature from S3, you can only enable/disable whether an object is accessible by public. S3tools have plans to implement more features.

~/demo$ s3cmd setacl --acl-public --recursive s3://admon-test/backup
	s3://admon-test/backup/file1.txt: ACL set to Public  [1 of 3]
	s3://admon-test/backup/dir2/file2.jpg: ACL set to Public  [2 of 3]
        ...

If you need to set read/write based permission control for different types of Amazon S3 accounts, it’s suggested to use the graphic tools like s3Fox.

Share Button

2 thoughts on “S3cmd: Amazon S3 command-line interface

Leave a comment

Your email address will not be published. Required fields are marked *