Before starting playing the trick, you need to have Amzon EC2 API tools installed.

Then the steps are:

1. Shutdown the instance and detach the volume.
2. Build a snapshot for the detached volume and create a volume with a larger size based on the snapshot.
3. Attach the new volume to the instance and start it.
4. Login the instance and resize the root file system to fill the new EBS volume.

Commands for the last step are listed as below:

# ext3 root file system (more common):
sudo resize2fs /dev/sda1

# XFS root file system (less common):
sudo xfs_growfs /

xfs_growfs is supplied by xfsprogs, which can be installed via yum (or apt-get).

The installation is quite simple, you just need to download the source code, and run below command to compile it:

$ gcc -Wall -O3 -o vmtouch vmtouch.c

The usage examples can be seen at its homepage.

Its key features are:

• How much data of a file (or a directory) is currently in cache.
• Bring a file (or part of a file) into system memory.
• Ask the system to evict a specified file from memory, making room for other files.
• Daemonise and lock specified files into physical memory (Make sense?)

Want to have a try? Please click this link: http://hoytech.com/vmtouch/

Enjoy!

If it’s a normal working environment, rsync is a nice solution and its –delete parameter does what we expect. Then, on Amazon S3 is there a solution for us to keep two buckets synced smoothly?

Here we’ll show a working solution with s3cmd. Other Amazon S3 clients may have similar features, but we will not explain them here.
By using the bucket sync feature, you need to update to version 1.0.0, prior to this version, bucket syncing is not supported, and you may face this error:

root@demo:~# s3cmd sync --delete-removed s3://admon/brands s3://admon-dev/
ERROR: Parameter problem: Invalid source/destination: 's3://admon/brands' 's3://admon-dev/'

When s3cmd updated, the command is as simple as what rsync does:

s3cmd sync --delete-removed --acl-public s3://admon/brands s3://admon-dev/

The –delete-removed option has the same meaning as rsync’s –delete parameter, and –acl-public is an extra parameter which is used to set your files publicly accessible. If you want to keep your pictures private, you can just remove –acl-public.

Note: If you have a extremly large number of pictures (like more than 100k pictures), this command takes a significate amount of memory. As for my test, it takes 2.7GB resident memory when the bucket contains 500k objects ( pictures and directories).

Before moving ahead, you need a S3 file manager. There are a couple of good S3 file managers that can be used to manage S3 files like S3Cmd, S3Hub, S3Fox and CloudBerry Explorer (only available on Windows). It’s suggested to use S3Hub (or Cloudberry Explorer) as it implements the features that we mentioned here.

Tip #1: Are S3 files misused by others? Enable the S3 access logging.

Amazon S3′s bandwidth rates are inexpensive and you pay for what you use. The problem is that if other websites are linking to your S3 hosted content (like images, MP3s, Flash videos, etc.), you’ll also have to pay for bandwidth that used by these sites.

Unlike Apache web servers where you can easily prevent referred links from other websites, Amazon S3 doesn’t offer such mechanism but what you can do is to enable logging for your S3 buckets. Amazon will log all client requests in logs and you can parse them later. If you noticed a file is heavily referred to by other websites, you may rename the file on S3 or just delete it if you’re ok with that.

By implementing it, you create a new S3 bucket, then right-click the bucket name, choose “Logging” then specify the bucket to store the log files.

Tip #2: Set Expiry Headers for Static Files

The cost for GET requests is small (just 1¢ per 10,000 requests), they can quickly add-up if you have have a popular site or if your website design uses too many images. It is important that you add an an Expires or a Cache-Control HTTP Header for static content on your site like images, css/js files.

The gist is that all web browsers store objects in their cache and this Expires header in the HTTP response tells the browser how long that object should stay in the cache. So if you can set the Expires date sometime in future and client browser won’t request the object again.

To set an expires header, right click the S3 object properties, choose HTTP headers and add a new header. Call it “Expires” and set an expiration date like “Fri, Apr 23 2021 10:18:36 GMT”.

Other than saving money, your site will also load relatively faster because the visitor’s browser will not establish as much connections/queries as it was.

If you are worried about setting Cache headers for JavaScript and CSS files as they may change frequently (especially when you are in the midst of a site re-design), just append a version number after the file name like how commonly rubyonrails application does.

When the main.png on S3 updated, you just need to change the version number and visitors browser will make a fresh GET request to Amazon S3 for the latest version of the S3 file.

Tip #3: Use Amazon S3 with a reasonable Domain Name

When a new bucket on Amazon S3 is created, setting the file access to public and Amazon will provide you with a public URL which is something like:

http://bucketname.s3.amazonaws.com/filename

http://s3.amazonaws.com/bucketname/filename

It’s suggested to use the first URL since you can do the Tip #4 mentioned below.

Tip #4: Block search engine crawlers on your Amazon S3

To prevent robots from indexing files stored in your Amazon S3 buckets, create a robots.txt file in the root directory with the content:

User-agent: *
Disallow: /

Make sure that you update the ACL (or access permissions) to public otherwise spiders won’t find your robots.txt file.

Tip #5: Other Optimizations on File transferring
Torrent Delivery for large files

If you are planning to distribute some large files of the web (like a software installer or some database dump) via Amazon S3, it makes sense to use BitTorrent with S3 so that you don’t necessarily have to pay for all the download bandwidth. The starting point for a BitTorrent download is a .torrent file and you can quickly create a .torrent file to any S3 object by adding “?torrent” to the original web URL like this:

http://bucketname.s3.amazonaws.com/software-installer.zip?torrent

Create Time Limited Links

By default all public files in your S3 account are available for download forever until you delete the file or change the permissions. However, if you are running some sort of contest on your site where you are giving away a PDF ebook or some MP3 ringtone to your visitors, it doesn’t make sense to have those file live on your S3 server beyond the duration of the contest. You should therefore consider creating “signed URLs” for such temporary S3 files – these are time limited URLs that are valid for a specific time period and expire afterwards (or return 404s).

Right click a file in the S3 bucket, choose Web URL and then set a Expiry Time. Click Generate to created a “signed URL”.

When you have a running EC2 instance, you can build your production environment there, install whatever the packages you need, tweak system-wide configurations to meet your needs… Then you might wanna build your own AMI image to simplify future deployment.

Here we’ll show a quick example on how to build an EBS-backed AMI. This AMI is generated from one of your running instance.

First: Create an EBS volume
This volume will be use as the root partition for new instances, and you can specify whatever size you need. It’s suggested to be at least 3GB. When this volume is created, attach it to the running instance as device /dev/sdc. We assume that the new volume id is vol-8ee49ce6, and the running instance’s kernel id is aki-6603f70f. We’ll use these two ID in the final step.
When the new volume attached, you can use these command to build the file-system on it, then mount it as /mnt/ebs

# mkfs.ext3 /dev/sdc
# mkdir /mnt/ebs
# mount /dev/sdc /mnt/ebs

Step 2: Sync the running system to the new volume
If there’s any service running on the instance, it’s suggested to shut them down. It’ll save you hassles later. Then, use rsync to copy everything over to the EBS volume:

# rsync -ax --exclude=/mnt/* --delete --progress / /mnt/ebs

Before copying the system, make sure your instance is tweaked well, like you might need to update /etc/fstab to remove some unused devices.
You may need to create some device files on the new EBS volume. If console, zero or null don’t exist in /mnt/ebs/dev, create them using some or all of these:

# cp -prf /dev/console /mnt/ebs/dev
# cp -prf /dev/zero /mnt/ebs/dev
# cp -prf /dev/null /mnt/ebs/dev

You can also use MAKEDEV to create device, here we use cp for simple.

Step 3: Make your AMI
Now you can make a snapshot of the EBS volume via Amazon EC2 command line tools. This is the basis of the AMI you’ll be creating. Whatever you copied to the EBS volume in step 2 will be there — user accounts, database data, etc.

# ec2-create-snapshot vol-8ee49ce6 -d "Active AMI for Production Server"

That’ll give you a snapshot-id back. You then need to wait for the snapshot to finish. Keep running ec2-describe-snapshots until it says it’s “completed“.
Finally, you can register the snapshot as an AMI:

# ec2-register --snapshot snap-a8f327c4 --description "Active AMI for Production Server" --name "web-server-image" ––kernel aki-6603f70f --architecture x86_64 --root-device-name /dev/sda1 -b /dev/sdb=ephemeral0

The above command will return an AMI-ID, you can use this AMI-ID to launch an instance and you’ll find it pretty much exactly where you left your original instance. Note that the first time load may take some time like 10 mins and become faster then.

The answer is Yes, sure. Since linux-2.6.16, there’s a tunable file in /proc can be used, it’s /proc/sys/vm/drop_caches.
Writing to this file will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.

To free pagecache:

echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation, and dirty objects are not freeable, the user should run “sync” first in order to make sure all cached objects are freed.

Please note that this post does not cover the topics on password strength. you can use some online tools to check password strength.

GMail Password Strength Check

You can use the following four commands directly in most of Linux distributions with default install.

joseph@test:~$ < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c8
3saMDFXB

I personally prefer to use this command. The other commands work great too, check below:

cat /dev/urandom | tr -cd 'a-f0-9' | head -c 16
dd if=/dev/random count=10 bs=1 | hexdump  | cut -d   -f 2-| head -n 1 | tr -d " "

uuencode which is supplied by sharutils is needed by this command (Sharutils is not installed by default):

head -c 12 /dev/random | uuencode -m - | tail -n 2 | head -n 1

By using the following command, you need to get mkpasswd installed. It’s man page is available here. It can be run like this:

mkpasswd -l 16 -s 0 -C 0|sed 's/[0ol1]/f/g'

Finally, if you have pwgen installed, you can also generate a random password in linux command line like this:

pwgen -ycn 16 1

PWGen supports Windows platform as well, you just need to install pwgen-win instead, it has a nice GUI.

1, rcconf - http://packages.debian.org/lenny/rcconf

This tool configures system services in connection with system runlevels. It turns on/off services using the scripts in /etc/init.d/. Rcconf works with System-V style runlevel configuration. It is a Text User Interface (TUI) frontend to the update-rc.d command.

To start rcconf, login as root and type the command “rcconf“, then select the service that you would like to enable or disable, and press Tab to apply a change.

A similar command is sysv-rc-conf, you can make more specified choices in this TUI based configuration tool.

2, sysv-rc – http://packages.debian.org/lenny/sysv-rc

Sysv-rc provides a terminal interface for managing “/etc/rcX.d/” symlinks. With the help of update-rc.d, you can allow turning services on or off simply. Actually it’s much what chkconfig does in Redhat/CentOS, which means you can edit startup scripts for any runlevel.

By default sysv-rc is installed even in a minimal installation. Otherwise you need to run “apt-get install sysv-rc” to install it. Here’re some common usage examples for update-rc.d:

To remove the service SERVICE-NAME:

update-rc.d -f SERVICE-NAME remove

To add the service SERVICE-NAME:

update-rc.d SERVICE-NAME defaults

update-rc.d SERVICE-NAME defaults 29

The argument 29 is optional, which ensures that SERVICE-NAME is called after all scripts whose number is less than 29 have completed, and before all scripts whose number is 30 or greater.

To start the script SERVICE-NAME in runlevels 2345 and stop in 456, run (as root):

update-rc.d SERVICE-NAME start 2 3 4 5 . stop 0 1 6 .

update-rc.d SERVICE-NAME start 30 2 3 4 5 . stop 70 0 1 6 .

You can find more details of this command by running “man 8 update-rc.d”.

As mentioned at Facebook.com:

We built Flashcache to help us scale InnoDB/MySQL, but it was designed as a generic caching module that can be used with any application built on top of any block device. For InnoDB, when the working set does not fit in the InnoDB buffer pool, read latency is significantly improved due to caching more of the working set in faster media, such as SSD’s. We also improve write performance by first caching writes in SSD’s and lazily flushing the data back to disk.

It looks like its based on the dm-cache work. It’s nice to see that moving again.

Read more details about Flashcache here at Facebook’s announcing page, and Flashcache’s source code is available at GitHub.com.

VENet consists of two Ethernet devices — the one in physical server and another one in virtualized guest. These devices are connected to each other, so if a packet goes into one device it will come out from the other device.

In this post, we’ll share some tips on how to enable venet. The content is mainly from OpenVZ’s official guide.

The commands that we used are listed below with explanations.

Firstly, assuming that we’re in the physical server, We need to add a new device named eth0

vzctl set 150 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

The second MAC address is from the mother side, and the first MAC address is self-generated. Click here if you’re not sure how to generate a MAC address.

Then enable forwarding and ARP proxy, and apply some changes in route table:

# cat veth150.sh
ifconfig veth150.0 0
echo 1 > /proc/sys/net/ipv4/conf/veth150.0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/veth150.0/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
ip route add 192.168.201.150 dev veth150.0

Warning: Before making these changes to your system, you’re suggested to verify what exactly you’re doing. For example, proxy_arp is problematic in a mixed network.

Secondly, in the VPS we need to config the new device eth0 like this:

/sbin/ifconfig venet0:0 0
/sbin/ifconfig eth0 0
/sbin/ip addr add 192.168.201.150 dev eth0
/sbin/ip route add default dev eth0

The IP address 192.168.201.150 can be in a different network from the physical side, you just need to make sure they are in the same VLAN.

When it’s tested OK, don’t forget to modify your network configuration files in /etc/sysconfig/network-scripts/ for a permanent change.