The Citrix NetScaler implementation of GSLB is DNS-based. DNS can be thought of as the Internet’s ‘Phone Book’, telling computers where different services are located. DNS-based Global Server Load Balancers are by far the most common type of GSLB device.

GSLB directs DNS requests to the best-performing GSLB site in a distributed Internet environment. When a client sends a DNS request, the system determines the best-performing site and returns its IP to the client. In the process of ascertaining the best- performing site, the system performs these intelligent decisions:

• Directs client requests to the geographically closest GSLB site (geographic and network proximity-based traffic redirection)
• Directs client requests to surviving data centers when an outage occurs
• Directs client requests to alternate data centers, when a pre-defined traffic load limit is reached
• Directs client requests to be distributed among multiple data centers (assigns each user to the GSLB site with lowest latency)

The system performs these intelligent decisions using the Metric Exchange Protocol (MEP), GSLB policies, and GSLB methods supported by the system.

GSLB policies direct the traffic to a pre-defined target site. GSLB methods are algorithms that control how the system load-balances client requests across distributed data centers. The system provides support for creating policies for distributing or redirecting client request.

Multiple sites exchange metrics with each other using the MEP. The system uses this protocol to exchange load, network, and persistence information between GSLB sites. The system also uses this information to perform load balancing between GSLB sites.

GSLB Entity Model

A typical GSLB deployment contains the entities described in the following figure. 

To configure GSLB, you must configure a GSLB site. As shown in the figure, a GSLB site is the logical collection of GSLB vserver, GSLB service, LB vserver, service, domain, and ADNS service. It is the central entity in a GSLB deployment, and is represented by a name and an IP address.

To create a GSLB site, you must configure load balancing on the system. You must create GSLB vservers and GSLB services for each site. You must bind GSLB services to GSLB vservers. You must then create an ADNS service that provides the IP address of the best performing site to the client’s request.

A GSLB vserver is an entity that performs load balancing for the domains bound to it by returning the IP address of the best GSLB service. A GSLB service is a representation of the load balancing/content switching vserver. An LB vserver load balances incoming traffic by identifying the best server, then directs traffic to the corresponding service. It can also load-balance external DNS name servers. Services are entities that represent the servers. The domain is the domain name for which the system is the authoritative DNS server. By creating an ADNS service, the system can be configured as an authoritative DNS server.

Note: This is a modified version from the article How+GSLB+Works.

When running in promiscuous mode, all traffic the network card receives can be read. This configuration is useful for us to do network monitoring, like for a network intrusion detection system.

How can I config my network card in promiscuous mode?

You can do this easily by one command. It works on both RedHat and Debian based distributions. Below is an example:

root@db1:~# ifconfig eth1 promisc
[2685638.719679] device eth1 entered promiscuous mode
root@db1:~# ifconfig eth1 -promisc
root@db1:~# dmesg | tail -1
[2685655.668037] device eth1 left promiscuous mode

Then, how can we setup the promiscuous mode in configuration files, so that it takes effect when system boots? As the configuration varies by distribution, here we raise two examples.

Setup promiscuous mode on Redhat / CentOS

To configure a network card in promiscuous mode, you need to put the line PROMISC=yes in its configuration file /etc/sysconfig/network-scripts/ifcfg-ethX.

BOOTPROTO=static
DEVICE=ethX
ONBOOT=yes
TYPE=Ethernet
PROMISC=yes
USERCTL=no

Don’t forget to replace ethX to the right device you are using.

Setup promiscuous mode on Ubuntu / Debian

Below is part of an example file of /etc/network/interface:

auto eth0
iface eth0 inet manual
up ifconfig $IFACE 192.168.1.100 up
up ip link set $IFACE promisc on
down ip link set $IFACE promisc off
down ifconfig $IFACE down

Again, for any issues, please leave us a comment, or raise a thread at the support forum.

VENet consists of two Ethernet devices — the one in physical server and another one in virtualized guest. These devices are connected to each other, so if a packet goes into one device it will come out from the other device.

In this post, we’ll share some tips on how to enable venet. The content is mainly from OpenVZ’s official guide.

The commands that we used are listed below with explanations.

Firstly, assuming that we’re in the physical server, We need to add a new device named eth0

vzctl set 150 --netif_add eth0,00:12:34:56:78:9A,veth101.0,00:12:34:56:78:9B --save

The second MAC address is from the mother side, and the first MAC address is self-generated. Click here if you’re not sure how to generate a MAC address.

Then enable forwarding and ARP proxy, and apply some changes in route table:

# cat veth150.sh
ifconfig veth150.0 0
echo 1 > /proc/sys/net/ipv4/conf/veth150.0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/veth150.0/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
ip route add 192.168.201.150 dev veth150.0

Warning: Before making these changes to your system, you’re suggested to verify what exactly you’re doing. For example, proxy_arp is problematic in a mixed network.

Secondly, in the VPS we need to config the new device eth0 like this:

/sbin/ifconfig venet0:0 0
/sbin/ifconfig eth0 0
/sbin/ip addr add 192.168.201.150 dev eth0
/sbin/ip route add default dev eth0

The IP address 192.168.201.150 can be in a different network from the physical side, you just need to make sure they are in the same VLAN.

When it’s tested OK, don’t forget to modify your network configuration files in /etc/sysconfig/network-scripts/ for a permanent change.

Firewalls are a required component in commercial and residential computer networks. For many installations, the firewall is a single point of failure between client systems and external resources. It can also become a liability when hardware or applications fail, leaving potential customers unable to reach your servers.

A properly designed and executed failover configuration for your primary firewall will address many of these concerns. This article introduces a proven method for installing redundant stateful firewalls using native OpenBSD features.

The OpenBSD project is known for creating a leading secure Unix-like operating system. They have always emphasized software robustness and security, while ensuring their code remains free for all purposes under the BSD license. A number of exciting features have been introduced to OpenBSD due to licensing disagreements. Many BSD users are familiar with the rift between Darren Reed, (the creator of IPFilter) and the OpenBSD developers. A change in the IPFilter license resulted in the rapid development of the OpenBSD PF firewall software. Not only is PF a competitor to expensive proprietary offerings, it is so successful that it has been ported to both FreeBSD and NetBSD distributions.

Within the past two years, OpenBSD recognized the need to support failover between OpenBSD firewalls. The pfsync protocol was completed, which sends state change messages via multicast over the pfsync interface. Using a secure connection (a crossover cable between systems is suggested), pfsync will notify other OpenBSD firewalls of changes to the local state table. If other firewalls are listening for the pfsync packets, they will update their own state tables with these announcements. This feature allows sessions to failover gracefully without losing connectivity or raising alerts in the firewall, providing the basic features required for stateful redundancy. However, the ability to dynamically failover to the stateful partners was still unavailable.

The Birth of CARP

The Virtual Router Redundancy Protocol (VRRP) eliminates the single point of failure in a static network by assigning a virtual gateway between multiple physical routers. This allows two or more routers to cooperate as a dynamic gateway; one will perform as the “master”, while the other system waits as a “backup”. If the master becomes unavailable, the backup will begin advertising itself as the master, allowing traffic to continue uninterrupted over the new physical path. Unfortunately, although VRRP is an IETF-standard protocol, it is also encumbered by a patent held by its author, Cisco Systems, Inc. They claim to have no intention of asserting patent claims against anyone implementing VRRP, but publicly reserve the right to assert patent claims defensively. OpenBSD needed this functionality to support failover between hosts, but the looming patent issue made VRRP a poor choice.

Based on their dedication to free software, the OpenBSD team went to work on creating a patent-free replacement for VRRP. This was released in the form of the Common Address Redundancy Protocol (CARP) in late 2003. CARP operates at the data-link and network OSI layers, using a virtual MAC and one or more virtual IP addresses. The master router of the CARP group responds to ARP requests for the virtual MAC with the shared IP address, allowing switches to quickly determine to which interface to forward traffic. CARP supports IPv4 and IPv6, load-balancing across the shared group, master preemption, and cryptographic hashing of the data-link announcements. Thanks to PF, pfsync, and CARP, users are now able to deploy truly redundant firewalls using free software and commodity hardware.

Installing OpenBSD is beyond the scope of this article, but should be familiar to NetBSD (and perhaps even Debian Linux) users. The media is available either via CD-ROM purchased from the OpenBSD store, or you can install via FTP by downloading one of the boot images. There are no CD-ROM ISO images available for download; CD-ROM and other project merchandise sales are used to support the ongoing development. The installation process usually takes no more than 10-20 minutes to complete, depending on media access and disk speed.

The scenarios presented in this article portray a typical dual-homed connection that you might encounter at any business or residence. Under most circumstances, it is suggested to incorporate a demilitarized-zone (“DMZ”) network to segregate inbound traffic to your public servers. This helps keep unwanted intruders out of your LAN. We will utilize a third interface for this example, but it will only carry pfsync notifications. Adding a DMZ is as simple as creating an additional physical and CARP interface for the DMZ.

Basic Configuration

Both systems should be configured to use unique (not shared between CARP group members) IP addresses for the physical interfaces. As of the time of this writing, code has been imported to allow CARP devices to operate without the need for a network interface. However, this code is still under heavy testing at the time of writing and should not be considered production-ready until the OpenBSD 3.7 release (May 2005).

Figure 1 shows that each firewall has a publicly routable IPv4 address for fxp0, a private RFC1918 address on fxp1, and another private address on fxp2. Each of the routing interfaces can be configured to use multiple CARP interfaces, although we will only be creating a single CARP interface on fxp1 to operate as our LAN gateway. The network settings for each physical device are stored in the /etc/hostname.fxp* files; each CARP interface has its settings stored in the corresponding /etc/hostname.carp* files. Media options are considered optional and will not be shown in the examples below. This will result in the connection duplex auto-negotiating at 100baseTX.

Each CARP interface must be configured with a virtual host ID (vhid) and virtual host IP address. The vhid must be unique among CARP interfaces on the same network segment. The advbase and advskew parameters are optional and are used to control how frequently a master host sends advertisements. A custom advskew setting will result in fewer advertisements, forcing the host into the backup role. The pass parameter is recommended as it is used to authenticate CARP advertisements between group members. The commands to manually enable the CARP interfaces are:

test1# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 pass foo
test1# ifconfig carp1 10.0.0.1 netmask 255.255.255.0 vhid 1 pass bar

test2# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 pass foo
test2# ifconfig carp1 10.0.0.1 netmask 255.255.255.0 vhid 1 pass bar

To enable the pfsync interface, we only need to pass ifconfig the “up syncif” keywords and the associated interface. However, in those cases where a dedicated pair of interfaces is not available for crossover connectivity, the syncpeer keyword can be used to designate a peer network address. If this is the desired effect, all pfsync traffic should be encrypted across enc0, the OpenBSD IPSec virtual interface. For our purposes, we will simply rely on passing unencrypted messages across the dedicated crossover between fxp2 on each firewall:

test1# ifconfig pfsync0 syncif fxp2

test2# ifconfig pfsync0 syncif fxp2

In situations where a preferred master is wanted, preemption can be enabled. We need to raise the advskew for the intended backup host. Once that is complete, both systems must enable preemption via the net.inet.carp.preempt sysctl variable. The sysctl changes must be stored permanently in /etc/sysctl.conf, and the advskew updates should be made to the hostname.carp* files on the backup host:

test2# ifconfig carp0 advskew 100
test2# ifconfig carp1 advskew 100
test2# sysctl -w net.inet.carp.preempt=1
net.inet.carp.preempt 0 -> 1

test1# sysctl -w net.inet.carp.preempt=1
net.inet.carp.preempt 0 -> 1

Listing 1 reveals the basic PF ruleset that will allow our firewall pair to block unwanted traffic. This example only allows outbound Network Address Translation (NAT) connections bound to the external interface, as well as connections initiated from the firewall itself. We must also allow pfsync traffic on fxp2 and CARP traffic on fxp0 and fxp1. Once the pf.conf has been saved, the new configuration should be tested for syntax errors. A quiet return prompt indicates a successful ruleset parsing:

test1# pfctl -nf /etc/pf.conf
test1#

test2# pfctl -nf /etc/pf.conf
test2#

With no errors reported, we can enable the ruleset:

test1# pfctl -f /etc/pf.conf

test2# pfctl -f /etc/pf.conf

Please note that all of the aforementioned network settings will need to be preserved in static configuration files. The output of these files can be seen in Listing 2

.Testing the Configuration

Running tcpdump on each CARP-enabled physical interface should reveal the master host multicasting its CARP advertisements. A sample capture from the LAN segment is shown below. Note that we want to look for packets matching protocol number 112. Although this is recognized as VRRP on many systems, OpenBSD has updated the /etc/protocols file to reflect this as CARP traffic:

test1# tcpdump -nvi fxp0 -c3 proto 112
tcpdump: listening on fxp0
20:32:55.110102 carp 10.0.0.2 > 224.0.0.18: CARPv2-advertise 36:
  vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] (ttl 255, id 15586)
20:32:56.120098 carp 10.0.0.2 > 224.0.0.18: CARPv2-advertise 36:
  vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] (ttl 255, id 8283)
20:32:57.130095 carp 10.0.0.2 > 224.0.0.18: CARPv2-advertise 36:
  vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] (ttl 255, id 18372)

The ifconfig command will convey the state of our physical, CARP, and pfsync interfaces. Passing the -A parameter to ifconfig will also show the state of all address aliases for each of our interfaces:

test1# ifconfig -A
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:d0:b7:bf:c6:95
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 66.77.24.2 netmask 0xffffff00 broadcast 66.77.24.255
        inet6 fe80::202:b3ff:fe0a:ce04%fxp0 prefixlen 64 scopeid 0x1
fxp1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:02:b3:0a:d1:28
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::202:b3ff:fe16:a957%fxp1 prefixlen 64 scopeid 0x2
fxp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:c0:4f:46:8d:ec
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.255.255.2 netmask 0xffffff00 broadcast 10.255.255.255
        inet6 fe80::2c0:4fff:fe46:9448%fxp2 prefixlen 64 scopeid 0x3
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=41<UP,RUNNING> mtu 1348
        pfsync: syncif: fxp2 syncpeer: 224.0.0.240 maxupd: 128
enc0: flags=0<> mtu 1536
carp0: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 66.77.24.5 netmask 0xffffff00
carp1: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 10.0.0.1 netmask 0xffffff00

The output clearly shows that both carp0 and carp1 on this system are serving as master of the group. Since preemption is enabled, this system will always retain the master role while it is able to advertise itself as available. If a designated “backup” firewall has accepted the master role, it will immediately relinquish that responsibility upon receiving advertisements from the system with the lower advskew. This behavior can also be monitored using the tcpdump techniques revealed earlier. Tcpdump can also be used to observe the pfsync state messages:

test1# tcpdump -nvvettti fxp2
Jan 31 21:18:22.135400 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 262:
  10.255.255.2 > 224.0.0.240: PFSYNCv2 count 1: INS ST:
 (DF) [tos 0x10] (ttl 255, id 58487)
Jan 31 21:18:23.130035 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 302:
  10.255.255.2 > 224.0.0.240: PFSYNCv2 count 3: UPD ST COMP:
 (DF) [tos 0x10] (ttl 255, id 64429)
Jan 31 21:18:25.940527 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 70:
  10.255.255.2 > 224.0.0.240: PFSYNCv2 count 2: DEL ST COMP:
        id: 41f3a32500007f5b creatorid: d0491513
        id: 41f3a32500007fe1 creatorid: d0491513
 (DF) [tos 0x10] (ttl 255, id 59638)
^C

All LAN workstations should be configured to use the internal CARP address as their network gateway. Then if the master firewall becomes unavailable, any existing connections will be immediately resumed by the next possible failover member. Simple tests — such as Web browsing and ping — should be performed to verify that clients are able to traverse the gateway.

Once basic connectivity and routing are confirmed, more advanced tests should be initiated to confirm stateful failover capabilities. SCP is an excellent tool for this test. Make sure to use a sufficiently large test file such that it will still be transferring when you unplug the cable. Immediately following the physical disconnect, you may experience a brief 1-2 second delay as the failover occurs. The session should abruptly resume, revealing that the connection has been preserved. After you plug the cable back in, traffic should immediately “bounce” to the preemptive master and continue unassumingly. By running pftop from the ports collection, you can watch the traffic “move” from one machine to the other.

Advanced Configuration

The next design introduces a method for load balancing connections between the redundant pair and a pool of internal servers. To distribute the load across both firewalls, a second CARP interface must be created on each firewall’s external and internal interfaces. Each firewall will serve as master and one as backup for each CARP interface. The net.inet.carp.arpbalance sysctl variable must also be enabled. An address alias is also being added to support additional services:

test1# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 pass foo
test1# ifconfig carp0 alias 66.77.24.10 netmask 255.255.255.0 vhid 1 pass foo
test1# ifconfig carp1 66.77.24.5 netmask 255.255.255.0 vhid 2 advskew
  100 pass foo
test1# ifconfig carp1 alias 66.77.24.10 netmask 255.255.255.0 vhid 2
  advskew 100 pass foo
test1# ifconfig carp2 10.0.0.1 netmask 255.255.255.0 vhid 1 pass bar
test1# ifconfig carp3 10.0.0.1 netmask 255.255.255.0 vhid 2 advskew 100
  pass bar
test1# sysctl -w net.inet.carp.arpbalance=1
net.inet.carp.arpbalance: 0 -> 1

test2# ifconfig carp0 66.77.24.5 netmask 255.255.255.0 vhid 1 advskew
  100 pass foo
test2# ifconfig carp0 alias 66.77.24.10 netmask 255.255.255.0 vhid
  1 advskew 100 pass foo
test2# ifconfig carp1 66.77.24.5 netmask 255.255.255.0 vhid 2 pass foo
test2# ifconfig carp1 alias 66.77.24.10 netmask 255.255.255.0 vhid 2 pass foo
test2# ifconfig carp2 10.0.0.1 netmask 255.255.255.0 vhid 1 advskew
  100 pass bar
test2# ifconfig carp3 10.0.0.1 netmask 255.255.255.0 vhid 2 pass bar
test2# sysctl -w net.inet.carp.arpbalance=1
net.inet.carp.arpbalance: 0 -> 1

Reviewing the output of ifconfig, we are able to confirm the presence of the new interfaces and aliases:

test1# ifconfig -A
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:d0:b7:bf:c6:95
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 66.77.24.2 netmask 0xffffff00 broadcast 66.77.24.255
        inet6 fe80::2d0:b7ff:febf:c695%fxp0 prefixlen 64 scopeid 0x1
fxp1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:02:b3:0a:d1:28
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::202:b3ff:fe0a:d128%fxp1 prefixlen 64 scopeid 0x2
fxp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:c0:4f:46:8d:ec
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.255.255.1 netmask 0xffffff00 broadcast 10.255.255.255
        inet6 fe80::2c0:4fff:fe46:8dec%fxp2 prefixlen 64 scopeid 0x3
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=41<UP,RUNNING> mtu 1348
        pfsync: syncif: fxp2 syncpeer: 224.0.0.240 maxupd: 128
enc0: flags=0<> mtu 1536
carp0: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 66.77.24.5 netmask 0xffffff00
        inet 66.77.24.10 netmask 0xffffff00
carp1: flags=41<UP,RUNNING> mtu 1500
        carp: BACKUP vhid 2 advbase 1 advskew 100
        inet 66.77.24.5 netmask 0xffffff00
        inet 66.77.24.10 netmask 0xffffff00
carp2: flags=41<UP,RUNNING> mtu 1500
        carp: MASTER vhid 1 advbase 1 advskew 0
        inet 10.0.0.1 netmask 0xffffff00
carp3: flags=41<UP,RUNNING> mtu 1500
        carp: BACKUP vhid 2 advbase 1 advskew 100
        inet 10.0.0.1 netmask 0xffffff00

The diagram in Figure 2 shows that we have added three HTTP servers and one SMTP server. Each of the Web servers has a private IP address, but they all use the same public address via NAT. Outbound connections will cause the source address to be translated; inbound connections will be redirected to one of the destination addresses, mapping packets based on a hash of the source address. This will allow the firewalls to persistently send traffic from one client to the same server. The SMTP server also has a private address, but its traffic is translated via bidirectional mapping (binat) to a new external carp interface. The revised pf.conf can be viewed in Listing 3. A compilation of all the revised network settings are detailed in Listing 4.

With arpbalance enabled, incoming packets will be distributed between the two firewalls in a round-robin fashion. New connections are tracked via the source-hash option to the rdr translation rule. This ensures that future packets originating from the same client are passed on to the same internal server. Additional load-balancing techniques can be used on the internal HTTP pool by incorporating CARP on each of the pool members. This way, we not only allow traffic to be distributed between all three hosts, but we provide failover capabilities as well, just like that of the firewall pair.

Summary

Through the use of native OpenBSD utilities, we have a high-availability firewall solution based on commodity hardware that competes favorably with commercial offerings costing thousands of dollars more. The practical applications for CARP are limitless, as it can be used anywhere that load-balancing needs exist. It has been ported to userland on Linux kernels 2.4 and 2.6, NetBSD and FreeBSD, making it an ideal redundancy tool no matter what Unix-like system you use. But combine CARP with the enhanced security of OpenBSD, PF and pfsync, and you have a stateful firewall that won’t let you or your customers down.

Resources

OpenBSD — http://www.openbsd.org/

OpenBSD FAQ — http://www.openbsd.org/faq/index.html

PF — http://www.benzedrine.cx/pf.html

PF User’s Guide — http://www.openbsd.org/faq/pf/index.html

Userland CARP — http://www.ucarp.org/

VRRP RFC 3768 — http://www.benzedrine.cx/pf.html

The hardware clock is generally only used to set the system clock when your operating system boots, and then from that point until you reboot or turn off your system, the system clock is the one used to keep track of time.

On Linux systems, you have a choice of keeping the hardware clock in UTC/GMT time or local time. The preferred option is to keep it in UTC. The disadvantage with keeping the hardware clock in UTC is that if you dual boot with an operating system (like DOS) that expects the hardware clock to be set to local time, the time might be wrong in that OS.

Set your timezone

The timezone under Linux is set by a symbolic link from /etc/localtime to a file in the /usr/share/zoneinfo directory that corresponds with what timezone you are in. For example, since I’m of same timezone as Hong Kong, /etc/localtime is a symlink to /usr/share/zoneinfo/Asia/Hong_Kong. To set this up, run:

[admon@planet ~]# ls -F /usr/share/zoneinfo/
Africa/      CST6CDT  Etc/     Greenwich  Kwajalein  Navajo     SystemV/   iso3166.tab
America/     Canada/  Europe/  HST        Libya      PRC        Turkey     posix/
Antarctica/  Chile/   Factory  Hongkong   MET        PST8PDT    UCT        posixrules
Arctic/      Cuba     GB       Iceland    MST        Pacific/   US/        right/
Asia/        EET      GB-Eire  Indian/    MST7MDT    Poland     UTC        zone.tab
Atlantic/    EST      GMT      Iran       Mexico/    Portugal   Universal
Australia/   EST5EDT  GMT+0    Israel     Mideast/   ROC        W-SU
Brazil/      Egypt    GMT-0    Jamaica    NZ         ROK        WET
CET          Eire     GMT0     Japan      NZ-CHAT    Singapore  Zulu

[admon@planet ~]# ln -sf /usr/share/zoneinfo/Asia/Hong_Kong /etc/localtime

Replace your timezone with something like US/Pacific or Europe/Paris. Have a look in the directories under /usr/share/zoneinfo to see what timezones meet your needs.

This assumes that you are running Red Hat (or Redhat based Linux) . On older systems, you’ll find that /usr/lib/zoneinfo is used instead of /usr/share/zoneinfo.

Set the system clock

To set the system clock under Linux, use the date command. As an example, to set the current time and date to Sun Dec 13 10:02:00 AM CST 2009, type date 12131002 (note that the time is in 24 hour notation).

[admon@planet ~]# date
Sun Dec 13 10:09:29 CST 2009
[admon@planet ~]# date 12131002
Sun Dec 13 10:02:00 CST 2009

You can reset system time without date information like this:

[admon@planet ~]# date -s 10:04
Sun Dec 13 10:04:00 CST 2009

If you wanted to change the year as well, just type date 121310022009. To set the seconds as well, type date 12131002.30 or date 121310022009.30. The following line is an example. When the command runs succesfully, it returns current system time:

[admon@planet ~]# date 121310212009.30
Sun Dec 13 10:21:30 CST 2009

An alternative way is to load system date and time from BIOS like this:

[admon@planet ~]# date
Sun Dec 13 10:02:04 CST 2009
[admon@planet ~]# hwclock --hctosys
[admon@planet ~]# date
Sun Dec 13 10:09:29 CST 2009

Set the hardware clock

When Linux boots, A initialization script will run the /sbin/hwclock program to copy the current hardware clock time to the system. To set the hardware clock, a common way is to set the system clock first, and then sync the new system time to hardware clock by typing /sbin/hwclock –systohc (or /sbin/hwclock –systohc –utc if you are keeping the hardware clock in UTC).

[admon@planet ~]# date
Sun Dec 13 10:29:57 CST 2009
[admon@planet ~]# hwclock --systohc
[admon@planet ~]# hwclock
Sun Dec 13 10:30:13 2009  -0.716300 seconds
[admon@planet ~]# date
Sun Dec 13 10:30:13 CST 2009

Sync local time with a time server
This can be done by an entry in root’s crontab like this:

[admon@planet ~]# crontab -l
MAILTO=""
30 4,16 * * * (/usr/sbin/ntpdate -s clock.redhat.com time.nist.gov ntp.admon.org)

It means your system time will be synced with these time servers every 12 hours. If you have a cluster of machines which must share the same time, it’s suggested to build your own time server, and get other servers synchronized with it.

Instead of keeping your system time up-to-date with the world, a local time server can be configed to supply delayed time, that means you can keep your system time delayed with the public. It’s helpful for some specific environment, like if you need some delayed mysql replication servers.

Luckily it’s easy to implement with the help of either sendmail or postfix, the two most common MTAs in Linux world. If you’re using sendmail as your default mail transfer agent, you can enable forwarding by MAIL_HUB, and all incoming mails would be sent to a centralized hub, then processed there.

Here we show an exmaple based on Postfix.
We use postfix to forward different mails to different smtp servers. To be specific, we forward mails that destinated to our own domains to an internal SMTP server, and relay outbound mails to another server which is able to access external network.
By implementing mail transfer like this, we can also speedup mail transfer.

All we need to do is to adjust postfix’s transport mapping table, it’s /etc/postfix/transport.

We just need to add two new lines in this file, it’s like this:

cards.example.com    :
example.com    :[192.168.19.198]

It directs mails for cards.example.com to local mail server, and mails for example.com to another mail server 192.168.19.198 which also has external IP address. The [] around the hostname is intended to disable MX lookups.

After making changes, use this command to make it take effect:

$ postmap /etc/postfix/transport

Then you may need to flush mail queue for the Postfix instance:

$ postfix flush

Some other tips may help you as well:

• To check mail queue list, run: mailq
• To remove all mail from the queue, run: postsuper -d ALL
• To remove all mails in the deferred queue, enter: postsuper -d ALL deferred
• To start/stop postfix instance: postfix start and postfix stop

It’s that simple, but much meaningful as you can define other rules in this file.

For any issues, dont forget to raise a ticket at our mail forum.

Every two years, HPTS brings together a lively and opinionated group of participants to discuss and debate the pressing topics that affect today’s systems and their design and implementation.

I’ve downloaded these papers on my website, you can read them on these links as well:

A Performance Puzzle: B Tree Insertions Are Slow on SSDs

A Sea Change is Coming on Transaction Drivers

Availability in the Cloud

Challenges and Lessons from Growing an e Commerce Platform to Planet Scale

Cloud DB

Enterprise Systems in the Cloud

Flash on Compute Servers

H Store: A specialized architecture for high throughput OLTP applications

Implementing Search in the Cloud

Implications of Storage Class Memories on Software Architectures

Lessons Learned Dealing with Massive Scale and Slow Networks in China

Look at Clouds from Both Sides

Memory Technologies for Data Intensive Computing

Meter Automation

Mobile Personal Sensing: A new driver for high performance transaction systems

RAMCloud: Scalable High Performance Storage Entirely in DRAM

Scalable transaction execution on multicore platforms

Scaling Out without Partitioning

SciDB: Unconventional Choices for Scientific Computing

Smart Grid Challenges

Supporting Large Scale Scientific and Engineering Applications Using DBMS Technology

Systems Software for Multicore Processors

Trustworthy Accounting

Write Caching with Reduced Durability

Xtremely Large File Systems for the small collaborative world

Zetta Enterprise Cloud Storage Service

Note: These are all in PDF format, and total size for these 26 documents are 76MB in total.

Network is the basis of the communication to your customers, page viewers. without network, computers are of no usage at all, arent they? But most of the guys always pay very little attention to the health of their networks. Generally, there are some type of networks issues should special pay attention:

1, abnormal network traffic, like pink traffic, highload traffic appeared some time with no clear reason, very low traffic during business time and so on. these type of issues can all be monitored by Hostpry, and the alerting messages can be scheduled separately in business time or non-business time.

2, ethernet card failure. If your fiber is not running at proper stat, you may find some error in dmesg like this:

tg3: eth0: Link is down.
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
tg3: eth0: Link is down.
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
tg3: eth0: Link is down.
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

sometimes, it will cause a highload and sometimes, your business critical connections would be reset by such issue. when you find this type of message, it’s recommended to use a new fibre instead.

3, connection statistics check Generally, there are three type of traffic need your attention:

SYN packet
mostly they are attacks if you face too many SYN packet above the normal ratio.(some network problem may also cause too much SYN packets).

$ netstat -an | grep -c SYN_RECV
510

Too much Established Connections
It should be treated as attack sometimes, and you need to check your system to make sure whether you are right.

$ netstat -st | grep estab
8436 connections established
265 packets rejects in established connections because of timestamp

The service log, for example apache’s access_log, sendmail’s maillog will be the key points for you to prove it.

Abnormal ICMP traffic
Ping of Death is a well-known ICMP Flood Attack. An ICMP attack can come in many forms, and sometimes it has some relation with UDP. ICMP flood attack can usually be accomplished by broadcasting either a bunch of  ICMP pings or UDP packets. UDP is widely used in softwares s like Bind, when an UDP cannot find its destination port or destination host, an ICMP reply will be generated to the source side.
The idea is that the bad guys generate so much data that destinated to your system( for example, using your IP address to create some faked packets and broad them, then your real server will be flooded by the returned ICMP. This will slow you down so much that you can not supply any service.
Also there are ways to find out such issue as they are slightly recorded by  modern operation systems. ]]> http://www.admon.org/what-network-issues-should-be-monitored/feed/ 0 http://www.admon.org/common-wget-usage-examples/ http://www.admon.org/common-wget-usage-examples/#comments Tue, 08 Sep 2009 13:01:24 +0000 joseph http://blog.admon.org/?p=108

Wget is one of my favorite tools in Linux/Unix world. Sometimes, you want to download all the rpm, deb, iso, or tgz files and save them into a directory. Sometimes you need to use it to check your web server status. Here are some of my favorite wget usage examples:

$ wget -i filename.txt
Put the URLs in filename.txt and run wget against it to download a list of files automatically.

How to download large files in a bad connection? You can have a try with –continue option:
$ wget -c http://www.020i.com/really-big-file.iso
The “-c” option tells wget to continue and retry until downloading completed.

$ wget –spider http://www.020i.com/
This command is very useful to check a web server’s running status. A “200 OK” in the output means your web server is ready for request.

$ wget -r -np -nd http://www.020i.com/files/
This little command is probably the most used variation. It downloads all files in the /files/ directory on 020i.com, without traversing up to parent directories (-np), and without recreating the directory structure on your machine (-nd).

$ wget -r -np -nd –accept=iso http://www.020i.com/centos-5/i386/
Adding the -–accept argument with a list of file extensions (comma separated) will grab only these files in the right extensions.

The most important part for this installation is openvpn’s two config files, one is the server side configuration, and the other is for client side.

This is a server side configuration file:

$ cat /usr/local/openvpn/etc/server.conf
# Which local IP address should OpenVPN listen on? (optional)
;local a.b.c.d

port 1194
proto udp
;dev tap
dev tun

;dev-node MyTap

ca /usr/local/openvpn/etc/keys/ca.crt
cert /usr/local/openvpn/etc/keys/server.crt
key /usr/local/openvpn/etc/keys/server.key
dh /usr/local/openvpn/etc/keys/dh1024.pem

server 10.10.10.0 255.255.255.0
ifconfig-pool-persist ipp.txt

;server-bridge 10.10.10.2 255.255.255.0 10.10.10.30 10.10.10.40

;push "route 10.10.10.0 255.255.255.0"
;push "route 192.168.20.0 255.255.255.0"

;client-config-dir ccd
;route 192.168.40.128 255.255.255.248
# First uncomment out these lines:
;client-config-dir ccd
;route 10.9.0.0 255.255.255.252
# Then add this line to ccd/Thelonious:
#   ifconfig-push 10.9.0.1 10.9.0.2

;learn-address ./script
;push "redirect-gateway def1"

# http://openvpn.net/faq.html#dhcpcaveats
push "dhcp-option DNS 85.17.150.123"
;push "dhcp-option WINS 10.8.0.1"

tls-auth /usr/local/openvpn/etc/keys/ta.key 0
client-to-client
keepalive 10 120

# Select a cryptographic cipher.
# This config item must be copied to the client config file as well.
;cipher BF-CBC        # Blowfish (default)
;cipher AES-128-CBC   # AES
;cipher DES-EDE3-CBC  # Triple-DES

# Enable compression on the VPN link.
# If you enable it here, you must also enable it in the client config file.
comp-lzo

# The maximum number of concurrently connected clients we want to allow.
;max-clients 100

# It's a good idea to reduce the OpenVPN daemon's privileges after initialization.
# You can uncomment this out on non-Windows systems.
user nobody
group nobody

persist-key
persist-tun

status /usr/local/openvpn/logs/openvpn-status.log
log /usr/local/openvpn/logs/openvpn.log
;log-append  openvpn.log

verb 4
;mute 20

Here’s an example of the client side configuration file, it’s used by windows client:

$ cat  netherlands.ovpn
client
;dev tap
dev tun
proto udp

remote openvpn.admon.org 1194

;remote-random
;resolv-retry infinite
;nobind
;user nobody
;group nobody

persist-key
persist-tun

;http-proxy-retry # retry on connection failures
;http-proxy [proxy server] [proxy port #]

;mute-replay-warnings

ca ca.crt
cert joseph.crt
key joseph.key
tls-auth ta.key 1

ns-cert-type server
;cipher x
comp-lzo
verb 3
;mute 20
redirect-gateway def1

Let’s start the deploying now. For a typic fresh installation, we need to install both lzo ( Real-time data compression library ) and OpenVPN, for the windows client side, we need its windows version, it’s available here: http://www.openvpn.se/

After created the applications, it’s time to generate key files, I simply listed the commands here:

$ openvpn-2.0.9/easy-rsa

$ vi vars
$ . vars
$ ./clean-all

$ ./build-ca
$ ./build-key-server server
$ ./build-dh  #Diffie Hellman parameters
$ openvpn --genkey --secret ta.key
$ ./build-key joseph  #Client key, keys/joseph.*
$ ./build-key client0  #Client0's key files, they're stored in keys/client0.*

After finished creating client keys, we may need to modify the client
side configuration file, to make sure it uses correct key file. In the
above example, we’re using joseph.*. When these key files generated, there’re two additional modifications, they both are critical:
On OpenVPN server, we need to turn on ip_forward, in order to redirect VPN traffic on localhost.

$ echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 10.10.10.0/255.255.255.0 -o eth0 -j SNAT --to-source 94.94.94.94

Now,all the configuratio is finished, let’s start the OpenVPN server now:

/usr/local/openvpn/sbin/openvpn --daemon --config /usr/local/openvpn/etc/server.conf