Functionality Added or Changed

Bugs Fixed

• InnoDB Storage Engine: A failed CREATE INDEX operation for an InnoDB table could result in some memory being allocated and not freed. This memory leak could affect tables created with the ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPRESSED settings. (Bug #12699505)
• Partitioning: Auto-increment columns of partitioned tables were checked even when they were not being written to. In debug builds, this could lead to a crash of the server. (Bug #11765667, Bug #58655)
• Partitioning: The UNIX_TIMESTAMP() function was not treated as a monotonic function for purposes of partition pruning. (Bug #11746819, Bug #28928)
• Replication: If a LOAD DATA INFILE statement—replicated using statement-based replication—featured a SET clause, the name-value pairs were regenerated using a method (Item::print()) intended primarily for generating output for statements such as EXPLAIN EXTENDED, and which cannot be relied on to return valid SQL. This could in certain cases lead to a crash on the slave.To fix this problem, we now name each value in its original, user-supplied form, and use that to create LOAD DATA INFILEstatements for statement-based replication. (Bug #11902767, Bug #60580)See also Bug #34283, Bug #11752526, Bug #43746.
• Compiling the server with maintainer mode enabled failed for gcc 4.6 or higher. (Bug #12727287)
• The option-parsing code for empty strings leaked memory. (Bug #12589928)
• Previously, an inappropriate error message was produced if a multiple-table update for an InnoDB table with a clustered primary key would update a table through multiple aliases, and perform an update that may physically move the row in at least one of these aliases. Now the error message is: Primary key/partition key update is not permitted since the table is updated both as ‘tbl_name1‘ and ‘tbl_name2‘(Bug #11882110)See also Bug #11764529.
• ALTER TABLE {MODIFY|CHANGE} … FIRST did nothing except rename columns if the old and new versions of the table had exactly the same structure with respect to column data types. As a result, the mapping of column name to column data was incorrect. The same thing happened for ALTER TABLE DROP COLUMN, ADD COLUMN statements intended to produce a new version of table with exactly the same structure as the old version. (Bug #61493, Bug #12652385)
• Incorrect handling of metadata locking for FLUSH TABLES WITH READ LOCKfor statements requiring prelocking caused two problems:
  • Execution of any data-changing statement that required prelocking (that is, involved a stored function or trigger) as part of transaction slowed down somewhat all subsequent statements in the transaction. Performance in a transaction that periodically involved such statements gradually degraded over time.
  • Execution of any data-changing statement that required prelocking as part of transaction prevented a concurrent FLUSH TABLES WITH READ LOCK from proceeding until the end of transaction rather than at the end of the particular statement.

  (Bug #61401, Bug #12641342)

• The fractional part of the “Queries per second” value in MySQL status output could be displayed incorrectly. (Bug #61205, Bug #12565712)
• LOAD DATA INFILE incorrectly parsed relative data file path names that ascended more than three levels in the file system and as a consequence was unable to find the file. (Bug #60987, Bug #12403662)
• For unknown users, the native password plugin reported incorrectly that no password had been specified even when it had. (Bug #59792, Bug #11766641)
• For MyISAM tables, attempts to insert incorrect data into an indexed GEOMETRY column could result in table corruption. (Bug #57323, Bug #11764487)
• In debug builds, Field_new_decimal::store_value() was subject to buffer overflows. (Bug #55436, Bug #11762799)
• A race condition between loading a stored routine using the name qualified by the database name and dropping that database resulted in a spurious error message: The table mysql.proc is missing, corrupt, or contains bad data (Bug #47870, Bug #11756013)

1, Create a configuration file for MongDB, which is located at /etc/mongodb.conf, here’s an example:

# This is an example config file for MongoDB.
dbpath = /var/lib/mongodb
logpath = /var/log/mongodb/mongod.log
bind_ip = 0.0.0.0
noauth = true # use 'true' for options that don't take an argument
verbose = false # to disable, comment out.
fork = true
nohttpinterface = true

2, Copy and paste the following init.d scripit to /etc/init.d/mongo (make sure it’s executable), then enable the init script by update-rc.d.

#!/bin/sh
#
# init.d script with LSB support.
#
# Copyright (c) 2007 Javier Fernandez-Sanguino
#
# This is free software; you may redistribute it and/or modify
# it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2,
# or (at your option) any later version.
#
# This is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License with
# the Debian operating system, in /usr/share/common-licenses/GPL;  if
# not, write to the Free Software Foundation, Inc., 59 Temple Place,
# Suite 330, Boston, MA 02111-1307 USA
#
### BEGIN INIT INFO
# Provides:          mongodb
# Required-Start:    $network $local_fs
# Required-Stop:
# Should-Start:      $named
# Should-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: An object/document-oriented database
# Description:       MongoDB is a high-performance, open source, schema-free
#                    document-oriented  data store that's easy to deploy, manage
#                    and use. It's network accessible, written in C++ and offers
#                    the following features:
#
#                       * Collection oriented storage - easy storage of object-
#                         style data
#                       * Full index support, including on inner objects
#                       * Query profiling
#                       * Replication and fail-over support
#                       * Efficient storage of binary data including large
#                         objects (e.g. videos)
#                       * Auto-sharding for cloud-level scalability (Q209)
#

#                    High performance, scalability, and reasonable depth of
#                    functionality are the goals for the project.
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/mongod
DAEMON_OPTS='-f /etc/mongodb.conf run'
NAME=MongoDB
DESC=database

test -x $DAEMON || exit 0

. /lib/lsb/init-functions

LOGDIR=/var/log/mongodb
PIDFILE=/var/run/$NAME.pid
DIETIME=10                   # Time to wait for the server to die, in seconds
                            # If this value is set too low you might not
                            # let some servers to die gracefully and
                            # 'restart' will not work

LOGFILE=$LOGDIR/$NAME.log  # Server logfile

# Include mongodb defaults if available
if [ -f /etc/default/$NAME ] ; then
        . /etc/default/$NAME
fi

DAEMONUSER=mongodb
# Check that the user exists (if we set a user)
# Does the user exist?
if [ -n "$DAEMONUSER" ] ; then
    if getent passwd | grep -q "^$DAEMONUSER:"; then
        # Obtain the uid and gid
        DAEMONUID=`getent passwd |grep "^$DAEMONUSER:" | awk -F : '{print $3}'`
        DAEMONGID=`getent passwd |grep "^$DAEMONUSER:" | awk -F : '{print $4}'`
    else
        log_failure_msg "The user $DAEMONUSER, required to run $NAME does not exist."
        exit 1
    fi

fi

set -e

running_pid() {
# Check if a given process pid's cmdline matches a given name
    pid=$1
    name=$2
    [ -z "$pid" ] && return 1
    [ ! -d /proc/$pid ] &&  return 1
    cmd=`cat /proc/$pid/cmdline | tr "\000" "\n"|head -n 1 |cut -d : -f 1`
    # Is this the expected server
    [ "$cmd" != "$name" ] &&  return 1
    return 0
}

running() {
# Check if the process is running looking at /proc
# (works for all users)

    # No pidfile, probably no daemon present
    [ ! -f "$PIDFILE" ] && return 1
    pid=`cat $PIDFILE`
    running_pid $pid $DAEMON || return 1
    return 0
}

start_server() {
# Start the process using the wrapper
        if [ -z "$DAEMONUSER" ] ; then
            start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
                        --make-pidfile --exec $DAEMON -- $DAEMON_OPTS
            errcode=$?
        else
echo $DAEMONUSER

# if we are using a daemonuser then change the user id
            start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
                        --make-pidfile --chuid $DAEMONUSER \
                        --exec $DAEMON -- $DAEMON_OPTS
            errcode=$?

        fi
        return $errcode
}

stop_server() {
# Stop the process using the wrapper
        if [ -z "$DAEMONUSER" ] ; then
            start-stop-daemon --stop --quiet --pidfile $PIDFILE
            rm $PIDFILE
            errcode=$?
        else
# if we are using a daemonuser then look for process that match
            start-stop-daemon --stop --quiet --pidfile $PIDFILE \
                        --user $DAEMONUSER \
                        --exec $DAEMON
            errcode=$?
        fi

        return $errcode
}
reload_server() {
    [ ! -f "$PIDFILE" ] && return 1
    pid=pidofproc $PIDFILE # This is the daemon's pid
    # Send a SIGHUP
    kill -1 $pid
    return $?
}

force_stop() {
# Force the process to die killing it manually
        [ ! -e "$PIDFILE" ] && return
        if running ; then
                kill -15 $pid
        # Is it really dead?
                sleep "$DIETIME"s
                if running ; then
                        kill -9 $pid
                        sleep "$DIETIME"s
                        if running ; then
                                echo "Cannot kill $NAME (pid=$pid)!"
                                exit 1
                        fi
                fi
        fi
        rm -f $PIDFILE
}

case "$1" in
  start)
        log_daemon_msg "Starting $DESC $NAME"
        # Check if it's running first
        if running ;  then
            log_progress_msg "apparently already running"
            log_end_msg 0
            exit 0
        fi
        if start_server ; then
            # NOTE: Some servers might die some time after they start,
            # this code will detect this issue if STARTTIME is set
            # to a reasonable value
            [ -n "$STARTTIME" ] && sleep $STARTTIME # Wait some time 

            if  running ;  then
                # It's ok, the server started and is running
                log_end_msg 0
            else
                # It is not running after we did start
                log_end_msg 1
            fi
        else
            # Either we could not start it
            log_end_msg 1
        fi
        ;;
  stop)
        log_daemon_msg "Stopping $DESC" "$NAME"
        if running ; then
            # Only stop the server if we see it running
                        errcode=0
            stop_server || errcode=$?
            log_end_msg $errcode
        else
            # If it's not running don't do anything

            log_progress_msg "apparently not running"
            log_end_msg 0
            exit 0
        fi
        ;;
  force-stop)
        # First try to stop gracefully the program
        $0 stop
        if running; then
            # If it's still running try to kill it more forcefully
            log_daemon_msg "Stopping (force) $DESC" "$NAME"
                        errcode=0
            force_stop || errcode=$?
            log_end_msg $errcode
        fi
        ;;
  restart|force-reload)
        log_daemon_msg "Restarting $DESC" "$NAME"
                errcode=0
        stop_server || errcode=$?
        # Wait some sensible amount, some server need this

        [ -n "$DIETIME" ] && sleep $DIETIME
        start_server || errcode=$?
        [ -n "$STARTTIME" ] && sleep $STARTTIME
        running || errcode=$?
        log_end_msg $errcode
        ;;
  status)

        log_daemon_msg "Checking status of $DESC" "$NAME"
        if running ;  then
            log_progress_msg "running"
            log_end_msg 0
        else
            log_progress_msg "apparently not running"
            log_end_msg 1
            exit 1
        fi
        ;;
  # Use this if the daemon cannot reload
  reload)
        log_warning_msg "Reloading $NAME daemon: not implemented, as the daemon"

        log_warning_msg "cannot re-read the config file (use restart)."
        ;;

  *)
        N=/etc/init.d/$NAME
        echo "Usage: $N {start|stop|force-stop|restart|force-reload|status}" >&2
        exit 1
        ;;
esac

exit 0

Finally Mongodb officially supplies mongodb apt-gettable packages which also contains an initial script, but as per my testings, it has some errors in Ubuntu 9.10.

As mentioned at Facebook.com:

We built Flashcache to help us scale InnoDB/MySQL, but it was designed as a generic caching module that can be used with any application built on top of any block device. For InnoDB, when the working set does not fit in the InnoDB buffer pool, read latency is significantly improved due to caching more of the working set in faster media, such as SSD’s. We also improve write performance by first caching writes in SSD’s and lazily flushing the data back to disk.

It looks like its based on the dm-cache work. It’s nice to see that moving again.

Read more details about Flashcache here at Facebook’s announcing page, and Flashcache’s source code is available at GitHub.com.

NOTE: On a MySQL replication environment, users must be careful in deleting binary logs. If slaves are lagging behind the server SOME days, the binary log purge might cause replication to fail. Make sure the logs file are not in use before removing them!

In any case here are some useful examples for purging binary log files easiliy. We’ll show you two quick ways on deleting the expired bin-log files.

As the purge work here is issued by mysql itself, please make sure you have MySQL-4.1 and later version installed. For version 4.0 and 3.xx, it does not support the following feathers.

1, Delete MySQL bin-log files by PURGE statement

For example, if you need to delete all binary logs older than 7 days, you just need to run this sql:

mysql> PURGE BINARY LOGS BEFORE DATE_SUB( NOW( ), INTERVAL 7 DAY);

To purge all logs before a specific date:

mysql> PURGE MASTER LOGS BEFORE '2010-03-01 00:00:00';

To purge logs automatically (every Monday at 3am) you could use a cron job like this:

0 3 * * mon mysql -uroot -e 'PURGE BINARY LOGS BEFORE DATE_SUB( NOW( ), INTERVAL 7 DAY)'

2, Remove bin-log by the variable expire_log_days

An alternative option is to set “expire_log_days” as MySQL server parameter. The default values for expire_log_days is not to purge logs. You can either set it in MySQL’s configuration file or in a command terminal like below:

mysql> SET GLOBAL expire_logs_days = 7;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like 'expire_%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| expire_logs_days |  7    |
+------------------+-------+
1 row in set (0.00 sec)

When this vavirable takes effect, the log files will be removed automatically on a weekly basis.
For any issues, don’t forget to raise a thread at our database support forum here.

Reset user password in MySQL
Let’s assume the normal user’s name is fatman, follow these steps to start:
1), Login to MySQL server, type following command at shell prompt to login as root

$ mysql -u root -p

2), Choose the right database mysql database, it’s mysql:

mysql> use mysql;

3), Change password for user fatman:

mysql> update user set password=PASSWORD("NEW-PASSWORD-HERE") where User='fatman';

4), Finally,dont forget to flush privileges:

mysql> flush privileges;

You can also use grant to reset a user’s password, the command is like this:

mysql> grant select,update,delete,insert on somedb.* to fatman@'somehost' identified by 'NEW-PASSWORD-HERE';

Reset DBA password in MySQL
When the DBA’s password lost, things become a little difficult, a restart might needed which is not so good for productive servers. There still have some possibilites to retrieval it without restart mysql service:

1), Is there any user has insert permission on the database “mysql”?
If so, he can insert into mysql.user and create another DBA account.

2), Is there any user has super user permissions?
For example, when the user was create like “GRANT ALL ON *.* TO user@’localhost’ WITH GRANT OPTION“, then this use is actually another DBA, you can ask him to reset root’s password.

If you got no luck till now, we have a common way here to fix the password. We can ask mysql server to bypass its grant tables when starting it. Thus it will not verify user passwords and permissions when they logon. That means all users will have full access to all databases as a database administrator.

$ /usr/sbin/mysqld --verbose --help | grep skip-grant-tables -A1
  --skip-grant-tables Start without grant tables. This gives all users FULL
                      ACCESS to all tables!
--
skip-grant-tables                 FALSE
skip-slave-start                  FALSE

Before making this changes, you need to make sure there’s no queries running on mysql. Then shutdown mysql instance first (mysql’s control script will help do that. In RPM based systems, it’s /etc/init.d/mysql) , and apply this option in the script, and start the service again.

When the DBA password got reset, do forget to remove the changes in mysql’s control script. If there any issue occures, please dont hesitate to create a topic in our support forum!

As described on Wikipedia.org:

Cassandra provides a structured key-value store with eventual consistency. Keys map to multiple values, which are grouped into column families. The column families are fixed when a Cassandra database is created, but columns can be added to a family at any time. Furthermore, columns are added only to specified keys, so different keys can have different numbers of columns in any given family. The values from a column family for each key are stored together, making Cassandra a hybrid between a column-oriented DBMS and a row-oriented store.

Facebook released Cassandra as open source in July 2008, it is now an Apache Incubator project.

Digg announced that it started to roll out its use of Cassandra on Sep 9th, 2009. Twitter is also working on using Cassandra to replace their currents storage for all tweets.

Twitter has a cluster in production that’s being populated outside the the user-critical path (ie, the cassandra writing is async).  They also evaluated a lot of solutions, including a custom mysql impl, voldemort, hbase, mongodb, memcachdb, hypertable, etc. Finally they settled on Cassandra.

It’s really a nice promation for the NoSQL Movement, and if you faced similiar issues on processing large amount of data, it’s suggested to have a try with it.

Basically what you need to do is to retrival a snapshot of your master database. It covers three types of information, a snapshot of your current mysql database, and master log file name and its running position when this snapshot created.

You can achieve these information on a running DB like this:

Firstly, frozen your database, in order to disable any writes/updates on the databases. It affects your service for sure. Run two sql queries on a terminal:

mysql> FLUSH TABLES WITH READ LOCK;
Query OK, 0 rows affected (0.53 sec)

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| binlog-db.000004 |  88689   |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.01 sec)

Keep this session open.  Note that when this session closes, READ LOCK will vanish, and your database will keep changing again.

Then open another terminal to create a copy of your current database files. There are three common ways to do that depending on your system environment. Either of these methods is okay.

1, Use mysqldump to copy all the data.

The disadvantage for mysqldump is that it uses very long time for dumpping large databases. It’s really unacceptable on productive servers.

shell$ mysqldump --database db1 db2 > fullbackup-timestamp.sql

2, Use a LVM snapshot

It only takes some seconds to create a LVM snapshot. When your snapshot created, you can copy the snapshot to the slave server.
As all the database files are exactly the same as master DB, you need to make sure the configuration files are similar as well.

3, Use XFS’ frozen feature to do that

SYSTEM xfs_freeze -f /var/lib/mysql;
SYSTEM YOUR_SCRIPT_TO_CREATE_SNAPSHOT.sh;
SYSTEM xfs_freeze -u /var/lib/mysql;

After the snapshot createed, you either leave the first session, or submit this query directly:

mysql> UNLOCK TABLES;

Finally you need to restore these databases from mysql dump file or the snapshot at slave side. When recovering a large database, time make lots of sense. it might take you one or more days to finish a recovery.
To speed up a mysql recovery, there’re always some valuable advises from wise man (The wise man here is … my friend). for example you can disable binlog when importing large sql files.

mysql> SET SQL_LOG_BIN=0;
Query OK, 0 rows affected (0.00 sec)

mysql> SOURCE /PATH/TO/MYSQL-BACKUP.SQL

When the files are importing, you need to make sure there are enough disk space avaiable, otherwise, it would crashed again. After importing all data files at slave side,  you need to tell the slave at which point it should read binlog from master:

mysql> SLAVE STOP;

mysql> SLAVE RESET;

mysql> CHANGE MASTER TO MASTER_HOST='pub.admon.org',MASTER_LOG_FILE='binlog-db.000004',MASTER_LOG_POS=88689;

myql> SLAVE START;

If all your recovery work is a remote operation, using Screen is highly suggested.

Every two years, HPTS brings together a lively and opinionated group of participants to discuss and debate the pressing topics that affect today’s systems and their design and implementation.

I’ve downloaded these papers on my website, you can read them on these links as well:

A Performance Puzzle: B Tree Insertions Are Slow on SSDs

A Sea Change is Coming on Transaction Drivers

Availability in the Cloud

Challenges and Lessons from Growing an e Commerce Platform to Planet Scale

Cloud DB

Enterprise Systems in the Cloud

Flash on Compute Servers

H Store: A specialized architecture for high throughput OLTP applications

Implementing Search in the Cloud

Implications of Storage Class Memories on Software Architectures

Lessons Learned Dealing with Massive Scale and Slow Networks in China

Look at Clouds from Both Sides

Memory Technologies for Data Intensive Computing

Meter Automation

Mobile Personal Sensing: A new driver for high performance transaction systems

RAMCloud: Scalable High Performance Storage Entirely in DRAM

Scalable transaction execution on multicore platforms

Scaling Out without Partitioning

SciDB: Unconventional Choices for Scientific Computing

Smart Grid Challenges

Supporting Large Scale Scientific and Engineering Applications Using DBMS Technology

Systems Software for Multicore Processors

Trustworthy Accounting

Write Caching with Reduced Durability

Xtremely Large File Systems for the small collaborative world

Zetta Enterprise Cloud Storage Service

Note: These are all in PDF format, and total size for these 26 documents are 76MB in total.

What follows is not a realistic benchmark, it’s not scientific, it’s just some quick and dirty tinkering. Baron threw up an Ubuntu 9.04 small server on Amazon. (Because there’s a .deb of MonetDB for it), and created a table with 200 integer columns and loaded it with random numbers between 0 and 10000. He used default tuning, out of the box, for all three databases.

The performance testings was led by three simple queries, which is much informative. And of course, the comments are as informative as well, which worths a careful read too.

Read the post at MPB:  Quick comparison of MyISAM, Infobright, and MonetDB

Thank MPB for the nice work

Query Writing Rules

For all database management systems (DBMSes), “round-trip” time is significant. This is the amount of time which it takes a query to get through the the language parser, the driver, across the network interface, the database parser, the planner, the executor, the parser again, back across the network interface, through the driver data handler, and to the client application. DBMSes vary in the amount of time and CPU they take to process this cycle, and for a variety of reasons PostgreSQL is a the high end of time and system resources per round-trip.

Further, PostgreSQL has significant per-transaction overhead, including log output and visibility rules which need to be set with each transaction. While you may think that you are not using transactions for singleton read-only SELECT statement, in fact every single statement in PostgreSQL is in a transaction. In the absence of an explicit transaction, the statement itself is an implicit transaction.

Offsetting this, PostgreSQL is only barely second to Oracle in processing large complex queries, and has the capability to handle complex multi-statement transactions with overlapping concurrency conflicts with ease. We also support cursors, both scrollable and non-scrollable.

Never use many small selects when one big query could go the job.

It’s common in MySQL applications to handle joins in the application code,
that is, by querying the ID from the parent record and then looping  through the child records with that ID manually. This can result in running hundreds or thousands of queries per user interface screen. Each of these queries carres 2-6 milleseconds of round-trip time, which doesn’t seem significant until you add it up for 1000 queries, at which point you’re losing 3-5 seconds to round trip time. Comparatively, retrieving all of those records in a single query only takes a few hundred milleseconds, a time savings of 80%.

Group many small UPDATES, INSERTS or DELETEs into large statements, or failing that, large transactions.

First, the lack of subselects in early versions of MySQL has caused  application developers to design their data modification statements (DML) in much the same way as joins-in-middleware. This is also a bad approach for PostgreSQL. Instead, you want to take advantage of subselects and joins in your UPDATE, INSERT and DELETE statements to try to modify batches in a single statement. This reduces round-trip time and transaction overhead.

In some cases, however, there is no single query which can write all the rows you want and you have to use a bunch of serial statements. In this case, you want to make sure to wrap your series DML statements in an explicit transaction (e.g. BEGIN; UPDATE; UPDATE; UPDATE; COMMIT; ). This reduces transaction overhead and can cut execution time by as much as 50%.

Consider bulk loading instead of serial INSERTS

PostgreSQL provides a bulk loading mechanism called COPY, which takes tab-delimited or CSV input from a file or pipe. Where COPY can be used instead of hundreds or thousands of INSERTS, it can cut execution time by up to 75%.

DELETE is expensive

It’s common for application developers to think that a DELETE statement is virtually without cost. You’re just unlinking some nodes, right? Wrong. DBMSes are not filesystems; when you delete a row, indexes need to be updated and free space needs to be cleaned up, making a delete in fact more costly than an insert. Thus applications which habitually delete all detail rows and replace them with new every time a use makes any change are saving effort on the application side by pushing it onto the database. Where possible, this should be replaced with more discriminating row-replacement behavior, like updating only modified rows.

Also, when clearing a whole table, always use the TRUNCATE TABLE statement rather than DELETE FROM TABLE. The former is up to 100 times faster than the latter because it batch processes the whole table as a unit instead of one row at a time.

Use PREPARE/EXECUTE for query loops

Sometimes despite your attempts to consolidate loops of near-identical queries into large statements, it’s not possible due to the structure of your
application. That’s what PREPARE … EXECUTE is for; it allows the  database engine to skip parsing and planning for each query iteration. For example:

Prepare:
query_handle = query(‘SELECT * FROM TABLE WHERE id = ?’)(parameter_type = INTEGER)

Then start your loop:
for 1..100
query_handle.execute(i);
end

Classes for prepared statements for C++ are explained in the libpqxx documentation.

This will reduce query execution time in direct proportion to the size of the looped set.

Use connection pools effectively

For a web application, you’ll find that as much as 50% of your potential  performance can be controlled through your use of, and proper configuration of, connection pooling. This is because creating and dropping database connections takes a significant amount of system time, and excess idle connections still require RAM and system resources.

There are a number of tools you can use to do connection pooling on PostgreSQL. A third-party Open Source tool which is very popular is pgPool. However, for a C++ application HA requirements, it’s probably better to use libpqxx’s native pseudo-pooling technique called “lazy connections” . I suggest contacting the libpqxx mailing list for more information on how to use this utility.

With PostgreSQL, you want to have as many persistent connections (or connection objects) defined as your normal peak concurrent connection usage. So if max normal usage (first thing in the morning, say) is 200 concurrent connections from agents, users and components, then you want to have that many defined so that your application does not have to wait for new connections during a peak when they will be slow to fork.

Josh Berkus (PostgreSQL Core Team Member) Original Version is here, posted in 11/28/2006, this forwared version is just for track record.