Updating Drupal from 5 to 6 causes “The post could not be saved” – Postgres version fix

I recently upgraded several Drupal sites to 6.2 Some were 5.1, 5.2, and 6 previously. Upon completion of what I thought was a successul upgrade, they all suffered from the same common problem – trying to save a new post resulted in a “The post could not be saved” error. There were also various other database access errors in logs. On one site it was observed that after trying to make a post, Drupal would take it upon itself to take the site offline with the famous “Welcome to your new Drupal website” message. The problem there turned out to be a permission access problem, but only occurred after a failed post was attempted. To repair, go to admin/content/node-settings and select the “rebuild permissions” button. Once the permissions were repaired it was fine – until you made another post and then Drupal repeated the offline cycle and the beginning of page access SQL errors in the log again.

Fortunately there were some hints as to the problem, but the real problem is that they were mostly for MySQL, and I was using Postgres. Here’s my solution to the problem.

First, the most verbose error message I got was on this very site, where it looked like this after trying to post:

* warning: pg_query(): Query failed: ERROR: duplicate key value violates unique constraint “node_revisions_pkey” in /path/to/drupal-6.2/includes/database.pgsql.inc on line 139.
* user warning: query: INSERT INTO node_revisions (nid, uid, title, body, teaser, log, timestamp, format) VALUES (0, 1, ‘test’, ‘test test test test test test test test test test test vv’, ‘test test test test test test test test test test test test test vv’, ”, 1294612270, 4) in /path/to/drupal-6.2/includes/common.inc on line 3538.
* warning: pg_query(): Query failed: ERROR: duplicate key value violates unique constraint “node_vid_idx” in path/to/drupal-6.2/includes/database.pgsql.inc on line 139.
* user warning: query: INSERT INTO node (vid, type, language, title, uid, status, created, changed, comment, promote, moderate, sticky, tnid, translate) VALUES (0, ‘blog’, ”, ‘test’, 1, 1, 1294612270, 1294612270, 2, 1, 0, 0, 0, 0) in /path/to/drupal-6.2/includes/common.inc on line 3538.
* warning: pg_query(): Query failed: ERROR: duplicate key value violates unique constraint “node_comment_statistics_pkey” in /path/to/drupal-6.2/includes/database.pgsql.inc on line 139.
* user warning: query: INSERT INTO node_comment_statistics (nid, last_comment_timestamp, last_comment_name, last_comment_uid, comment_count) VALUES (0, 1294612270, NULL, 1, 0) in /path/to/drupal-6.2/modules/comment/comment.module on line 607.
* The post could not be saved.

If you are using postgres on the command line via the psql client, you can examine all sequences using the command

\ds

You’ll see that both node_nid_seq and node_revisions_vid_seq have been created. If you’ve tried creating a number of new pages or posts and failed, you’ll notice if you use nextval() like:

SELECT nextval('node_nid_seq');

It will be quite a bit advanced from your actual last successful post’s nid, which you can see like this:

SELECT max(nid) FROM node;

However, if you do this:

SELECT nextval('node_revisions_vid_seq');

Make a note of it. Then find your actual last node_revisions max vid like this:

SELECT max(vid) FROM node_revisions;

And you’ll see it is probably only one higher.

At this point, examine the node table like this:

\d node

                                  Table "public.node"
  Column   |          Type          |                     Modifiers
-----------+------------------------+----------------------------------------------------
 nid       | integer                | not null default nextval('node_nid_seq'::regclass)
 vid       | integer                | not null default 0
 type      | character varying(32)  | not null default ''::character varying
 uid       | integer                | not null default 0
 status    | integer                | not null default 1
 created   | integer                | not null default 0
 changed   | integer                | not null default 0
 comment   | integer                | not null default 0
 promote   | integer                | not null default 0
 moderate  | integer                | not null default 0
 sticky    | integer                | not null default 0
 mos_id    | integer                |
 language  | character varying(12)  | not null default ''::character varying
 tnid      | integer                | not null default 0
 translate | integer                | not null default 0
 title     | character varying(255) | not null default ''::character varying

And you can correctly see under “Modifiers” that the default value for nid is the next value in the node_nid_seq, which is the equivalent of the more familiar MySQL AUTO_INCREMENT attribute.

Now try the same on node_revisions:

\d node_revisions

                       Table "public.node_revisions"
  Column   |          Type          |               Modifiers
-----------+------------------------+----------------------------------------
 vid       | integer                | not null default 0
 uid       | integer                | not null default 0
 body      | text                   | not null
 teaser    | text                   | not null
 log       | text                   | not null
 timestamp | integer                | not null default 0
 format    | integer                | not null default 0
 nid       | integer                | not null default 0
 title     | character varying(255) | not null default ''::character varying

And you’ll see that the default value for vid is zero (0). Of course it isn’t auto-incrementing and so you are getting errors trying to write because you already have a vid with the value 0 in it:

SELECT min(vid) FROM node_revisions;
 min
-----
   0
(1 row)

So what is the solution?

ALTER TABLE node_revisions ALTER COLUMN vid SET DEFAULT nextval('node_revisions_vid_seq'::regclass);

Double check to make sure it went through:

\d node_revisions
                                   Table "public.node_revisions"
  Column   |          Type          |                          Modifiers
-----------+------------------------+--------------------------------------------------------------
 vid       | integer                | not null default nextval('node_revisions_vid_seq'::regclass)
 uid       | integer                | not null default 0
 body      | text                   | not null
 teaser    | text                   | not null
 log       | text                   | not null
 timestamp | integer                | not null default 0
 format    | integer                | not null default 0
 nid       | integer                | not null default 0
 title     | character varying(255) | not null default ''::character varying

…And it looks good!

We almost switched to WordPress for one site because of this error, but thinking about converting 5 years and 1250 posts from PostgreSQL Drupal to MySQL WordPress didn’t sound like fun. Fortunately the fix was easy, but nailing down the problem took some time first. Hopefully this helped someone else.

Echo Layla Working in Ubuntu Linux

The Echo pro audio products are great. I’ve been using an Echo Layla 24/96 since about 2002. I primarily do my recording with 64Studio, but I like to listen to music while I work on programming in Ubuntu.

I haven’t had problems getting it working in most versions of Linux for the last few years. Easiest install was recognized right off the bat by Mandriva. My memory doesn’t serve me too well, but 64Studio works flawlessly with record low latencies.

But it didn’t’ work right out of the box with Ubuntu, and that’s no big deal. Finding out how to get it to work was tougher. Lot’s of people complaining that Linux doesn’t “just work.” Well, when I installed Layla on Windows it didn’t just work either, I had to install a driver. Fortunately in Linux the driver is already included. What’s missing is the mixer and firmware.

It’s too bad but there are many solutions out there including compiling ALSA from scratch. But the solution I found over at the Echo forums was much, much simpler. Here’s the excerpt from I want to use Layla 3g in Ubuntu

1) Add the Medibuntu (https://help.ubuntu.com/community/Medibuntu) repository to your software sources.
2) Install the alsa-firmware package to get the drivers.
3) Install alsa-tools-gui package to get echomixer (a mixer for Echo Audio cards).
4) If you’re using PulseAudio you need to manually add it to your /etc/pulse/default.pa file *.
5) Reboot & enjoy high quality sound.

And that worked for me. Make sure after you reboot to have your volume down low on your speakers, then after it starts up load the echomixer program from your Sound menu, and adjust it properly.

No problem. You should have your Echo Layla, Layla24, and possibly 3g, Mia, Gina and other Echo pro audio cards working in Linux.

Upgrading PostgreSQL to a new major version

Upgrading PostgreSQL to a major version can be a challenge but isn’t difficult. You need to plan ahead and do some testing afterward, but PostgreSQL includes everything you need to make the change almost seamlessly. I give you a step by step rundown of how to make the upgrade and gain all the benefits that come with a new Postgres version.

Upgrading PostgreSQL

Upgrading PostgreSQL need not be hard. The creators have provided all facilities you need to upgrade nearly painlessly, and depending on the extent of your use, nearly seamlessly.

These notes were made when I was upgrading from PostgreSQL 8.2.6 to 8.3.0, but the information should apply equally to other versions. Keep in mind the version number are specific to these releases, and to substitute your own accordingly.

According to the PostgreSQL INSTALL doc, here’s their order of install (not upgrade), but the procedure is almost identical with the addition of backing up and restoring your old data.

./configure
gmake
su
gmake install
adduser postgres *for new install only
mkdir /usr/local/pgsql/data
chown postgres /usr/local/pgsql/data
su - postgres
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data >logfile 2>&1 &
/usr/local/pgsql/bin/createdb test
/usr/local/pgsql/bin/psql test

configure

First, run configure. I changed the install path to reflect the version so that I can run the two servers in parallel.
./configure --prefix=/usr/local/postgres-8.3.0

gmake

Watch for errors. Run gmake, not make. Most systems have gmake as a symlink to a capable compiler. Pretty straight forward. The email list is there to help if you have problems with compilation, which is usually rare. Googling for particular errors is also helpful, and will reveal a fix if you’ve encountered a bug or platform specific problem.

gmake install OR gmake install-strip

If you are not using the --enable-debug option then consider using gmake install-strip when installing. This should reduce the file size of the binary files and libraries as well as save some memory while loading. I use this on any non-development machine.

Make the /data directory

Straight forward here. Create the directory and initialize the new database cluster.

mkdir /usr/local/postgres-8.3.0/data
chown postgres /usr/local/postgres-8.3.0/data
su - postgres
/usr/local/postgres-8.3.0/bin/initdb -D /usr/local/postgres-8.3.0/data

While running initdb you’ll see:

The default text search configuration will be set to "english".
creating dictionaries ... ok

This is evidence of the brand new tsearch which is integrated into the core. Many people had upgrade problems with tsearch in the past, and for me tsearch is an important tool, especially needed for those people migrating from MSSQL, so it is a great step forward to see this included in the core now. Please see below if you were using the old tsearch contrib module.

Compile contrib modules

After you perform the install I would say pause here. Take a look at the contrib directory in the source. Are you using any contributed modules? If so, I would say this is a good place to compile and install them, before you go restoring your old database. If you don’t do this, and the contrib modules had you insert some SQL into your database, when you restore your dump the SQL will be present, but the module won’t, and you will get errors. Read the README file in the top /contrib directory for the procedure. It is usually gmake all;gmake install . Don’t do the regression tests now because you aren’t inserting the SQL code that goes with the modules yet. Be aware that regression tests try to connect to the default post, that means the original server, and will probably fail if you run the new server in parallel on a different port later on. From the manual for contrib modules:

After a major-version upgrade of PostgreSQL, run the installation script again, even though the module’s objects might have been brought forward from the old installation by dump and restore. This ensures that any new functions will be available and any needed corrections will be applied.

Also, if this is version >= 8.3.0 and you are using an older version of tsearch you’ll need to read the manual pages Migration from Pre-8.3 Text Search. Loading a pre-8.3 database that contained contrib/tsearch into an 8.3 will cause problems. It caused my restore to fail. The manual suggests loading contrib/tsearch before restore because it will prevent loading the old objects from the old database.

Edit postgres.conf and other configuration files

Do a diff on your postgres.conf and pg_hba.conf files with the new version. If moving between major versions I would edit them by hand. When moving between minor versions last time I just copied the old over the new as suggested, and found errors in the log at startup because a configuration option had been removed and I was still calling it.

Backup your old database

Seems like an odd place to be performing the backup procedure doesn’t it? Yes, but the PostgreSQL manual has a good reason for it:

To make the backup, you can use the pg_dumpall command from the version you are currently running. For best results, however, try to use the pg_dumpall command from PostgreSQL 8.3.0, since this version contains bug fixes and improvements over older versions. While this advice might seem idiosyncratic since you haven’t installed the new version yet, it is advisable to follow it if you plan to install the new version in parallel with the old version. In that case you can complete the installation normally and transfer the data later. This will also decrease the downtime.

The reason here is that the pg_dump or pg_dumpall binary from the new release has bug fixes over the old version. Now it makes sense. With that in mind I backed up to a file first, just as a precaution.

/usr/local/postgres-8.3.0/bin/pg_dumpall -p 5432 -U postgres | bzip2 -zv --best > pre_8.3_dump.sql.bz2

This command is using the NEW version of the pg_dumpall binary (8.3.0) and telling it to run on the currently running PostgreSQL backend (8.2.6) which is on port 5432, and store the output in a bzipped filed in the current directory.

Start new PostgreSQL server on a different port


su - postgres
/usr/local/postgres-8.3.0/bin/postgres -p 5543 -D /usr/local/postgres-8.3.0/data >logfile-8.3.0 2>&1 &

This starts the new server in parallel to the old running at the same time by using port 5543 for restore and testing. We’ll change it back to the default port once we know everything works right. Make sure you ps aux to see both servers working. The first time I did this the random port I chose for the new server was in use by something else and PostgreSQL refused to start.

Loading the data

There are a couple ways to do this. You can reload the dump that you just saved, or you can pipe the output of the pg_dumpall command into the new postmaster process that is running on an alternate port. I usually prefer to go with the pipe.

/usr/local/postgres-8.3.0/bin/pg_dumpall -p 5432 -U postgres | /usr/local/postgres-8.3.0/bin/psql -p 5543 -U postgres

This runs a dump on the old server (port 5432) and immediately restores it into the new server (port 5543) using a pipe to psql. Use the pg_dumpall and psql binaries from the new version you just compiled to gain bugfixes and performace benefits.

Remember after you load the old data into the new server to apply the SQL files for any contrib modules to your new database. You should also check the output of your server file (logfile-8.3.0) for any notices errors using a grep command like cat -n logfile-8.3.0 | grep -i error . Also vacuum the entire cluster after reloading:
/usr/local/pgsql-8.3.0/bin/vacuumdb -p 5543 -a -z

Switch servers

You could run the new server on the alternate port for a period of time with some of your test or staging applications to make sure everything is working properly and that any of the feature changes that may affect your software have time to be changed. If you do this, then just reinit the cluster and do the parallel dump again when you are ready. If you don’t you’ll lose any data inserted in that time – which could be days.

Otherwise, making the switch is simple. Shutdown both servers, and then restart the new server on the same port that the old one ran on. If you started the original PostgreSQL with an init script then shut it down using that init script, edit the script to refer to the new version, and use that init to start the new server with /etc/init.d/postgres start

Lastly, remember to change your environment to point to the new postgres version. This isn’t strictly required but if you use psql on the command line often you should set your PATH and MANPATH to refer to the correct location. If your postgres is installed in /usr/local/postgres-8.3.0 for example, you could use a symbolic link called /usr/local/pgsql that points to the currently installed PostgreSQL version. That way you won’t have to change profile, or setup scripts for web based interfaces, etc.

Upgrading PostgreSQL to a major version takes some time, but it isn’t difficult. If you employ this same procedure for your minor version upgrades you’ll find it become second nature. Practise makes perfect.

Forcing Redirect to a www URL with Apache Rewrite for SEO

Redirecting requests for the non www version of your domain to the www version only can help to boost your SEO, eliminate duplicate search engine entries and improve your ranking. And it is easy to do with Apache and mod_rewrite.

I registered a new domain about a week ago and then posted my new social networking style project to go live. I was still fiddling with server upgrades so I didn’t tell anybody about it. Sure enough Google comes crawling and takes all pages. Excellent.

I went to Google and performed a:


site:example.com

…and it returned all my pages. Excellent again, fully indexed in less than a week (but not showing in search results). Wait a minute. I forgot the www. I try it again with:


site:www.example.com

And Google returns one page – the doorway for my web host! Sometime after I registered the domain and pointed the DNS to my machine Google crawled the www. version of the URL. I don’t know how or where it got the link. Probably from a list of recently registered domains. Then after that week was up and I posted the full site, Google crawled the non-www version of the URL, and indexed both.

Anyone see a problem there? Well first, I’ve got a doorway page for my host showing up for the www version of my URL! Second, if Google crawls both versions on their own without any links on the net, sooner or later they are going to see duplicate content. Both versions will eventually be the same I hope. Third, they are crawling the two versions at different rates, giving out of date results for one of them. Last, they are splitting up the results for the domain, and I don’t see that being good.

A solution that has been proposed by others in the past, and one that I would recommend again, is to redirect all traffic from non-www requests to the the www host version of your domain, if that is the version you are using. If you are using Apache and have mod_rewrite you can do the following:


<IfModule mod_rewrite.c>

RewriteEngine on
Rewritecond %{HTTP_HOST} !^www\.example\.com
RewriteRule ^/(.*) http://www.example.com/$1 [R=301,L]
</IfModule>

I did this in my http-vhosts.conf file but I believe it should work equally well in .htaccess . See note at the bottom of the page for the differences.

The first Rewritecond line says for every entry where the HTTP_HOST DOES NOT EQUAL www.example.com …. then redirect them to www.example.com. This won’t work if you subdomains like subdomain.example.com because it will redirect those requests to www.example.com as well.

Very important, don’t forget the R=301 which is a seamless HTTP redirect using the status code 301. According to W3.org:

301 Moved Permanently

The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs.

So Google SHOULD remove old references to the non-www domain and all your future results should look the way you like.

Proponents of SEO will tell you this automatic redirect to the www version of your domain using Apache and mod_rewrite is good for ranking, elimination of duplicate entries and boosting your visibility. Give it a try if you haven’t yet and see how it works for you.

UPDATE: If you put this in your root Apache configs instead of .htaccess make sure you write ^/(.*) instead of (.*) or you may get a double slash between your domain name and pages which Google will also consider duplicate content.

Monitoring the Postgres Daemon

Following up on Running Postgres with Daemontools – Shutdown Errors, the Postgres mail list had some advice on monitoring and automatically restarting Postgres.

It seems the danger in automatically restarting the postmaster process from Postgres is a dangerous proposition. As mentioned in the mail list, if a process quits for whatever reason, daemontools will simply restart it. Now, that could theoretically fail and restart 3600 times per hour, every hour, every day without you ever being notified that there is a problem that requires attention. This may be fine for some things, but not Postgres.

Some people indicated they use a custom script to try and connect to postgres, and if the postgres fails they are notified. Some said the program monit will do the same job. I believe monit is quite extensible and will allow you to be notified as well as restart the server.

Still another person said they run the nagios monitoring program and are notified by email and pager of a failure.

Whatever the solution is, it seems only right to me that you should be notified of a failure if you so desire. At the same time, the mechanism that is handling the process should provide a clean shutdown. Apparently monit does that.

While I can’t advocate daemontools for Postgres, I would certainly say it is time to revisit nagios and monit. I’ve used nagio is the past but had resource issues with it. I’ve perused the monit documentation and enjoyed the flexibility. Now I’ll be looking at them both again and reporting back.

Running Postgres with Daemontools – Shutdown Errors

Running anything under daemontools seems like a great idea, but I’ve had a particularly bad time trying to shutdown postgres under daemontools while running Apache and PHP with persistent postgres connections.I had a VPS that was running postgres with daemontools and I just let it lapse because I wasn’t getting enough use out of it. Instead I focused on my main machine. When I went to upgrade Postgres I noticed that I didn’t have it running under daemontools.

I set to work getting it running. I even prepared a tutorial for you on running postgres under daemontools. I got everything working relatively easy using some of the how-to’s found on the net. Everything was working well and just as I was about finished I decided that bringing it up was good, but that I should verify shutdown.

I issued the required svc -d /service/postgres and everything went wacky. Logging in the log, every 3 minutes it kept saying “FATAL: the database system is shutting down’”. After examining ps output I found a defunct postmaster and a bunch of active connections.

After several hours of reading and trying I’m pretty sure I came up with the answer. From the postgres manual regarding pg_ctl:

In stop mode, the server that is running in the specified data directory is shut down. Three different shutdown methods can be selected with the -m option: “Smart” mode waits for all the clients to disconnect. This is the default. “Fast” mode does not wait for clients to disconnect. All active transactions are rolled back and clients are forcibly disconnected, then the server is shut down. “Immediate” mode will abort all server processes without a clean shutdown. This will lead to a recovery run on restart.

svc -d sends a TERM and CONT signal. It certainly wasn’t adding the -m option. But I did notice that the init script that ships with postgres (which was successful) performs the following on a ‘stop’ request:


stop)
echo -n "Stopping PostgreSQL: "
su - $PGUSER -c "$PGCTL stop -D '$PGDATA' -s -m fast"
echo "ok"
;;

In other words, the -m fast switch means it doesn’t wait for clients to disconnect. This is an important point if you run PHP with persistent connections. That explains why those process entries after svc -d were persistent despite the defunct postmaster. The svc -d /service/postgres was probably waiting on those persistent connections to disconnect. That’s my guess. Svstat kept saying it was waiting to shutdown.

Thus ends my attempt to run postgres under daemontools for now. Starting it was successful, reloading confiuration files was successful, but that’s where it ends. From a guy who has had both data loss and corrupt filesystems from non-clean postgres shutdowns, I’m pretty concerned what will happen when the computer reboots if it can’t disconnect those connections.

Or for that matter, what will happen if postgres shuts down, then supervise restarts it, but those persistent connections remained open. I’m betting Apache will be showing that it is unable to connect to the database server. I’ve noticed in the past that changing a users search path in postgres on command line (through psql) and then reloading a web page has absolutely no effect because the web server is holding open a persistent connection to which those changes don’t apply.

This will take some more examination, but for now it may be better to look at another tool such as monit, or consider getting rid of persistent connections.

Validating POST values from HTML in PHP revisited

In a recent article titled “Submitting POSTed HTML Forms and Registering Variables in PHP” we examined a better way to treat tainted POST variables passed to your PHP script from an HTML form. We decided to create an array called $FORM_FIELDS that would contain only the names of the elements that we wanted in our new array thereby cleaning up all fields in one statement (loop) and omitting unwanted fields from becoming part of memory.

In a recent article titled “Submitting POSTed HTML Forms and Registering Variables in PHP” we examined a better way to treat tainted POST variables passed to your PHP script from an HTML form. We decided to create an array called $FORM_FIELDS that would contain only the names of the elements that we wanted in our new array thereby cleaning up all fields in one statement (loop) and omitting unwanted fields from becoming part of memory or registered session variables.

The creators of PHP went one further in this step by making the $FORM_FIELDS array a multi-dimensional array. They added a couple fields in this array to represent the data type and an additional function to process the data.

The array may look like this:

$FIELDS = array(
'country_id' => array('type' => 'int', 'function' => 'addslashes')
);

Now when you process your loop as done in the previous article you would just add a line for the datatype:


if (isset($FIELDS[$name]['type']))
{
$tmp[$name] = settype($FIELDS[$name], $FIELDS[$name]['type']);
}

The reason this is important is incoming values from an HTML form all seem to be of type ‘string’. If you are inserting this information into a database or otherwise performing calculations on it then it is nice to know it is of the correct data type that your functions expect.

But there’s more. If someone is attacking your program by sending POST values different from those you expect or different from those in your form then this will help.

You simply set the data type before you perform validation. If you have a field where a user selects the country name from a drop down list box and the value sent to your form is the country_id then you would pick ‘integer’ as the data type. But what happens when a user sends a string? If you try inserting it to a database you get an error. If you try to compare it against a list of known country_id’s for validation purposes you will also get an error.

If you perform the settype() function in PHP before your validation you will either set it to the correct datatype or get a value of zero. If a malicious user sent a string of ‘Aruba’ for your country_id field and you cast it to an integer you would get a zero. Anything other than an INT would become a zero. You can then proceed with your validation which would presume that a value of zero is not valid.

You will want to take a look at the PHP manual section called “Type Juggling.” There are certainly some caveats and you may not expect some conversions to be logical. For example, if you had the string “10thingsIhate” and you cast it to an INT you would have int(10).

Brush up on the PHP types. They can save you some work in your validation of HTML form input in your PHP scripts.

PHP’s APC cache and how it relates to Apache Bloat

You just finished your install of Apache, PHP and APC cache on a VPS and you’re beat. You spent about a day going over every configuration and compile option trying to eek performance out of your VPS machine and web server. Then you take a look at the output of top or ps and notice, hey, my Apache process is 150 Megs! Well isn’t that grand. Compiling the Big Apache on a limited resource VPS can be a challenge for those of us who like to tinker. On the last VPS I got I scrapped the packaged stuff and started on compiling an Apache for the machine. The problem is trying to meet your needs with the limited memory you usually get on these machines. Although I like lighttpd, I’m not ready to give it a go on a production server that is being monitored by the media all the time. I need something as guaranteed as I can get. Although lighttpd may be such a beast, I’m not going to test it out on this site at this time!

I sat down and started to go over all configuration option for Apache, PHP and APC to get a nice small package to do the job. I was surprised how much junk PHP has compiled in by default. Compile it once and execute a phpinfo() and you’ll see. Just start with the “–disable” switches for anything you don’t want.

After about a day of thinning out the size of my stack I slipped into TOP and used the SHIFT-A toggle to see the alternate views. I noticed something odd…..

3  PID %MEM  VIRT SWAP  RES CODE DATA  SHR nFLT nDRT S  PR  NI %CPU COMMAND
 24140  0.1 78592  69m 7228  504 2124 2952    6    0 S  16   0    0 httpd
 11329  0.1 78408  69m 6908  504 1964 2736    2    0 S  16   0    0 httpd
 24141  0.1 78564  70m 6752  504 2120 2492    0    0 S  16   0    0 httpd
 24142  0.1 78564  70m 6732  504 2120 2476    0    0 S  16   0    0 httpd
 27813  0.1 78560  70m 6684  504 2116 2432    0    0 S  16   0    0 httpd

Doesn’t that seem like a lot of VIRT and SWAP for Apache2? Yes, it is. If you used the suggested 128 for apc.shm_size it will be a lot bigger too! I used 64M and you can see that at 78592 minus the 64M shared mem for the APC cache, the actual size is around 13M for apache with a 6.7M RES. That makes me feel better! I was so worried I had gotten something horribly wrong.

If you happen to run across this kind of thing in testing, simply set your apc.enabled in php.ini to ’0′ to disable it, restart Apache and check ‘top’ again. Likely nothing to worry about.

The machine I am running is only hosting 4 sites so 64M is a good starting point, but what you should do is copy the file ‘apc.php’ from your APC source directory into a folder on your webserver and visit it frequently to see how it behaves. Depending on your other settings like the cache lifetime and garbage collection, if you have a large portion of FREE and a small USED then you can probably set the apc.shm_size lower. If it makes you feel better :)

PHP 5.2 and Drupal 4.7.4 don’t work together

While doing a software upgrade on a server today I ran into some problems. Apparently Drupal 4.7.4 and PHP 5.2.1 don’t work together.

After upgrading PHP, Apache and APC cache I couldn’t get Drupal to work on all 7 sites on one machine. I could login to any drupal site as the admin and get shown my user page, but when trying to do anything at all as an authenticated user it considered me logged out.

I initially thought APC had something to do with it and began to disable it. That didn’t fix it. On a haunch and after a lot of reading with no answer I decided to install a lesser version of PHP and that fixed it. PHP 5.1.1 was laying around and did the trick. Unfortunately I needed 5.2 to fix a mem leak!

I decided to update the Drupal sites to the latest version first to see if that helps. Lo and behold when I visit the Drupal download page I am greeted with:

PHP 5.2 compatibility is only available from Drupal 4.6.11 / 4.7.5 / 5.x.

That pretty much cinches it. I need PHP 5.2 to fix a memory leak so I’ll take time to upgrade to Drupal 5 now.

Be warned, if you are running Drupal 4.7.4 sites and attempt to upgrade to PHP 5.2 you may be faced with being unable to login or navigate as an authenticated user in Drupal. Downgrade to PHP 5.1, upgrade to > Drupal 4.7.4 and then continue your upgrade to PHP 5.2 and you should be alright.

Apache, Postgres and Drupal on a VPS

I really would prefer to have my own server but sticking a box in colo is expensive. Where I live, getting access to the physical colo space would be nearly impossible too. As a result I run on a VPS. Unfortunately VPS has some horrible limitations depending on who is on the box with you.

Recently I decided to move my personal blog off of b2evolution and stick it on Drupal. Too bad drupal is such a resource hog. Most CMS and blog software is though and it is really hard to find a minimalized, optimized blog software that uses any form of caching. Today, it hit the skids and my patience hit the wall. Argh!

I was converting my personal blog by hand because I only have about 30 entries so it didn’t pay to write a conversion script. Everytime I hit the ‘post’ button in Drupal I wound up with a blank screen, could not connect or worse, the results of any command in SSH terminal window showed a “could not allocate memory”. As a result, I had to do some fast tuning of something because I had to reboot the server after every blog post!

I chose to tackle Apache first because they have an Apache Performance Tuning Guide that helps a bunch. Postgres, well I’m running an old version that I really need to upgrade before I tackle configuration and optimization of it. That’s not a 30 minute job.

VPS, Apache and Low Memory

VPS has a low memory for sure. Even though you can sometimes burst more, in general it is pretty low. The very first thing in the Apache performance tuning guide is tackling memory.


You can, and should, control the MaxClients setting so that your server does not spawn so many children it starts swapping. This procedure for doing this is simple: determine the size of your average Apache process, by looking at your process list via a tool such as top, and divide this into your total available memory, leaving some room for other processes.

Using top and sorting by thread using ‘h’ I am able to see that the average Apache processes is using WAY TOO MUCH memory at 25M a piece – and 10 processes running. I don’t have time to tune the size now so I’ll tune the number of servers using very simple configuration parameters. Since we are using MPM Prefork, the directives can be found in extra/httpd-mpm.conf file under mpm_prefork_module.

Since I am supposed to be guaranteed 256M memory burstable to 1G I’ll optimize to the lower number. 256M / 25M is 10. Not including room for other processes. The current setting is 150!

From:


StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0

To:


StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxClients 10
MaxRequestsPerChild 0

I also changed it to only start 2 servers, only keep 2 spare servers instead of 5, and only allow 10 clients instead of 150. This will essentially create a queue if anyone is waiting but it shouldn’t dip into swap space and it will save a bunch of memory usage. This will of course be monitored. Once time permits and I am able to minimize the size of each Apache2 process (the other factor) then I will revisit this and likely increase the MaxClients accordingly.