WordPress v. 2.2 Upgrade Exposes Cracks in the Foundation

Can this Popular Blogging Software Mature to World Class?

I have just gone through more time than I care to admit upgrading to version 2.2 of WordPress. I did this not only because the WP developers have signaled their intent to abandon 2.1 with a more formal upgrade and release schedule. I also did it because I was beginning — with site and user growth — to experience some noticeable performance issues. And, I also wanted to give the site its second-year anniversary tune-up.

Since I had upgraded about four previous times in the past two years (my earlier and popular Professional Blogging Guide was written for version 1.5 of WP), I pretty much assumed this next upgrade would also be generally smooth. Boy, was I wrong!

This whole experience, though not completely disillusioning, does raise some real questions and concerns. Likely some of it has to do with my own stupidity or neglect or the fact I am not a system admin. But those same factors apply to the majority of WordPress users that administer their own sites as well. If you fit in this category, Watch out! You may be sailing unawares into a perfect storm.

Standard Upgrade Instructions

Just for reference, I have a virtual dedicated (private) server (VPS) with 256 MB of RAM burstable to 1 GB. I’m running Linux CentOS 4, Apache 2.0.52, MySQL 4.1.20 and PHP 4.3.9. In other words, my VPS is a fairly typical LAMP installation. My VPS plan provides no support, but I do have a Simple Control Panel (Turbopanel), though most of my management occurs through SSH. Disk space and bandwidth are not issues, though I sometimes suspect those sharing the server are resource hogs (doesn’t everyone with a VPS suspect so?; at least it is not a standard hosting arrangement!).

As with past upgrades, the instructions for upgrading WordPress have become fairly standard. You can download WordPress version 2.2 from here and the WordPress folks offer pretty clear initial installation and upgrade guides. Read these and follow them.

If upgrading, it is critical that you backup your database and backup your relevant site changes (including theme changes and plug-ins) before attempting any installation.

(Though I personally did not have to resort to a full restore when I encountered the problems noted below, it was close. And, in any event, if you are able to put up with some downtime, even if everything goes to hell in an basket, you can still get back to your original configuration. Back up!).

The actual process of upgrading went pretty smooth for me. Physically copying files and getting the site to come back up wasn’t the problem. The problems arose in how the site (mis)behaved after the upgrade.

Problems and Inadequate Answers

For many reasons related to (a perhaps too-long list of plug-ins) I had no end of problems with this WordPress v. 2.2 upgrade. Here are the key ones I discovered:

1. While WP v. 2.2 may be more efficient overall with 200 bugs fixed, claims I have not frankly been able to verify, its initial resource use has gone way up. I suspect this is because of the more aggressive use and addition of more JavaScript, but there may be other explanations as well
2. One net upshot is that if you have been a user of many plug-ins in the past, you may find you have inadequate resources for the upgrade
3. You may therefore see hard-to-diagnose behavior due to too many plug-ins. Tested individually, the plug-ins may look fine; in combination and in weird ways, you may see problems (for myself, it showed worse on my Chronological Listing page using the wp_get_archives call). White screens or partial page renderings are a telltale sign
4. Many plug-ins are simply no longer working (this may have occurred in version 2.1, but I had not upgraded since version 2.0.5)
5. Watch out for plug-ins that add tables to your MySQL database, such as stat packages or spam blockers. Not only do these insidiously cause your database to grow, but they introduce another possible point of failure and synchronicity issues with caches
6. The visual editor (TinyMCE) as implemented in WP cause DIVs to be deleted in code without warning when switching back and forth in code view; nasty! there are other editor weaknesses as well (see below)
7. Caching, either through individual options, or more especially in combinations, work horribly! I had numerous Apache race conditions where load averages headed through the roof bringing the virtual server to a complete standstill (including the inability to log in with SSH!)
8. Many prior guides regarding performance tuning no longer seem useful or applicable, likely again to the JavaScript additions and other fundamental changes to the internal “loop”
9. With or without caching, I was seeing anomalous and episodic loading delays and racing conditions with Apache/MySQL when using the internal Feedburner scripts
10. Various other documentation and general project problems (also see below)
11. A general lack of understanding of what or where to look regarding performance, configuration or (frankly, virtually) many of the operative aspects of my actual blog site, and
12. Being attentive and investigating these matters has made me a better administrator by becoming familiar with existing reporting, monitoring and testing utilities already extant on my system or easily added to it.

More on some of these points is expanded upon below. A very helpful compilation of WordPress v. 2.2 issues and tips and resources is provided by the Blog Herald.

Now, I somewhat violated one of my personal cardinal rules for WordPress by upgrading at a single sub-version number. (I actually don’t understand or track the rationale for WordPress version upgrades; 2.2 was huge, as was 2.1, seemingly as great as the transition from 1.x to 2.x!). Somehow, however, I had the impression that 2.2 was a simple increment over 2.1, which did have a number of sub-sub releases. I did, however, wait a couple of weeks after initial release.

I suspect there will be a 2.2.1 release very soon, because the scope of issues that are emerging are pretty big. One source of new fixes will be hosting providers; I’m sure they are seeing some pretty bad stuff recently. I’d hate to own a hosting service at the moment with many WP users and have to both field their calls and wonder what the heck I have to do with my current server infrastructure to get all of my users back up to snuff!

If you don’t have to upgrade immediately, my advice would be to wait. There is quite a bit that needs to settle down, and some of the plug-in developers I believe have some big efforts ahead getting their stuff stable again.

Specific Areas Warranting Further Discussion

When all hell breaks loose and panic takes hold, it is easy to loose perspective. Plugging holes in the dike, while potentially effective, can also divert and exhaust from identifying and then diagnosing real root causes.

Truthfully, I don’t understand at all with documentation, code forums, and hundreds of thousands of WordPress Google results why it is so difficult to get authoritative answers to many questions and problems. Moreover, hosted sites versus self-administered ones and dedicated v. virtual v. shared servers all have different needs and configuration options.

So, let me add to the general background noise . . . .

Non-working Plug-ins

There is a page on the WordPress site that lists known working and non-working plug-ins for version 2.2. While useful, this list is incomplete.

I followed the general advice of individually re-activating and testing each plug-in after the upgrade. However, because of caching issues and general load problems due to using internal Feedburner scripts, some of my testing may have indicated a plug-in was not working when it was these interaction effects that are the problem. With a starting roster of about 25 plug-ins (see figure), the potential interactive and combinatorial effects are huge!

[Click on image for full-size pop-up]

In my testing, I believe these plug-ins to not be compatible with WP v. 2.2: Advanced WYSIWYG, COinS Metadata Exposer, Customizable Post Listings, EzStatic, Kimili Flash Embed, Most Popular posts, and Smart Archives.

I strongly suspect these plug-ins are not working properly, and have de-activated in any case: ImageManager, Popularity Contest, Post Teaser, Spam Karma 2 (WP site indicates it is OK, but see below), StatTraq, and wp-cache.

This is more than half of my initial plug-ins that showed likely problems. As the figure above shows, I am now only using eight plug-ins, one of which is new that I updated myself (the Advanced TinyMCE Editor, discussed in an upcoming posting).

Of course, I could be wrong on any of these plug-ins due to the testing and combinatorial challenges noted above. I do not mean to impugn any individual plug-in. Indeed, I installed them initially because I perceived them to provide value. So, if I have erred in my assessment, I apologize in advance. But no matter how measured, compatibility is a real concern.

Too Many Plugins?

There has been discussion for some time regarding the impact of the number of plug-ins on WordPress performance. For example, a listing in early 2007 from Alex King’s blog seems to have the largest nucleus of discussion. The earlier consensus was that too many plug-ins may be an issue, but 30 or so were OK. (Naturally, some plug-ins are more resource-intensive than others.)

Frankly, up until now, I really had not given the question much thought. I would encounter what looked to be a useful functional addition, install and activate the plug-in, and go on my merry way.

The tip off that something might not be right arose when my chronological listing of posts broke after the upgrade. My first iteration of this page used the wp_get_archives function, often used for archive listings in sidebars of WP blogs. (I later used the Customizable Post Listings, which is a very nice one, but it is broken in 2.2.)

What I stumbled upon late in my upgrade testing was the observation that as more plug-ins were added to my system, the number of listed archive postings would decrease (also losing the footer). Under conditions of too many plug-ins, the entire page would go blank.

It was total serendipity seeing this behavior, and only occurred after much hair pulling and gnashing of teeth as to why seemingly OK plug-ins individually or in small combinations worked fine, but when all invoked worked to trash the site.

I suspect some big bug resides somewhere in the system related to resource use. My temporary fix has been to set an absolute limit in the ‘postbypost’ parameter in the function that is well below my actual total post counts. Go figure.

Stat and Spam Packages Not Worth the Overhead?

When everything appears to be broken no stone goes unquestioned or unturned. Because of their complexity, I spent considerable time looking at my statistics and spam plug-ins.

One possible cause for degrading site performance was a rapidly growing database. I had been getting complaints from users regarding slow load times in the past couple of months. I really did not want to bite the MySQL optimization bullet and tuning, but could avoid it no longer.

The first thing I had noticed was that my MySQL database had more than doubled in size in six months to 368 MBs! (43.5 MB zipped). I knew I was writing some long posts, but that seemed ridiculous.

So, while I had been faithfully backing up my site, I tended to avoid probing into MySQL. Faced with the unavoidable, I happened to notice, however, that 280 MB, or 75%! of my database size was due to storage of StatTraq statistics. (I know, you idiot, why didn’t I see this sooner?) I was blown away about how this silent accumulator had become the eggplant eating MySQL due to my inattentiveness.

While I had earlier looked into StatTraq speedups thinking that might be a cause of some of my performance issues, I frankly was shocked at the resource demands of this application. In fact, the only reason I had installed the application in the first place was in order to extract the minute amount of data for providing a popularity listing of prior posts in my sidebar. This was the result of an early decision when I first set up my blog and one I had not questioned or re-considered since.

That raises the broader question of why embed stats in your internal blog? You can get the same basic tracking and statistics from Google, Feedburner, Sitemeter, Stat Counter, your own internal Webalizer on most virtual servers, etc. Now, with experience and reflection, it is absolutely crazy to burden WP and MySQL further when such services can so easily and freely be obtained elsewhere.

I still liked the idea of a popular posts listing, and tried a couple of alternatives, including Alex King’s Popularity Contest. However, once I saw that these systems, too, added tables to MySQL, I decided to swear off the sweets and let go of being enamored with popularity listings.

So, with the removal of internal stats, my database reduced to a more manageable 80 MB (7.5 MB zipped).

Now on the war path for insidious MySQL sneaks, I next looked into Spam Karma 2. Actually, this plug-in has performed great for me, but again, it is complicated and writes considerable data to MySQL. In fact, even with new trimmed database, SK2 was occupying about 25% of total database space!

I presently have SK2 disabled, and am getting surprisingly good spam control with the simple use of the Comment Timeout. It seems to take some time for spammers to harvest new posting listings, so, if you are not a comment heavy site (as is the case for this AI3 blog), then timing out comments on older postings looks to be quite effective with not nearly so much overhead.

I haven’t yet removed SK2 tables, and may choose to re-activate it again in the future. But I am definitely now in the mood to question the operating profiles of statistics and spam plug-ins.

Caching

In the past week, I’ve researched and tested Apache server settings, MySQL queue caching, WP internal caching, external caching plug-ins (WP-Cache 2) and PHP opcode caches, alone and in combination.

Truth be told, despite coming across a few guides that appeared authoritative, I’m not sure that all systems work well with WP v 2.2, some definitely have issues (WP-Cache 2), combinations can produce race conditions, and, in general, nothing feels stable enough at present to use. I will likely need to await 2.2.6 with specific announcements of compatibility made by specific cache mechanisms before I try using them again.

Server Resources

Of course, the 256 MB of RAM item above should have caught most everyone’s attention. I know my current RAM is at the bare minimum, and perhaps most of the issues I’ve flogged in this post could be solved in a rich RAM environment.

I will look into this issue, but I’m also cheap, cheap. So, RAM and support and a host of considerations will go into any future hosting provider changes. Who knows, I might even choose to host myself?

(Actually, I need to get back to work. After the experience of the past week, I’m itching to let my blog site run as configured for quite a while before I attempt any major new changes, let alone migrating to another provider!)

Visual (Rich-text) TinyMCE Editor

TinyMCE is, IMHO, a fantastic JavaScript based WYSIWYG editor that has matured nicely over time with a good feature set and stable operation. However, the philosophy and implementation to adopting this editor by the WordPress team simply sucks! I repeat, sucks!

A posting later this week will address this topic further.

Longer-term Cracks in the Foundation?

I still (painfully) remember my transition from WordPerfect to MS Word, necessitated by market and technology shifts. But I loved WordPerfect and knew how to make it sing. I’m sure I’m not as proficient with WordPress, but I have come to understand and work around its quirks sufficient to be pretty productive. Yet, based on current trends, I’m fearful those days may be coming to an end.

There is an overarching set of issues behind the WP project that is starting to become apparent to me. Though I have used and recommended the system for two years, and have written one of the popular guides about how to roll your own professional WP blog site, here are those troubling trends and gaps I am seeing:

• I’m actually pretty disappointed with both the pace and bugginess of WordPress upgrades. As my site has gotten more complicated, so have these upgrades and the unwelcome downtime and diversion of working through (often) poorly tested releases. Actually, I have been getting in the habit of skipping releases and then only upgrading after a major subversion number has come out (2.2.1 or 2.2.2 versus initial 2.2, for example). Unfortunately, in my current case, I was having some performance problems that required tuning and other caching adjustments. Why can not WP have upgrade approaches with the ease and professionalism of Mozilla and Firefox?
• The WP plug-in system, if one can call it that, is a nightmare. Though there are guidances for how to write a plug-in, there is nothing that might be called a formal API. Registration procedures are non-existent; all one need do is enter a few lines of text in the head of a PHP file. And, there is absolutely poor resources for listing available plug-ins (WordPress has tried its own plug-ins directory and is making slow improvements, but the Weblogs Tool Collection is still better, though incomplete.) Again, where is the elegance and togetherness shown by the Firefox/Mozilla add-on system and registry?
• While I may be off on its technical feasibility, it would be attractive to be able to better internally configure WP and its resource use. There are literally no configuration parameters that can be set in WP, other than poorly documented options such as ENABLE_CACHE. What it does and why or how is unclear, let alone other such possible settings. The WP user must go into multiple external apps including the OS, MySQL and Apache (or other Web server; what are the supported options??) and attempt via trial-and-error to do performance tuning
• Resource guides and documentation are unnecessarily poor, especially regarding system configuration, performance and tuning, and
• Obsoleted code documentation is kept online well after updates, and professional APIs are non-existent.

Granted, perhaps it is unfair to compare WP with Mozilla and its installed base of perhaps 100-fold more users. Yet, on the other hand, WP has been around for quite some time and has millions of its own users.

Without a doubt, most would observe (and I would agree) that some of the upgrade difficulties covered in this article were of my own doing. Add a plug-in here, one there, keep blogging, and only occasionally dig out those reference sheets with the hidden steps, logins and passwords to speak with such things as MySQL, phpMyAdmin, Apache, “root” Linux access and the rest. Guilty as charged.

Yet, on the other hand, writing and maintaining a blog is a means — not an end — for me. Sure, I can learn this stuff, spend the time, and even blog about it on occasion. But that is not my daily focus or passion.

It is one thing to set up a blog, try a few posts, and then move on. Literally millions of people have done so.

Yet there are also many hundreds of thousands of content-rich sites that have become more-or-less permanent and often destinations in their own right. As the space matures, so must the platforms upon which it is based. I hope that WordPress will continue to be that choice for me, but some key structural work on its foundation is well in order.