Mohamed Ibrahim’s Blog

Since I updated the website this weekend and it’s been downloading images from MRIS non-stop. In a previous post I said that I had around 900,000 images. I later on discovered that I had dangling images whose listings were deleted. Since this weekend, the update script downloaded almost 820,000 images from MRIS and inserted those binaries inside bytea fields in postgresql. I was expecting things to slow down, however I don’t feel any slow down since the process started. It seems the postgres team really optimized the b-tree indexes to the extreme.

The homepage gives quick stats and the number of images and virtual tours is actually the number of rows in the media table. I’m anxious and looking forward to experiment postgres version 8.3 when it gets released.

Another non-expected result was the stability and efficiency of the Linux kernel. Currently I’m updating a county every 20 minutes. Since the media table was basically empty, every update process will not finish in 20 minutes and they will start to overlap. For example Fairfax county has approx. 6000 listings. Some of those listings has 30 images in three resolutions - that’s 180 images only for one listing. The number of concurrent processes hammering postgresql and using the disk i/o rapidly grows. Other than apache which is running three other websites hosted on the same machine and also using the same database. Each has its own wordpress weblog that uses another database (mysql) also running on the same machine but with very limited features just to cover the weblogs. I honestly expected that when I wakeup on Monday the machine will be dead - but I found that it is still working, and the website is actually responding very fast even when I tested it from outside.

So still till today many images are being downloaded, and I expect that to continue at least another day! I’m still working on the older bugs, and from time to time adding a new feature that I expect to rock the website.

Technorati Tags: postgresql, bytea, binary fields, images, million rows, performance, hosting, website

Tags: binary fields, bytea, hosting, images, million rows, performance, postgresql, website

This entry was posted on Tuesday, January 15th, 2008 at 8:29 pm and is filed under Google / Internet / Programming, Linux. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.