Website updates
A lot of work was done this weekend. Although I refer to it as a lot, yet the amount that is actually reflected on the pages is small. I changed the way mibrahim.net is reading the media from MRIS to a better way, since MRIS update of photo manager.
The old sneeky way that I used was to increment the last digit in the main photo URL. It worked great for most of the listings, and in very rare cases it did not work. After the photo manger update, I noticed that the number of misses started to increase and by further research I found that MRIS now stored more images in the database - i.e. there is no weblink to the media and those images can only be retrieved using a RETS query.
Luckily, the media URL stored in the media entry is actually the RETS query, and all what we really need to do is to login to the RETS sever, issue the query. A caveat that they didn’t warn us about is although the server replies with a mime type of image/jpeg yet the data in the http response is not a jpeg image. Actually the data is formatted in RETS chunks and needs to be filtered before being used as a jpeg. I figured that out after little bit of trials/misses but finally got it sometime on sat.
So from my side, and after running into a lot of referential integrity problems from images being stored in local directories, I thought I’d try storing them in postgres bytea and test the performance. The results amazed me, unexpectedly I found that it is not as slow as I previously thought and it does not overload the cpu either.
I updated the listing pages by focusing more on the media. Now the media is pulled up before the fold, and every picture has a text title. Also, I now use cookies to view larger images, which does not break the back button and that’s one big usability improvement, which will be totally invisible and unseen. The older way had a post for every click that viewed the larger image.
Some glitches are still there esp. on the main page. I believe the javascript tabs were never meant to be used in that number 
when you look at the main screen, and start flipping between counties a tab mess will be generated. I should work on that tonight, and try to figure out a better way of organizing that information in one screen.
Another thing that really suck is virtual earth. It does not work on opera! After I modified the code to show/hide the map, IE didn’t like it - and although Firefox and safari both renders it correct, IE does not even load the page, and generates an error. I commented the maps code yesterday before I sleep, and I’ll have to deal with that tonight.
Technorati Tags: mris, rets, media, images, database, storageTags: database, images, media, mris, rets, storage
January 15th, 2008 at 8:30 pm
[...] downloaded, and I expect that to continue at least another day! I’m still working on the older bugs, and from time to time adding a new feature that I expect to rock the [...]
February 19th, 2008 at 4:38 pm
Nice solution!
So, do you write unique html pages every day, or do you only grab the homes that are updated? If so, which field do you use to scrub for homes that are updated?
February 20th, 2008 at 9:41 am
My school of thought is different from what you’re saying. To be precise, a way to design a big website is by running the scripts that generate the pages, and storing the HTML daily. And I guess that’s what you imply by “writing unique html pages”.
The problem with that approach is that you will never be able to personalize the results - or for example show a shopping cart to store selections, since your content is now static. Unless you have an ajax driven part.
From an engineering point of view, I would focus on providing functionality until I have a performance bottleneck - and at that point the correct approach is to cache some of the slow query results since the database usually consumes 90% of the script run time. Another approach is to speedup the database. Since the disk is what slows it down ( you can identify that by running atop ) the straight solution is to speed up disks - use RAID !
A third approach is to use faster storage engines, like BerkelyDB or derby instead of postgres or mysql. That would require more work on the code side, but will produce faster pages without changing the hardware.
If the CPU is a bottle neck then just cache some of the slow queries as I described earlier - however, you would really need a significant load with super fast RAIDs to reach that point.
So my school of thought is to continue using dynamic pages.
For updating listings, I compare the ModificationTimestamp in the Property search type. A similar time stamp exists with media.