28 Nov 2008
From Colourphon
Richard Francis says: I've just done some reading up - and we might want to time the difference of using memcache to mysql ... basically what it said was on a one to one basis mysql prob is quicker ....
Richard Francis says: but if we redefined our objects as being a recently submitted image (i.e we have a key of an ISBN and all data is brought back without having to analyse everything may be a better model
Richard Francis says: So 1 image is checked to see if it is memcache (with a suitable timeout) -> if not normal analysis occurs with a mysql back end -> when a full image has been scanned the whole image object is put into memcache for fast retrieval
Richard Francis says: then if that image is scanned again - it instantaneously gets retrieved
Richard Francis says: although it might not be quicker completely ... therefore we stick with the memcache model as is,
Richard Francis says: but maybe add a 10+ minute timeout on the whole image object
Richard Francis says: therefore any second run is v.fast
Tim Hodson says: well, we could try it. but if we make the assumption that while harvesting the chance of us wanting to lookfor the same isbn twice is slim... then we still need to make sure the mysql could cope with the number of requests for the colourProfile info. 1 per pixel.
Richard Francis says: i was thinking more of the fact that someone testing might submit an isbn several times
Richard Francis says: yeah, it basically said that some models work as is
Tim Hodson says: sure, and agree that this would be a good way to protect the server load from multiple requests. I think we could serealise the data returned from scan)image.php and cache that.
Tim Hodson says: was this online reading?
Richard Francis says: yeah, http://www.socialtext.net/memcached/index.cgi?faq
Richard Francis says: also added comment to blog post Richard Francis says:
Tim Hodson says: already authorised it
Richard Francis says: I think the argument is that memcache might be faster if you are doing complex joins backend
Richard Francis says: a particular object might be the result of those complex joins Richard Francis says: therefore is faster
Richard Francis says: but maybe a simple select on a flat table is quicker than memcache
Tim Hodson says: yes. - well, depends how big that flat table is . 16.7 million rows? Richard Francis says: well, I thought about that too ... and wondered what the performance impact on memcache would be too
Tim Hodson says: memcache would be the equivalent of 16.7 million tables of one row... hmmm.
Richard Francis says: yeah think we need to be careful anyway - it's good we're writing to csv - we could quite easily convert to sql
Tim Hodson says: but I think the way memcache works, it already knows where the data is for retrievel, whereas mysql would have to read the table and look for the location of the data.
Tim Hodson says: yes. I am already thinking that I can halve the amount of memory we are using by redesigning the way the data is stored.
