Scraping Google Shopping for Magento or any Ecommerce platform

If you advertise via Google’s product listing ads and sell products that are commercially available elsewhere , it’s key to be competitive on price, one way of doing this is through Google Shopping as all competitors who also advertise on Google are of course publishing their prices and most do. If your business is very price driven or there’s some lines you would like to be competitive on then this is a unique way of keeping up with your competition.

The module can work in a variety of ways or be configured to work in a way that suits your business , I would be able to integrate it into other ecommerce platforms.

The main functions are:

  • Lookup a product on Google Shopping record pricing data
  • Price matching feature beat or match a competitors price – Automatically set the product price
  • Define which competitors you need to match or beat
  • Other conditions e.g not going below the product cost price
  • Works for grouped or single products
  • Trigger other functions upon product save
  • Receive a report of what products where changed, when and which competitor
  • Display competitor prices vs your price on the product view

Integration with Google Shopping API:

As well as updating the prices on your store, it can now also update Google Shopping directly via the Google Shopping API. Without this you may run the risk of the product listing ads being pulled because the prices in your feed are not up to date.

Of course the above runs on auto pilot and can run once a day or however frequently you need it, a daily file can be uploaded/emailed to your site

Contact me to find out more about this or discuss a project you have in mind that involves scraping google for Magento Ecommerce.


Magento 1.9 upgrade template changes needed

The main things I’ve found on upgrading:

Form Keys
Form keys are required on all the forms so in update cart,login,register etc…

 <?php echo $this->getBlockHtml('formkey'); ?>

Checkout SSL
If your on Nginx and have the redirect issue using SSL it’s because you forgot to add this in index.php

 if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
 $_SERVER['HTTPS'] = 'on';

Custom templates not loading

Make sure they are surrounded by the <cms> tag in your config.xml


Passwords disappearing
I had to enter all the configuration password info again e.g for Mandrill, Fishpig wordress, centinel 3ds etc.. Not sure if that’s because of the way I upgraded

Payment Gateways
Even re-entering and saving the data did not work here , I had to delete all data > save and then re-enter the data.

Review step not place order not working

Make sure your template looks like this

review = new Review('<?php echo $this->getUrl('checkout/onepage/saveOrder', array('form_key' => Mage::getSingleton('core/session')->getFormKey())) ?>', '<?php echo $this->getUrl('checkout/onepage/success') ?>', $('checkout-agreements'));


Skip review step magento 1.9 UK PayPal

I’m not sure if i’m missing a setting somewhere but it appears to me that in the UK version e.g if you have Website Payments Pro Payflow Edition (Includes Express Checkout) enabled, this does not skip the review step as per the 1.9 update suggests.

You can find the setting in /app/code/core/Mage/Paypal/etc/config.xml

By default it is set to 1


and in  /app/code/core/Mage/Paypal/Controller/Express/Abstract.php around lines 194 it checks for this parameter.

if ($this->_checkout->canSkipOrderReviewStep()) {
} else {
$this->_forward('placeOrder'); // skip review steps for all checkouts
// $this->_redirect('*/*/review'); // default

The only possible thought is that I am using the crius skip step 1 and that is changing the name of the checkout. I haven’t the time to debug fully at the moment.

 public function canSkipOrderReviewStep()
        $isOnepageCheckout = !$this->_quote->getPayment()
        return $this->_config->isOrderReviewStepDisabled() && $isOnepageCheckout;

As the functions only check is for the config.php function isOrderReviewStepDisabled() which in turn checks the configuration value, which we hope is set to 1. So as a quick hack I commented out the review and changed it to placeorder as above.

Kind of annoying that they didn’t include this as a configurable option from the admin area and that it does not seem to work in the UK versions. Anyways that’s how to get rid of the place order/review step in 1.9.

If your using 3ds centinel then you also need this:

in review/info.phtml to auto place the order after the customer successfully completes the 3ds secure authentication

$centinel = Mage::getSingleton('centinel/service');

if($centinel->isAuthenticateSuccessful()==1) {?>;
<? }

Responsive design for Magento Checkout

One part of Magento that performs particularly poorly on Mobiles is the use of tables mainly because there just isn’t enough room to see everything in a legible manner.

Some clever people have invented ways to overcome this through scrolling tables or but for ecommerce I don’t think it works well as the information often important information is hidden without scrolling the table.

For example on the cart table you typically would get something like this


Which on mobile looks like this


I find it’s easier to have a hidden alternative layout for mobile



@media screen and (max-width: 800px) {

#shopping-cart-table {

#shopping-cart-mobile {



This is only shown at the breakpoint you define in my example 800px at the same time hiding the normal table output.

This method does create extra markup though and a more organised approach might be to have a separate theme but this would rely on sniffing out the user agent and has the expense of keeping another theme up to date with any changes.

Perhaps the best solution is to ditch the table markup all together for DL’s, I would like to hear your thoughts.

Speed up Magento without Varnish Cache – The Alternatives

More often than not you should probably stay clear of implementing varnish in Magento until you’ve exhausted all other options. It seems like the Holy Grail in terms of performance and will literally make your site fly, the problem is Magento is a complicated beast and any extension you use that claims a quick integration with Magento couldn’t be further from the truth! (unless of course your using a stock site, unmodified.) It’s likely you will need a ton of customisation and debugging to any of the Varnish plugins you can buy off the shelf. Not only that you will need to set up additional servers, learn how to debug varnish and it’s various admin tools and generally spend a lot of time getting it to work properly.

Turpentine offers the most reliable product from the one’s I tested but with limited documentation and help your going to struggle to get it set up correctly. The official Varnish Pagecache extension from the makers of Varnish isn’t much better, with a number of unfixed bugs and generally poor documentation, but before getting into all that you should really consider your options on improving the site speed without the use of Varnish.

Firstly set a benchmark for how your site is now, go over to and record the speed of the homepage, category pages etc as you make each change you can run the tests again to track your progress

Optimisation for any platform

  • Tune .htaccess for speed , Creare have an excellent blueprint here
  • Use a CDN for images Pica CDN works with Amazon, Rackspace and many others
  • Install SOLR search free for CE edition alternatively this plugin works well and offers a more comprehensive set of features (be prepared for some debugging depending on your site) both these plugins require tomcat server to be installed
  • Use a CDN for content delivery e.g CloudFlare
  • Remove redundant code , php comments, html comments anything that adds to the page load however small
  • Minify your media files (JS,CSS etc) there’s a few extensions for this most widely used is the fooman speedster
  • Remove any unused styles or JS
  • Organise your main theme assets into CSS sprites
  • Reduce the quality of the images (save your images in the smallest file size possible) ideally combine the main images into a sprite
  • Use the cloud for your DNS, this can shave off an extra 50MS+ depending on your current provider, CloudFlare and other providers offer this service.
  • Use DNS pre-fetching , resolve the IP address for your assets before you use them. Something for modern browsers – Find out more in this guide from Mozilla
  • Use HTML5 browser caching through an app.manifest there’s an excellent guide on html5rocks
  • Move blocking JS to the footer, Google’s pagespeed insights offers a great tool for this (you can only do this for fonts, and external JS libaries. Don’t try moving Magento’s Core JS to the footer). Some libraries need to be in the header.

Client Side

A modern approach

Most browsers now support HTML5’s new app cache,  a cache on the users browser but browser already do that right? Yes they do – However AppCache works differently in that it’s designed for Apps where the user isn’t always online and in doing so when a cache is created on the clients side and when a page is loaded, cached resources are loaded directly from the cache, no connections to the live site, no checking if there’s a newer version –  a big speed increase.

However there’s some downsides to using this approach on how the cache expires and clearing it,  rather than re-iterate a very good explanation can be found on I’m working on a module for Magento the makes use of AppCache if you have any thoughts please leave a comment.

Server Side

Tune Apache

Apache users can achieve quite a performance boost from tweaking the configuration:

  • Put .htaccess rules directly in your vhost conf and turn off htaccess – Magento has hundreds of directories and files each time a call is made to a file apache has to recursively loop through all these to check for .htaccess files. It might not sound like much but it can make a difference. Not only this you will be securing your server from exploits.
<Directory />
Options FollowSymLinks
AllowOverride None

You can find more about some of the settings here

  • Check the apache config,  are the settings correct for your server spec? There’s numerous guide on this so I won’t detail this here, you could also try adjusting settings and running the benchmark several times to see what works best, in normal load and under stress.
  • There’s 25 tips here on tweaking apache

Use Nginx + PHP-FPM

Nginx uses much less resources as it works a bit differently from Apache, if your already familiar with Apache and don’t have the time to learn how to use Nginx it’s probably best to take it as far as you can with Apache before considering this option, if you already use it then it works pretty well out of the box. What you might need though is some extra configuration settings there’s a guide here on that.

Make sure you install php-fpm if your running nginx.

apt-get install php5-fpm

In most cases the above configuration can be left as is in nginx, if your using port 9000, you will want to edit the php-fpm configuration usually www.conf found in:


Change the user and group to match that of your nginx installation as default this is www-data

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
;       will be used.
user = www-data
group = www-data

There are three different ways of running php-fpm and that is ondemand,dynamic or static I won’t go into the detail of these, but i’ve found ondemand to work better for Magento which basically runs as many process’s as is required up to a maximum rather than fixing a certain amount to be running all the time. It’s worth noting here you can enable a status page that gives you some information about what’s going on.

pm.status_path = /status

In your php-fpm uncomment this line, you can then visit to get an output, there’s a great tutorial on using php-fpm status page here.

Tweak Mysql

There’s many guides out there on mysql and I make no claims to be an expert these settings have helped

log_slow_queries       = /var/log/mysql/mysql-slow.log

If your mysql is listening locally e.g bind-address in my.cnf says localhost or etc then you don’t need to resolve the name , this will avoid any delay from the DNS.

The rest of your config should be configured as per the report from mysqltuner, if that’s new to you check out this post on how to use it.


I have recently come across Redis It’s an extremely fast cache storage engine and works seamlessly with Magento, it does require some server configuration but I would say this is one of the easiest to configure and get up and running quickly. There’s also support for sessions via Redis Sessions have not tried this yet but also looks very solid as of CE 1.8 Redis comes part of the default install so it’s as simple as configuring it via the local.xml if you want to find out more there’s some benchmark information here and a guide to using Magento with Redis on the Magento Site.


Review coming soon on this.

APC caching

Install it onto your server

sudo apt-get install php-apc
sudo service apache2 restart

Whilst it’s installing make a note of the version, we will need this later to tweak the settings e.g

Get:1 precise/universe php-apc i386 3.1.7-1 [79.2 kB]

Enable it via/magento/app/etc/local.xml and add the following lines (note if you have more than one Magento Install on the same server make sure the prefix is unique for each one)


Tweaking APC couldn’t be easier, first check what version is installed (as noted earlier) download and browse the archive which matches on

Inside there will be a file called apc.php , you need to put this somewhere that is served by Apache or Nginx ideally on a password protected area of your site.

What you want to achieve here is a high Hit ratio and little fragmentation, usually this happens because there is not enough memory allocated to APC. You can alter this setting by changing the config file in /etc/php5/conf.d/apc.ini or wherever your php install is located until you achieve the desired result.

it should look something like this:

The maximum amount of memory APC can use, one it runs out it has to purge cached items which leads to fragmentation.

The maximum file size that can be cached by APC, this defaults to a low value and I would recommend changing this to 3M or 5M.


There’s a decent guide here on installing Memcache once you’ve done that follow this guide on enabling it in Magento

Single split servers

This is generally a good idea for failover, and that’s to have a separate MySQL and Web Server, this takes the load off one server and allows you to upscale each individually. Ideally if you have enough traffic also separate out the SOLR instance. Splitting the connection between mysql and your web server can have negative effect though depending on the connection between each you will need a gigabit connection to remove network latency so if you haven’t got much load coming from apache/nginx then it’s probably not worth it.

Google PageSpeed

This can be installed server side on Apache as a module or Nginx (Nginx requires a re-build) , this allows you to do a lot of optimisation on the fly like removing whitespace, minifying JS/CSS and even optimising images. more about Google PageSpeed

App code

Non-Varnish Caching (build your own)

Magento’s built in cache

There’s a few ways to add caching that don’t involve varnish, first is that Magento comes with comprehensive caching out of the box.

There’s really only four methods available to us.

save($value, $key, $tags = array(), $lifeTime=null)
clean($tags = array()

So let’s add something to the cache in this example we are retrieving the lowest price from a grouped product, it’s pretty intensive as it has to loop through each simple product to retrieve it’s price. The prices don’t change very often so we don’t have to do this everytime!

// load the cache
$cache = Mage::app()->getCache();

// The Code we are caching
if(!$cache->load($_product->getId())) {

// The cache doesn't exist
$aProductIds = $_product->getTypeInstance()->getChildrenIds($_product->getId());
                                    $prices = array();
                                    foreach ($aProductIds as $ids) {
                                        foreach ($ids as $id) {

                                            $aProduct = Mage::getModel('catalog/product')->load($id);
                                            if($aProduct->isSaleable()) {
                                            $prices[] = $aProduct->getPriceModel()->getPrice($aProduct);
                                    $prices = array_shift($prices);
                                    $grouped_price = $helper->currency($prices,true,false);
// save
$cache->save($grouped_price, $_product->getId(), array("grouped_prices"), 3600);
} else {

// load the saved price
$grouped_price = $cache->load($_product->getId());


echo $grouped_price;

So in the above block we are:

  1. First checking if the the cache named “id of product” exists
  2. If it returns false we then run the code to save the price to the cache with a lifetime of 1 hour (3600 seconds) and with a tag of array(“grouped_prices”)
  3. If the cache doesn’t exist we run the code as usual
  4. In all scenarios $grouped_price is returned with the price

If you don’t set a lifetime value then the item would be cached until it’s removed manually, to remove this value earlier than 1 hour we would do so by using remove


We can also remove by the tag if for instance you wanted to clear the cache for all grouped product prices you can use clean.


Full Page caching and others

Unicache Inchoo

This builds on the default caching system but really only offers convenience and an admin section allowing you to clear the cache for individual items. check it out here

Lesti FPC

Gordon Lesti wrote his own FPC for Magento, it’s easy to install follow the guide on using it, although this suffers from the same problems as using Varnish if you have custom blocks you will need to configure them for the site to work properly. However it does not require additional servers or software so it cuts a lot of set up time.

Do you know of any other ways to speed up Magento?, get in touch.

Quafzi Performance Tweaks

Recently came across this module, that offers a lot of optimisations based on recommendations from Ecommerce devs

One thing I had to disable on this particular module is the CMS block caching if for example like me you are using it to load a template that changes for each product category. It’s well commented so it’s easy to see the particular changes that might affect your site.