Using logrotate with your Laravel projects

Hard disk filled up by logs? Often overlooked until it becomes an issue. It can become a cycle of death for a server, application errors occur > write to log file > disk space full.

There’s a simple way to overcome that, and prevent the logs growing to an un-manageable size.

Laravel 4 and 5 do support some log configuration, for example to set it to daily. But this isn’t a concrete prevention of the logs getting out of hand, for that you need the logrotate utility.

Not that this guide is laravel specific you could follow this guide on any log file your keeping.

logrotate configuration files are stored in /etc/logrotate.d (Ubuntu, though likely to be the same on most linux environments)

add your file like so:

nano /etc/logrotate.d/laravel

and it’s contents:

/var/www/project/app/storage/logs/laravel.log { 
rotate 14 
daily 
compress
maxage 14 
}

Options explained

rotate – How many files are kept before deleting old files
daily – How often to rotate the files, (this could be hourly*, weekly, daily, monthly)
compress – Yep you will want this as text files compress well
maxage – Files older than 14 days will be deleted

*hourly needs the cron job updating to run hourly, best practice would be changing it to run every 5 minutes, to allow all your configuration to run whenever it’s required.

*/5 * * * * /etc/cron.daily/logrotate

So in the above example we are having a maximum of 14 log files, compressed, rotated daily and kept for 14 days. In our log directory we should have the following files:

laravel.log
laravel.log.1.gz
laravel.log.2.gz
laravel.log.3.gz
laravel.log.4.gz
laravel.log.5.gz
etc..

There’s also some neat tricks you can do with logrotate like finding all log files, if for example we had more than one log file in the same directory.

/var/www/project/app/storage/logs/*.log { 
rotate 14 
daily 
compress
maxage 14 
}

Or you could specify multiple files

/var/www/project/app/storage/logs/laravel.log /var/www/project/app/storage/logs/debug.log { 
rotate 14 
daily 
compress
maxage 14 
}

You can also run other tasks when the log is rotated

/var/www/project/app/storage/logs/*.log { 
rotate 14 
daily 
compress
maxage 14 
postrotate
    /usr/sbin/apachectl restart > /dev/null
endscript
}

Furthermore you can also restrict the size of the files by using minsize,size and maxsize specifiying a unit in M GB or K so a production example might look like the following:

/var/www/project/app/storage/logs/*.log { 
rotate 7  
daily
compress
maxage 7 
maxsize 10M
}

We now know that the logs will take up 70MB of disk space as a maximum, it’s important to note that if you do use the any of the size options they work irrespective of the rotation frequency, so as above even if the log is set to rotate daily it will be overridden if the file grew to 10MB it would be rotated immediately.

That would also be dependent on having the cron job set up at a higher frequency as mentioned above.

 

Understanding CORS and Pre-Flight

Ok so this isn’t a new thing, but most developers have been thwarted by this at some point for anyone finding this I’m going to explain it in plain english as I’ve lost count of the times I need to explain these issues to someone, or yourself 🙂

cross-over

  1. CORS is a browser thing, that means using Postman, Curl whatever other means your doing a http request your only going to experience the problem in a browser as other tools do not respect CORS.
  2. Simply put it’s a set of rules that are checked by the browser in the response headers of a request, these rules determine which websites are allowed access to the resource.
  3. It does not care what framework (Angular,React,Jquery) or Vanilla JS your using to make your request, CORS issues are generally down to how it’s configured on the resource (the server) your querying.

So when making a request the browser checks what headers the server returns, this can happen before the actual request is sent in something called a pre-flight request (more on that later).

For example I make a POST request from website.co.uk to domain.co.uk and the server returns

Access-Control-Allow-Origin: http://www.domain.co.uk

In this case only domain.co.uk is allowed to access this resource.
Another example would be that it returns:

Access-Control-Allow-Origin: *

This means anyone can access this resource. Read more below for info on setting these headers and what a pre-flight request it.

The other typical CORS issue is related to the methods

response

Here only the POST method is allowed, trying to PUT or DELETE will not work, you need to configure your server (see below) on how to enable this.

What exactly is CORS for?

CORS protects the end user the client YOU from a third party making requests on your behalf that originated from a different website than the one your on it does this by checking the Origin of the request to validate where it came from, if it’s still not making sense then, there’s some more detailed examples on Stackoverflow

What is a Pre-flight request?

So many posts i’ve seen where Angular, Jquery etc describe this as a feature of the library, wrong again it’s the browser doing this not your JS library!

preflght

Pre-flight is when the browser sends an OPTIONS request before your actual request gets sent, which the server typically responds with the options available e.g POST, PUT etc.. along with that in the headers will be Access Control rules.

Here’s an example

pre-flight

As you can see the response details what methods, headers and origins are allowed in the response headers.

If your website is not in the Access-Control-Allow-Origin: part or it’s not * that means your not allowed and it will not make your actual request and instead will return an error like:

Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

You might be asking, why don’t you see pre-flight on my request? Well this only happens for what is deemed a non-standard request.

The first part of the spec explains this https://www.w3.org/TR/cors/

It gets slightly more complicated if the resource author wants to be able to handle cross-origin requests using methods other than simple methods. In that case the author needs to reply to a preflight request that uses the OPTIONS method and then needs to handle the actual request that uses the desired method

Simple is GET, HEAD, POST without additional headers so any header modification will result in a pre-flight request.

Configuring your server to allow CORS

By default most server software e.g Apache or similar will block cross site requests, and you will need to modify the rules in your .htaccess or .conf

Apache example using .htaccess

<IfModule mod_headers.c>
    Header add Access-Control-Allow-Origin "*"
    Header add Access-Control-Allow-Headers "accept, origin, x-requested-with, content-type"
    Header add Access-Control-Allow-Methods "PUT, GET, POST, DELETE, OPTIONS"
</IfModule>

Typically the above will resolve CORS issues whilst using JS ajax requests, be careful as to what methods you need to allow.

But what are the consequences of doing this?, well there’s a good reason it’s not allowed by default see above “What is exactly is CORS for?”. Short answer is that you would generally have other protections in place for example a custom header that is submitted along with the request that only the client and the server know and you would also most likely wan’t to allow access to only a select few e.g

Access-Control-Allow-Origin: http://www.webapp.com, http://www.webclient.com

Using the above safegaurds it’s generally ok to enable it.

Migrating Angular1.x to 2 with TypeScript

Whichever release candidate your using, or the final release. This a quick heads up guide on how to port over some simple tasks, something the Hero example app doesn’t show in full.

Upgrade path

I tried using the upgrader but in reality if your strapped for time I think using that is a very lengthy time consuming process and only appeals to projects that are running continuous delivery (that’s if you can get it set up in the first place).

As your adding the overhead of creating the upgrade/dowgrade code and then after that having to refactor.

I opted for a re-build of the current app refactoring service by service etc, it also allows you to make some architectural changes to the api of your app along the way, how much dependant on the time you have.

The biggest challenges are where the previous app uses $q, $watch and where you’ve used angular’s built in methods like angular.extend some can be fairly quickly swapped for lodash alternatives though or another framework, but overall it’s the sheer amount of typing you will need to do that gets tedious especially for Typescript.

Loading config

In Angular 1 you might have had some app wide config going on that sat in the main app.js e.g

angular.module('myApp')

    .config(function (appConfig) {
//configure stuff here

And also through a constant

angular.module('myApp').constant('appConfig', {
    "version": "1.0"
})

In Angular2 there’s two ways to solve this problem one is to provide the value on the root level app component like so:

providers: [
        { provide: VERSION, useValue: '1.0' }
    ],

You can then get that value by doing

var version = injectorValue.get(VERSION);

That’s great for small one liner configurations but a better way would be to load it from a local file. We can do that with a simple service.

import { Injectable } from [email protected]/core';
var config = require('./env.json');

@Injectable()
export class AppConfigService {

    private _config: Object;

    constructor() {
        this._config = config;
    }

    get(key: any) {
        return this._config[key];
    };
    getAll() {
        return this._config;
    }
}

In this case the env.json is generated externally from a gulp task. If you wanted to load an external file over http you could do something like below, here we would need to provide the endPoint value as we did above to the main app component.

providers: [
        { provide: endPoint, useValue: 'http://127.0.0.1/api' }
    ],
import {Http} from [email protected]/http';
import {Injectable, Inject} from [email protected]/core';

@Injectable()
export class AppConfig {

    constructor(private http: Http, 
                @Inject('AppConfig') private endPoint: string) { }

    getConfig(): {
        return this.http.get(`${this.endPoint}/`)
            .then((response) => {
                //do something with the response
            });
    } 
    // the rest of the code...
}

2. Promises

I won’t go into too much detail on this as there are some excellent guides around but where your code uses $q in Angular1 you will need to convert that to using an ES6 promise.

var promise = $q.when([]);
promise = promise.then(function() {
                        return getSomeData();
                    });
promise = promise.then(function() {
// do something else
});

In ES6 we could convert it like this

return new Promise((resolve,reject) => {
    getSomeData().then((result) => {
       //do something with result
       resolve(result);
}).catch((err) => {
       reject(err);
}
)};

or chain them together

let promise = new Promise((resolve,reject) => {
    getSomeData().then((result) => {
       //do something with result
       resolve(result);
}).catch((err) => {
       reject(err);
}
)};
promise.then((res) => {
   //do another thing
});

3. using $this _self etc

In most cases you should avoid doing that anyways either using .bind(this) although there is the odd case you need to, but with ES6 it makes it much easier to avoid through using arrows. In simple terms it’s shorthand for a function() {}, but the added feature of keeping the context of this to the parent. E.g

var $this = this;
Catalogue.getChildren(catId)
            .then(function(categories) {
                availableFilters.subCategories = categories;
            }).then(function(categories) {
                return CatalogueRepository.getFilters($this.getCriteria());
            })

Using arrows we no longer need $this

Catalogue.getChildren(catId)
            .then((categories) => {
                availableFilters.subCategories = categories;
            }).then(function(categories) => {
                return CatalogueRepository.getFilters(this.getCriteria());
            });

4. Angular forEach,copy etc..

for(var data in key) {
   key //key
   data[key] //value
}

Whilst there is vanilla js alternatives for these the quickest solution is to just import lodash or other library, lodash has _.deepClone _.forEach I use it like so after the imports.

var _ = require('lodash');

5. Window

For RC4 Provide window in your root component

providers: [
        { provide: Window, useValue: window }
    ],

In your component constructor use

constructor(
     @Inject(Window) private _window: Window
)

There may be a better way of doing the above, please let me know if you do it a different way.

6. Using pipe inside a service

Import the pipe and then call it like below.

import {TranslatePipe} from "../pipes/translate.pipe";
constructor(private translate: TranslatePipe) {
}
myFunction() {
this.translate.transform('some text');
}

7. Global variables

If for example your using another framework or something else in the global namespace you can use this to avoid TS errors, e.g using Ionic with Cordova.

declare var cordova: any;

@Injectable()
export class MyService {}

8. Code organisation

The sheer amount of imports can become a bit bewildering, to overcome this use barrel files, you have a bunch of services in /services then create a file inside there called index.ts

Inside this put:

export {appService} from "./appService";
export {configService} from "./configService";
export {dbService} from "./dbService";
export {UserDb} from "./userDb";

You can import it into your components like so:

import * as coreServices from "./services/";

You can provide or bootstrap them using [coreServices.UserDb,coreServices.dbService] etc..

One disadvantage to this is that you can’t quickly see the list of services upfront, so it’s best to group them for example /db /file /network and have the corresponding index files.

import * as network from "./network/";
import * as db from "./db/";
import * as files from "./file/";

This is much more readable.

9. Data in templates

If you need to load some data before it’s available to the template in Angular2 and want to avoid the undefined errors, there are a few ways of solving this.

1. ngOnInit( this.state = getMyState(); );
2. “The Elvis Operator” before your data e.g {{state?.name}}
3 ngIf e.g *ngIf=”state”
4. construct your model which will give you an empty object. this.state = new myModel();

10. Global services

In angular1 every service was a singleton, v2 has the same concept if you inject the service at the app level, but you can also provide it separately to to individual areas of the app through the providers:[], read more about that here

That’s it for now, let me know if you have any other simple things.

 

Hiding metabox from backend in WordPress

In my case hiding themify and yoast seo metabox from the backend for pages (this only works for pages, you will need to change that to whatever type of post it is see here for more) , use below snippet.

function hide_meta() {
    global $post;
    if ( get_page_template_slug($post->id) == 'services-new.php') {
        remove_meta_box('wpseo_meta', 'page', 'normal');
        remove_meta_box('themify-meta-boxes', 'page', 'normal');
    };
}
add_action( 'admin_notices', 'hide_meta' );

Reset npm/bower permissions

Useful snippet for resetting bower/npm permissions if you get errors like this

Error: EACCES: permission denied, open '/Users/jdavey/.config/configstore/bower-github.json'
You don't have access to this file.

sudo chown -R $USER:$GROUP ~/.npm
sudo chown -R $USER:$GROUP ~/.config

You can also run this on whatever folder is causing the permissions error. This resets them to the current user.

Setting different meta title,description per social network

The only way to get different title,meta responses per social media channel is to have the response set on the backend, in this case when Facebook, Pinterest and Google Plus hit the page check the user agent and display the relevant info, Twitter you can still force the text/image that’s shown, the rest rely on the open graph tags or whatever the title/description is set to. From my experience it’s impossible to set this via plugins like addthis etc.. although it may be possible through some of the API’s it’s easier to check the user agent though.

jstl example

<c:choose>
<c:when test="${fn:contains(header['User-Agent'],'Facebot') || fn:contains(header['User-Agent'],'facebookexternalhit/1.1') || fn:contains(header['User-Agent'],'facebookexternalhit/1.0')}">
<c:if test="${not empty facebookTitle}">
<c:set var="ogTitle" value="${facebookTitle}" />
</c:if>
<c:if test="${not empty facebookDesc}">
<c:set var="ogDescription" value="${facebookDesc}" />
</c:if>
<c:if test="${not empty facebookImage}">
<c:set var="ogImage" value="${facebookImage}" />
</c:if>
</c:when>
<c:when test="${fn:contains(header['User-Agent'],'pinterest/0.1 +http://pinterest.com/')}">
<c:if test="${not empty pinterestTitle}">
<c:set var="ogTitle" value="${pinterestTitle}" />
</c:if>
<c:if test="${not empty pinterestDescription}">
<c:set var="ogDescription" value="${pinterestDescription}" />
</c:if>
<c:if test="${not empty pinterestImage}">
<c:set var="ogImage" value="${pinterestImage}" />
</c:if>
</c:when>
<c:when test="${fn:contains(header['User-Agent'],'Google (+https://developers.google.com/+/web/snippet/)')}">
<c:if test="${not empty gplusTitle}">
<c:set var="ogTitle" value="${gplusTitle}" />
</c:if>
<c:if test="${not empty gplusDesc}">
<c:set var="ogDescription" value="${gplusDesc}" />
</c:if>
<c:if test="${not empty gplusImage}">
<c:set var="ogImage" value="${gplusImage}" />
</c:if>
</c:when>
<c:otherwise>
<!-- use default title value -->
</c:otherwise>
</c:choose>

PHP example

function get_meta()
{
switch (true) {
case stristr($_SERVER['HTTP_USER_AGENT'], 'Facebot') || stristr($_SERVER['HTTP_USER_AGENT'],'facebookexternalhit'):
$title = 'Facebook Title';
$description = 'Facebook Description';
break;
case stristr($_SERVER['HTTP_USER_AGENT'], 'developers.google.com'):
$title = 'Google Title';
$description = 'Google Description';
break;
case stristr($_SERVER['HTTP_USER_AGENT'], 'pinterest'):
$title = 'Pinterest Title';
$description = 'Pinterest Description';
break;
default:
$title = 'Default title';
$description = 'Default Description';
}
}

Frontend Development in CQ5

I’m new to using CQ5 in this instance version 5.5, as a frontend developer I found the software a little confusing at first, mainly in working in a VM development environment like below:

dev

 

As CQ5 is platform independent it can easily be set up on Ubuntu with Apache acting as a reverse proxy to your local content repository allowing you to preview the content through your shared folder. The above can be extended to include https and the author instance.

In short Apache passes any requests to etc/project* to your localhost e.g 8080 which is serving the shared folder and so serves up your local repo.

If you want to find out more about how the repository works in terms of folder structure etc there is some useful documentation here on Adobe’s site.

Once this is set up you can get on with developing again there’s some useful guides

Component Properties and structure

Component best practices

Developing on CQ 5.6

Widget docs

All available Xtype’s

Any starting point should be copying an existing component and playing around with the attributes and settings.

JSP is server side so there’s not really the prospect of trying out your jsp without simply adding it to your component, although I did come across http://zkfiddle.org/ which does work albeit a bit slow.

If you do get an error and your not in the component.jsp you need to undo your changes and then modify/save component.js to clear any cache.

Another niggle is the VM not shutting down properly, something I’m not aware of there being a fix, which gives you a nice blank page when you fire up the author or publisher. The only solution being copying over a new instance of your VM. I did read about the Lucene indexes getting corrupt but these take hours to re-compile so it’s not worth the time unless you where really stuck (there’s a good article here on that http://www.wemblog.com/2011/12/how-to-rebuild-index-in-cq5-wem.html) and that’s if that is indeed the error.

If you have any suggestion for the above please leave a comment.

Calculating selling price from the cost price

Recently in my price matching system I had to calculate the selling price of a product in a way that also calculates a transaction fee percentage on the final selling price without knowing the selling price, and without knowing what the transaction fee is until you’ve calculated the total price. Had fun chasing your tail yet?

You can do this in Excel with circular references but you will get circular reference warnings and the only way is to allow these warnings and an iteration limit of 100.

Stepping aside from the magic of excel I was able to work out the exact formula for this and below is a calculation for anyone else struggling to work out how to do this.

$cp = 9.56; // cost price
$d = 2.50; // delivery fee
$m = 0.15; // margin 15%
$tf = 0.019; // transaction fee

$sellingPrice = (($cp+$d)*(1+($m)))/(1-((1/6)+$tf)*(1+($m)));

$costPrice = $x / (1+(.15));

echo $sellingPrice."\n";
echo $costPrice;

Magento image not loading when using cron, only getting placeholder

I have a module which sends out an email via Mailchimp with a scheduled task for Magento’s internal cron and in doing so looks up the product image.

Mage::helper('catalog/image')->init($product, 'small_image')->resize(150));

The problem being only the placeholder was being returned when the cron job was executed. To confuse matters when I ran the script in testing from the browser it would return the correct image.

This is a nasty problem at first I thought this might be the answer Sangay details there that if your config does not specify a M,GB etc that it will return the wrong calculation, but then why the different results depending on how I ran the script?

To get to the bottom of this I ran what is being run in Mage_Catalog_Helper_Image

$model = Mage::getModel('catalog/product_image');
$model->setBaseFile($this->product->getData('small_image'))->resize(600,600);
$url = $model->saveFile()->getUrl();

PHP Fatal error:  Uncaught exception 'Varien_Exception' with message 'Memory limit has been reached.' in /lib/Varien/Image/Adapter/Gd2.php:58

Returning this shows the actual exception I did not get any Exception in the Mage exception log, and not in the host’s error log. On my part a server configuration oversight as the CLI php.ini did not have the error log set up. In any case that’s the error.

Because the value returned from _convertToByte function in Mage_Catalog_Helper_Image is -1 as the CLI php.ini sets the memory_limit to -1 (unlimited).

This could be an oversight by Magento or a design feature not to run this task on unlimited memory, personally I think it’s a bug. I modified the function to the following to set a 2G limit in this case.

   if (stripos($memoryValue, 'M') !== false) {
            return (int)$memoryValue * 1024 * 1024;
        }
        elseif (stripos($memoryValue, 'KB') !== false) {
            return (int)$memoryValue * 1024;
        }
        elseif ($memoryValue == "-1") {
            return '2147483648'; //2G
        }

Clearing expired magento shopping/sales rules programatically

Here’s a nice cleanup script I use to clear out expired shopping cart rules and catalog price rules and apply those changes. It’s one of those annoying things that you would think there would be a button for.

I’ve set the expiry check to the previous day and date comparison using strtotime you might want to adjust that, and use php’s dateTime library for a more thorough comparison as this relies on your server’s timezone configuration matching magento’s and being set up correctly.

If your wondering what setState does that’s to remove the message in magento admin about un-applied rules.

require '../app/Mage.php';

umask(0);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);


$model = Mage::getModel('catalogrule/rule')
    ->getCollection();


foreach($model as $rule) {
    $today = strtotime('Yesterday');

    if(strtotime($rule->getData('to_date')) < $today) {
      $rule->delete();
    }

}

$model = Mage::getModel('salesrule/rule')
    ->getCollection();


foreach($model as $rule) {
    $today = strtotime('Yesterday');

    if(strtotime($rule->getData('to_date')) < $today) {
       $rule->delete();
    }

}

try {
    Mage::getModel('catalogrule/rule')->applyAll();
    Mage::getModel('catalogrule/flag')->loadSelf()
        ->setState(0)
        ->save();
    Mage::app()->removeCache('catalog_rules_dirty');
    echo Mage::helper('catalogrule')->__('The rules have been applied.');
} catch (Exception $e) {
    echo Mage::helper('catalogrule')->__('Unable to apply rules.');
    print_r($e);
}