Install & Configure Varnish Cache for Magento 2

May 19, 2016  |  13 Comments  |  by Raj (MagePsycho)  |  Linux, Magento 2

The performance of a website is always the key factor for the higher conversion rate. Fortunately, Magento 2 (both CE & EE) comes by default with the support for FPC (Full Page Caching), Varnish & Redis to make your store fly.
In this tutorial we will be discussing brief info about Varnish, it’s installation and configuration so bear with me.

Magento-2-Banner

Magento 2 supports Varnish versions 3.0.5 or later or any Varnish 4.x version.
And Magento team strongly recommends using Varnish in production as the built-in full-page caching (to either the file system or database) is much slower than Varnish, and Varnish is designed to accelerate HTTP traffic.
varnishcache_rgb-gimp2-alpha

Varnish Cache is an open source web application accelerator (also referred to as an HTTP accelerator or caching HTTP reverse proxy). Varnish stores (or caches) files or fragments of files in memory; this enables Varnish to reduce the response time and network bandwidth consumption on future, equivalent requests. Unlike web servers like Apache and Nginx, Varnish was designed for use exclusively with the HTTP protocol.

Assumption

We have carried out the Varnish installation in the following enviroment:

  • Ubuntu – 14.04 LTS
  • Nginx – 1.9.6
  • MySQL – 5.6.28
  • PHP – 5.6.20 (but PHP 7 is recommended)
  • Magento EE – 2.0.5

Installation

Run the following series of command as a root user (sudo su).

apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/ubuntu/ trusty varnish-4.1" \
     >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish

If everything goes right you should be able to see the Varnish version with:

varnishd -V

Configuration

Once Varnish is installed, you need to configure it and your web server(Nginx in our case).

Configure your Web Server(Nginx)

Configure your web server to listen on a port (say: 8080) other than the default port 80 because Varnish responds directly to incoming HTTP requests, not the web server.
For this add the ‘listen’ directive in ‘server’ block as:

server {
    listen 8080 default_server;
    server_name  your-mage2-store.dev;
    ...
}

to default nginx configuration file (generally located at /path/to/nginx/sites-enabled/default)

Configure Varnish Configuration

1. Modify Varnish System Configuration

vim /etc/default/varnish

And edit the file to have similar code:

START=yes

# Maximum number of open files (for ulimit -n)
NFILES=131072

# Maximum locked memory size (for ulimit -l)
# Used for locking the shared memory log in memory.  If you increase log size,
# you need to increase this number as well
MEMLOCK=82000

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -p http_resp_hdr_len=64000 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

2. Modify default.vcl
Make a backup of default one.

cp /etc/varnish/default.vcl /etc/varnish/default.vcl.bak
vim /etc/varnish/default.vcl

And edit the lines under ‘backend default’:

# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
}

Where

  • .host – fully qualified host name or IP address and listen port of the Varnish backend or origin server; that is, the server providing the content Varnish will accelerate. Typically, this is your web server.
  • .port – the web server’s listen port (8080 in this example)

Note:This is the minimal configuration. You need to use the configuration which you get from Magento 2’s Varnish Export (Ref – #3).

3. Configure Magento to use Varnish
Instead of manually editing default.vcl, Magneto 2 comes up with an export option for it.
First enable the Full Page Cache to use Varnish (Go to Admin > Stores > Configuration > Advanced > System > Full Page Cache), you will see as:
magepsycho-magento-2-varnish-cache
Save the settings and hit the Export VCL button (Varnish 3 or 4 as per your Varnish Version).
And use this default.vcl to replace the file in /etc/varnish/default.vcl

Finally, restart the Varnish & Nginx

service nginx restart
service varnish restart

Verification

I assume You have followed all the above steps (Installation & Configuration). Now lets check if Varnish cache server is taking into account for HTTP response or not.

Check if Varnish is listening to port 80
Run the following command:

netstat -tulpn | grep varnishd

varnishd-listing-toport-80

Verify contents in var/cache, var/page_cache folder is not regenerated
Since FPC is configured to serve the content from Varnish, once cleared, cache folders (var/cache, var/page_cache) should always be empty .

Verify the HTTP response headers
Load your Magento 2 Frontend and check the response from browser console, you will see similar as shown in the below screenshot:
magepsycho-varnish-http-response-headers

Or you can simply use CLI as:

curl -I http://your-mage2-store.dev/

And You will see some similar Varnish keywords.

Voila, you’re done. Let your store fly!
Please do comment or contact us if you have any issues regarding Varnish + Magento2 Installation & Configuration.

Run Magento database repair tool from CLI using Ruby’s Mechanize

June 11, 2015  |  No Comments  |  by Raj (MagePsycho)  |  Linux, Magento, Ruby

Shell Scripting which is great for task automation, falls short when it comes to web browser automation.
In case of Magento, using Shell script you can do many operations like Installation, Migration, Deployment, Backup etc. & even more. But when it comes to run the Database Repair Tool(a great tool for repairing Magento database while upgrading) it becomes trickier and complex.

Magento Database Repair Tool

Magento Database Repair Tool

While Upgrading Magento say from 1.3.2.4 to 1.9.1.1, running DB repair tool for every version manually from browser is repetitive & hectic. So I thought of writing a Ruby CLI script to automate it which will be a simple command with just two parameters(DB repair Url and Magento version).

This console script is a Ruby script which uses Mechanize gem/library.

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

Install Ruby

Ruby comes pre-installed in Mac. If you are in other operating system please refer to the Installation Document
To test if ruby is already installed, run the following command:

ruby -v

Which will give you the output like:

ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

Install Mechanize Gem

You need to install RVM(Ruby Version Manager) first.
First install the stable version of RVM stable with ruby:

\curl -sSL https://get.rvm.io | bash -s stable --ruby

Update your gems:

sudo gem update --system

Now you can instally any gems. Here we will be installing mechanize gem:

gem install mechanize

Prepare & Run the Script

After installation of Ruby, RVM and all it’s required gems, we are ready to go for coding the script.
Assumptions:
OS: Mac OS X Yosemite 10.10.4
Ruby: 2.0
Magento Reference Database: magento[version]_vanilla

1. Prepare the Script
Copy the following ruby script to the root of your Magento Dir:
File: mage-db-repair-tool.rb

#!/usr/bin/env ruby

#
# Magento DB Repair tool using CLI
# Uses Ruby's Mechanize gem
#
# @author   Raj KB <magepsycho@gmail.com>
# @website  http://www.magepsycho.com
#
# Tested on Mac OS-X 10.X
#

require 'mechanize'
require 'fileutils'

#################################
# FUNCTIONS
#################################
def checkError(page)
  if page.search('.msg_error').length > 0
    puts "[ERROR]"
    page.search('.msg_error li').each do |li|
      puts li.text.strip
    end
    exit
  end
end

def checkResult(page)
  puts "[RESULT]"
  page.search('.msg_success li').each do |li|
    puts li.text.strip
  end
end

def checkNotice(page)
  puts "[NOTICE]"
  page.search('.msg-note').each do |note|
    puts note.text.strip
  end

  if page.search('.msg-note').length > 1
    puts "See log for more details"
  end
end

#################################
# SCRIPT CODE
#################################
abort "#{$0} Argument Missing" if (ARGV.size < 1)

dbRepairUrl     = ARGV[0]
mageVersion     = ARGV[1]
mageDir         = Dir.pwd
dbRepairLogDir  = "#{mageDir}/var/dp-repair-tool"

FileUtils.mkdir_p "#{dbRepairLogDir}"
fp = File.new("#{dbRepairLogDir}/mage-#{mageVersion}-result.html", "a+")

agent = Mechanize.new
page = agent.get(dbRepairUrl)
puts "Loading page: #{page.title}..."
form = page.forms.first

puts "Setting db repair form values..."

# Get DB value from app/etc/local.xml
xmlFile = File.open("#{mageDir}/app/etc/local.xml")
doc = Nokogiri::XML(xmlFile)
corruptedHostname = doc.xpath('/config/global/resources/default_setup/connection/host').text()
corruptedUsername = doc.xpath('/config/global/resources/default_setup/connection/username').text()
corruptedPassword = doc.xpath('/config/global/resources/default_setup/connection/password').text()
corruptedDatabase = doc.xpath('/config/global/resources/default_setup/connection/dbname').text()
xmlFile.close

form['post_form']           = 'true'
form['corrupted[hostname]'] = corruptedHostname
form['corrupted[database]'] = corruptedDatabase
form['corrupted[username]'] = corruptedUsername
form['corrupted[password]'] = corruptedPassword

# Edit reference database credentials
form['reference[hostname]'] = corruptedHostname
form['reference[database]'] = "magento#{mageVersion}_vanilla"
form['reference[username]'] = corruptedUsername
form['reference[password]'] = corruptedPassword

#p form; exit

puts "Submitting db repair form..."
result_page = form.submit(form.button_with(:id => "button-continue"))

# Check if there is an error & exit
checkError(result_page)

# Continue if there is not an error
# Check if it further requires submission
if result_page.search('button#button-continue').length > 0

    checkNotice(result_page)
    # Form found again?
    puts "Submitting again..."
    form = result_page.forms.first
    result_page = form.submit(form.button_with(:id => "button-continue"))
    checkResult(result_page)
    checkNotice(result_page)
else
    checkResult(result_page)
    checkNotice(result_page)
end

fp.write(result_page.body)

You can also download the script from: Magento DB Repair Tool Using CLI – Ruby + Mechanize

2. Run the Script
Give script the executable permission:

cd /path/to/mage-db-repair-tool.rb
chmod +x ./mage-db-repair-tool.rb

Now, You can run the script either by using command:

ruby mage-db-repair-tool.rb <magento-db-repair-url> <magento-version>

or simply using:

./mage-db-repair-tool.rb http://magento-test.dev/magento-db-repair-tool-1.2.php 1702

Example:

./mage-db-repair-tool.rb http://magento-test.dev/magento-db-repair-tool-1.2.php 1702
Ruby DB Repair Tool Console Output

Ruby DB Repair Tool Console Output

Apart from the results shown at the console output (refer to the above snapshot), You can also check the detailed output logged in your /path/to/magento/var/dp-repair-tool/mage-[magento-version]-result.html

Bonus Tips:

If you want to make system-wide command, then copy the file to the system-wide paths like /usr/local/bin, /usr/bin, /bin etc:

sudo cp /path/to/your/mage-db-repair-tool.rb /usr/local/bin/mageDbRepairToolRb

Now you can run the command from anywhere

mageDbRepairToolRb http://magento-test.dev/magento-db-repair-tool-1.2.php 1702

Shall you have any issues please post a comment below and I’ll try and help you out.

Backup WordPress project files / db using bash script

August 8, 2013  |  5 Comments  |  by Raj (MagePsycho)  |  Linux, Magento

WordPress is undoubtedly the most popular blogging platform on the Internet. And when it comes to the maintenance or regular backups, life is not easier unless you have some backup script which works great via SSH & cron daemons.

Here is the custom backup script used for taking backup of wordpress project files & db:
(If you want backup script for Magento then please go here)

#!/bin/bash
#@author		MagePsycho <magepsycho@gmail.com>
#@website		http://www.magepsycho.com
#@blog			http://www.blog.magepsycho.com
#@version		0.1.0

#/************************ EDIT VARIABLES ************************/
projectName=magepsycho_blog
backupDir=/home/magepsycho/_backups
#/************************ //EDIT VARIABLES **********************/

fileName=$projectName-$(date +"%Y-%m-%d")
host=$(grep DB_HOST "wp-config.php" |cut -d "'" -f 4)
username=$(grep DB_USER "wp-config.php" | cut -d "'" -f 4)
password=$(grep DB_PASSWORD "wp-config.php" | cut -d "'" -f 4)
dbName=$(grep DB_NAME "wp-config.php" |cut -d "'" -f 4)

printf "What kind of backup you would like?\n[ d ] DB backup only\n[ f ] Files backup only\n[ b ] Files backup with DB\n"
read backupType
if [[ $backupType = @(d|b) ]]; then
	echo "----------------------------------------------------"
	echo "Dumping MySQL..."
	mysqldump -h "$host" -u "$username" -p"$password" "$dbName" | gzip > $fileName.sql.gz
	echo "Done!"
fi

if [[ $backupType = @(f|b) ]]; then
	echo "----------------------------------------------------"
	echo "Archiving Files..."
	tar -zcf $fileName.tar.gz * .htaccess
	echo "Done!"
	echo "----------------------------------------------------"
	echo "Cleaning..."
	rm -f $fileName.sql.gz
	echo "Done!"
fi

if [[ $backupType = @(d|f|b) ]]; then
	echo "----------------------------------------------------"
	mkdir -p $backupDir;
	echo "Moving file to backup dir..."
	if [[ $backupType == d ]]; then
		mv $fileName.sql.gz $backupDir
	fi

	if [[ $backupType = @(f|b) ]]; then
		mv $fileName.tar.gz $backupDir
	fi
	echo "Done!"
else
	echo "Invalid selection!"
fi

Or you can download it from here: [WordPress Backup Script]
Notes: If you get the following error:

syntax error in conditional expression: unexpected token `(‘
line 20: syntax error near `@(d’
line 20: `if [[ $backupType = @(d|b) ]]; then’

then this means you are using older version of bash (< 4.0). And you need to patch the script by adding the following line after bash script declaration: [code language="bash"]shopt -s extglob[/code]

What does this script do?

1. Gives options for backup type which are:
– DB backup only
– Files backup only
– Files backup with DB
2. Dumps the database by taking db info from wp-config.php
3. Makes a copy of project files and compresses it in .tar.gz format.
4. Deletes the dumped SQL file as it’s already copied in compressed file.
5. Creates backup dir if not exists
6. Copies the compressed project file to the backup dir.

Besides, you can also setup cron job to run this backup script at regular intervals.

How to run backup script?

1. Edit _wpbackup.sh to configure variables ‘projectName’ & ‘backupDir’
2. Upload the edited _wpbackup.sh to the root of your WordPress installation
3. Run following series of commands in terminal:

cd /path/to/wordpress/root
chmod +x _wpbackup.sh
ex -sc $'%s/\r$//e|x' _wpbackup.sh
sh _wpbackup.sh

4. You will get the compressed backup file at backup dir

Snapshots for different backup types

DB Backup Only

DB Backup Only


File Backup Only

File Backup Only

Files Backup with DB

Files Backup with DB

Thanks for reading & sharing.

Backup Magento project files / db using bash script

July 1, 2013  |  8 Comments  |  by Raj (MagePsycho)  |  Linux, Magento

I have googled for the bash script to backup Magento sites/db but none of them worked for me the way I wanted.
So I decided to create a custom bash script which is simple and does the job perfectly.

I have developed the script for my own need. But I thought it would be helpful if shared with you guys as well.
Here goes the overall backup script:

#!/bin/bash
#@author		MagePsycho <magepsycho@gmail.com>
#@website		http://www.magepsycho.com
#@version		0.1.0

#/************************ EDIT VARIABLES ************************/
projectName=magepsycho
backupDir=/home/magepsycho/_backups
#/************************ //EDIT VARIABLES **********************/

dbXmlPath="app/etc/local.xml"
{
host="$(echo "cat /config/global/resources/default_setup/connection/host/text()" | xmllint --nocdata --shell $dbXmlPath | sed '1d;$d')"
username="$(echo "cat /config/global/resources/default_setup/connection/username/text()" | xmllint --nocdata --shell $dbXmlPath | sed '1d;$d')"
password="$(echo "cat /config/global/resources/default_setup/connection/password/text()" | xmllint --nocdata --shell $dbXmlPath | sed '1d;$d')"
dbName="$(echo "cat /config/global/resources/default_setup/connection/dbname/text()" | xmllint --nocdata --shell $dbXmlPath | sed '1d;$d')"
}

fileName=$projectName-$(date +"%Y-%m-%d")
printf "What kind of backup you would like?\n[ d ] DB backup only\n[ f ] Files backup only\n[ b ] Files backup with DB\n"
read backupType
if [[ $backupType = @(d|b) ]]; then
	echo "----------------------------------------------------"
	echo "Dumping MySQL..."
	mysqldump -h $host -u $username -p$password $dbName > $fileName.sql
	echo "Done!"
fi

if [[ $backupType = @(f|b) ]]; then
	echo "----------------------------------------------------"
	echo "Archiving Files..."
	printf "Skip /media folder?\ny: Yes\nn: No\n"
	read skipMedia
	if [ $skipMedia == y ]; then
		tar -zcf $fileName.tar.gz --exclude=var --exclude=includes --exclude=media * .htaccess
	else
		tar -zcf $fileName.tar.gz --exclude=var --exclude=includes * .htaccess
	fi
	echo "Done!"
	echo "----------------------------------------------------"
	echo "Cleaning..."
	rm -f $fileName.sql
	echo "Done!"
fi

if [[ $backupType = @(d|f|b) ]]; then
	echo "----------------------------------------------------"
	mkdir -p $backupDir;
	echo "Moving file to backup dir..."
	if [ $backupType == d ]; then
		mv $fileName.sql $backupDir
	fi

	if [[ $backupType = @(f|b) ]]; then
		mv $fileName.tar.gz $backupDir
	fi
	echo "Done!"
else
	echo "Invalid selection!"
fi

Or you can download it from [here]
Notes: If you get the following error:

syntax error in conditional expression: unexpected token `(‘
line 24: syntax error near `@(d’
line 24: `if [[ $backupType = @(d|b) ]]; then’

then this means you are using older version of bash (< 4.0). And you need to patch the script by adding the following line after bash script declaration:[code language="bash"] shopt -s extglob [/code]

What does this script do?

1. Gives options for backup type which are:
– DB backup only
– Files backup only
– Files backup with DB
2. Dumps the database by taking DB info from XML configuration.
3. Makes a copy of project files and compresses it in .tar.gz format.
While copying it also gives options whether to exclude media folder or not.
Note that by default ‘var’ & ‘includes’ folder are excluded by the script.
4. Deletes the dumped SQL file as it’s already copied in the compressed file.
5. Creates backup dir if not exists
6. Copies the compressed project file to the backup dir.

Besides, you can also setup Cron job to run this backup script at regular intervals.

How to run backup script?

1. Edit _magebackup.sh to configure variables ‘projectName’ & ‘backupDir’
2. Upload the edited _magebackup.sh to the root of your Magento installation
3. Run following series of commands in terminal:

cd /path/to/magento/root
chmod +x _magebackup.sh
ex -sc $'%s/\r$//e|x' _magebackup.sh
sh _magebackup.sh

4. You will get the compressed backup file at backup dir

Snapshots for different backup types

DB backup only


Files backup only


Files backup with DB


Let’s comment if there’s any room for improvement in this script.
Thanks for reading & sharing.

500 Internal Server Error: Mismatch between target GID (XXX) and GID (XXX) of file

April 29, 2012  |  No Comments  |  by Raj (MagePsycho)  |  Apache, Latest, Linux, Magento

Today (at the time of writing) i tried to install Magento 1.7.0.0 via SSH as ‘root’ user using following bash script:

#!/bin/bash
wget http://www.magentocommerce.com/downloads/assets/1.7.0.0/magento-1.7.0.0.tar.gz
wget http://www.magentocommerce.com/downloads/assets/1.2.0/magento-sample-data-1.2.0.tar.gz
tar -zxvf magento-1.7.0.0.tar.gz
tar -zxvf magento-sample-data-1.2.0.tar.gz
mv magento-sample-data-1.2.0/media/* magento/media/
mv magento-sample-data-1.2.0/magento_sample_data_for_1.2.0.sql magento/data.sql
mv magento/* magento/.htaccess .
chmod -R o+w media
mysql -u dbUserName -pdbPass dbName < data.sql
chmod o+w var var/.htaccess app/etc
rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*
rm -rf magento/ magento-sample-data-1.2.0/
rm -rf magento-1.7.0.0.tar.gz magento-sample-data-1.2.0.tar.gz data.sql

And when i tried to browse the url for Magento installation, i got the screen with ‘500 Internal Server Error’.
Being aware of all possible causes of 500 Internal Server Error during Magento Installation, I tried all possible solutions that i know like:

  1. 1. Checking of Web / Unsecure / Base Url & Web / Secure / Base Urls
  2. 2. Changing permission setting for index.php:
    chmod 0755 index.php
  3. 3. Changing permission setting for folder / files recursively:
    find . -type d -exec chmod 755 {} \;
    find . -type f -exec chmod 644 {} \;
  4. 4. Renaming .htaccess file:
    mv .htaccess .htaccess.bak

etc…
But none of them worked for me.

When i checked at the apache error log file then i got the following error in log file:

SoftException in Application.cpp:431: Mismatch between target GID (508) and GID (1072) of file “/home/username/public_html/index.php”

Note: 1072 -> id of ‘root’ user
508 -> id of ‘username’ user

From above it’s clear that error is due to mismatching between ownership and got solved by using following command:

chown -R username:username /home/username/public_html

Where [username] = username of hosting(hostagator.com)

Final Note:
Always make a habit of checking the Apache error log file in case of ‘500 Internal Server Error’ as the error can be caused due to various reasons instead of directily diving into solutions.