Run Magento database repair tool from CLI using Ruby’s Mechanize

June 11, 2015  |  No Comments  |  by Raj (MagePsycho)  |  Linux, Magento, Ruby

Shell Scripting which is great for task automation, falls short when it comes to web browser automation.
In case of Magento, using Shell script you can do many operations like Installation, Migration, Deployment, Backup etc. & even more. But when it comes to run the Database Repair Tool(a great tool for repairing Magento database while upgrading) it becomes trickier and complex.

Magento Database Repair Tool

Magento Database Repair Tool

While Upgrading Magento say from 1.3.2.4 to 1.9.1.1, running DB repair tool for every version manually from browser is repetitive & hectic. So I thought of writing a Ruby CLI script to automate it which will be a simple command with just two parameters(DB repair Url and Magento version).

This console script is a Ruby script which uses Mechanize gem/library.

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

Install Ruby

Ruby comes pre-installed in Mac. If you are in other operating system please refer to the Installation Document
To test if ruby is already installed, run the following command:

ruby -v

Which will give you the output like:

ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

Install Mechanize Gem

You need to install RVM(Ruby Version Manager) first.
First install the stable version of RVM stable with ruby:

\curl -sSL https://get.rvm.io | bash -s stable --ruby

Update your gems:

sudo gem update --system

Now you can instally any gems. Here we will be installing mechanize gem:

gem install mechanize

Prepare & Run the Script

After installation of Ruby, RVM and all it’s required gems, we are ready to go for coding the script.
Assumptions:
OS: Mac OS X Yosemite 10.10.4
Ruby: 2.0
Magento Reference Database: magento[version]_vanilla

1. Prepare the Script
Copy the following ruby script to the root of your Magento Dir:
File: mage-db-repair-tool.rb

#!/usr/bin/env ruby

#
# Magento DB Repair tool using CLI
# Uses Ruby's Mechanize gem
#
# @author   Raj KB <magepsycho@gmail.com>
# @website  http://www.magepsycho.com
#
# Tested on Mac OS-X 10.X
#

require 'mechanize'
require 'fileutils'

#################################
# FUNCTIONS
#################################
def checkError(page)
  if page.search('.msg_error').length > 0
    puts "[ERROR]"
    page.search('.msg_error li').each do |li|
      puts li.text.strip
    end
    exit
  end
end

def checkResult(page)
  puts "[RESULT]"
  page.search('.msg_success li').each do |li|
    puts li.text.strip
  end
end

def checkNotice(page)
  puts "[NOTICE]"
  page.search('.msg-note').each do |note|
    puts note.text.strip
  end

  if page.search('.msg-note').length > 1
    puts "See log for more details"
  end
end

#################################
# SCRIPT CODE
#################################
abort "#{$0} Argument Missing" if (ARGV.size < 1)

dbRepairUrl     = ARGV[0]
mageVersion     = ARGV[1]
mageDir         = Dir.pwd
dbRepairLogDir  = "#{mageDir}/var/dp-repair-tool"

FileUtils.mkdir_p "#{dbRepairLogDir}"
fp = File.new("#{dbRepairLogDir}/mage-#{mageVersion}-result.html", "a+")

agent = Mechanize.new
page = agent.get(dbRepairUrl)
puts "Loading page: #{page.title}..."
form = page.forms.first

puts "Setting db repair form values..."

# Get DB value from app/etc/local.xml
xmlFile = File.open("#{mageDir}/app/etc/local.xml")
doc = Nokogiri::XML(xmlFile)
corruptedHostname = doc.xpath('/config/global/resources/default_setup/connection/host').text()
corruptedUsername = doc.xpath('/config/global/resources/default_setup/connection/username').text()
corruptedPassword = doc.xpath('/config/global/resources/default_setup/connection/password').text()
corruptedDatabase = doc.xpath('/config/global/resources/default_setup/connection/dbname').text()
xmlFile.close

form['post_form']           = 'true'
form['corrupted[hostname]'] = corruptedHostname
form['corrupted[database]'] = corruptedDatabase
form['corrupted[username]'] = corruptedUsername
form['corrupted[password]'] = corruptedPassword

# Edit reference database credentials
form['reference[hostname]'] = corruptedHostname
form['reference[database]'] = "magento#{mageVersion}_vanilla"
form['reference[username]'] = corruptedUsername
form['reference[password]'] = corruptedPassword

#p form; exit

puts "Submitting db repair form..."
result_page = form.submit(form.button_with(:id => "button-continue"))

# Check if there is an error & exit
checkError(result_page)

# Continue if there is not an error
# Check if it further requires submission
if result_page.search('button#button-continue').length > 0

    checkNotice(result_page)
    # Form found again?
    puts "Submitting again..."
    form = result_page.forms.first
    result_page = form.submit(form.button_with(:id => "button-continue"))
    checkResult(result_page)
    checkNotice(result_page)
else
    checkResult(result_page)
    checkNotice(result_page)
end

fp.write(result_page.body)

You can also download the script from: Magento DB Repair Tool Using CLI – Ruby + Mechanize

2. Run the Script
Give script the executable permission:

cd /path/to/mage-db-repair-tool.rb
chmod +x ./mage-db-repair-tool.rb

Now, You can run the script either by using command:

ruby mage-db-repair-tool.rb <magento-db-repair-url> <magento-version>

or simply using:

./mage-db-repair-tool.rb http://magento-test.dev/magento-db-repair-tool-1.2.php 1702

Example:

./mage-db-repair-tool.rb http://magento-test.dev/magento-db-repair-tool-1.2.php 1702
Ruby DB Repair Tool Console Output

Ruby DB Repair Tool Console Output

Apart from the results shown at the console output (refer to the above snapshot), You can also check the detailed output logged in your /path/to/magento/var/dp-repair-tool/mage-[magento-version]-result.html

Bonus Tips:

If you want to make system-wide command, then copy the file to the system-wide paths like /usr/local/bin, /usr/bin, /bin etc:

sudo cp /path/to/your/mage-db-repair-tool.rb /usr/local/bin/mageDbRepairToolRb

Now you can run the command from anywhere

mageDbRepairToolRb http://magento-test.dev/magento-db-repair-tool-1.2.php 1702

Shall you have any issues please post a comment below and I’ll try and help you out.

Send New Product Review Notification Email

January 13, 2015  |  1 Comments  |  by Raj (MagePsycho)  |  Latest, Magento

This is probably a feature which store owner might be looking for.
As of now store owner has to check for new product reviews from Magento backend in frequent manner.

Won’t it be better if admin gets a instant email whenever a new product review is posted? So that whatever action needs to be carried out with the pending review can be done quickly. You don’t need to buy expensive extension in order to get job done. Here I will be sharing a simple trick to achieve this.

1. Crate a skeleton module(for example: MagePsycho_Reviewnotifier) and register the event: review_save_after
File: app/code/local/MagePsycho/Reviewnotifier/etc/config.xml

<frontend>
    ...
    <events>
        <review_save_after>
            <observers>
                <magePsycho_reviewnotifier_review_save_after>
                    <type>singleton</type>
                    <class>magePsycho_reviewnotifier/observer</class>
                    <method>reviewSaveAfter</method>
                </magePsycho_reviewnotifier_review_save_after>
            </observers>
        </review_save_after>
    </events>
    ...
</frontend>

2. Implement the observer model
File: app/code/local/MagePsycho/Reviewnotifier/Model/Observer.php

<?php

/**
 * Observer Model for Review Notifier
 * 
 * @author MagePsycho<magepsycho@gmail.com>
 * @package MagePsycho_Reviewnotifier
 * Class MagePsycho_Reviewnotifier_Model_Observer
 */
class MagePsycho_Reviewnotifier_Model_Observer
{
    /**
     * Send notification email when product review is posted
     *
     * @param Varien_Event_Observer $observer
     */
    public function reviewSaveAfter(Varien_Event_Observer $observer)
    {
        $review = $observer->object;
        if ($review) {
            $emails     = array('info@yourstore.com', 'email1@yourstore.com'); #Edit admin emails
            try {
                //@todo - use configurable email templates
                $this->_sendNotificationEmail($emails, $review);
            } catch (Exception $e) {
                Mage::logException($e);
            }
        } else {
            Mage::log('ERROR::UNABLE TO LOAD REVIEW');
        }
    }

    protected function _sendNotificationEmail($emails, $review)
    {
        if (count($emails)) {
            $product        = Mage::getModel('catalog/product')->load($review->getEntityPkValue());
            $starRatings    = array_values($review->getRatings());

            $body = 'New product review has been posted!<br /><br />';
            $body .= 'Customer Name: ' . $review->getNickname() . '<br />';
            $body .= 'Product: ' . sprintf('<a href="%s" target="_blank">%s</a>', $product->getProductUrl(), $product->getName()) . '<br />';
            $body .= 'Star Rating: ' . (isset($starRatings[0]) ? $starRatings[0] : 'N/A') . '<br />';
            $body .= 'Review Title: ' . $review->getTitle() . '<br />';
            $body .= 'Review Description: ' . $review->getDetail() . '<br />';

            foreach ($emails as $toEmail) {
                $mail = Mage::getModel('core/email');
                $mail->setToName('YourStore Admin');
                $mail->setToEmail($toEmail);
                $mail->setBody($body);
                $mail->setSubject('YourStore: New Product Review');
                $mail->setFromEmail('donotreply@yourstore.com');
                $mail->setFromName("YourStore");
                $mail->setType('html');

                try {
                    $mail->send();
                }
                catch (Exception $e) {
                    Mage::logException($e);
                }
            }
        }
    }
}

Please leave your comments if it works for you or if you have any queries.

Debugging Tips: Filename cannot be empty

November 21, 2014  |  No Comments  |  by Raj (MagePsycho)  |  Latest, Magento

Problem

You may have seen this kind of error frequently logged in your Magento log file var/log/system.log:

Warning: include(): Filename cannot be empty in /app/code/core/Mage/Core/Block/Template.php on line xxx

This error means Magento is trying to include a template block with an empty string set as it’s template.

Solution

In order to find which template file is missing, simply edit the code in Mage_Core_Block_Template::fetchView() as:

public function fetchView($fileName)
{
    ...
    try {
        $includeFilePath = realpath($this->_viewDir . DS . $fileName);
        if (strpos($includeFilePath, realpath($this->_viewDir)) === 0 || $this->_getAllowSymlinks()) {
            /* edit - start */
            if (!is_file($includeFilePath)) {
                Mage::log($fileName, null, 'missing-phtml.log', true);
            }
            /* edit - end */
            include $includeFilePath;
        } else {
            Mage::log('Not valid template file:'.$fileName, Zend_Log::CRIT, null, null, true);
        }

    } catch (Exception $e) {
        ob_get_clean();
        throw $e;
    }
    ...
}

You will see the missing .phtml filenames being logged. And search for that name in your layout xmls which will show you where it went wrong.
Happy Debugging!

Welcome Mass Importer Pro: Price Importer Ver 1.1.0 with new features

August 8, 2014  |  1 Comments  |  by Raj (MagePsycho)  |  Latest, Magento, Updates

Mass Importer Pro: Price Importer is the fastest price import tool available for Magento CE/EE.
Recently MagePsycho team released the most awaited features in Mass Importer Pro: Price Importer Ver 1.1.0

ChangeLog (0.1.5 – 1.1.0)

  • Compatible with Magento CE 1.3.x – 1.9.x & EE 1.12.x – 1.14.x
  • Added cron job based price importing with logging feature
  • Added shell/command line based price importing with logging feature
  • Added price rounding feature (normal rounding, rounding to nearest)
  • Added relative(fixed & percentage) pricing for special, tier & group prices.
  • Changed store view based pricing import to website based(as prices have either global or website level scope)
  • Added store wise price export
  • Added option for deleting special_price, special_from_date, special_to_date by marking the value with ‘x’ or ‘X’
  • Added button for checking csv file format (beside of Price Import button)
  • Added Change log file: Changelog.txt
  • Added Uninstallation file: Uninstall.txt
  • Refactored the code

New Features

1. Cron Job based price importing

To configure this, go to Mass Importer Pro > Manage Settings > Price Settings
You will see the following setting:

Price Importer: Cron Settings

Price Importer: Cron Settings

FieldDescription
Enable CronYes
File Import DirectoryPath from where CSV files will be imported via cron.
Default: var/magepsycho/massimporterpro/price_importerr/cron
Process Imported FilesWhat to do after the files are imported?
Available options are:
No Action
Move To Archive Folder
Delete
Start TimeAt what time cron will run.
If ‘Start Time’ = 00 00 00 and ‘Frequency’ = Daily then equivalent cron expression will be 0 0 * * * (which means cron will be run once a day at midnight)
REF: http://en.wikipedia.org/wiki/Cron
FrequencyFrequency at which cron will run. Options:
Daily
Weekly
Monthly
Error Email RecipientError email will be sent to this address in case of failure
Error Email SenderError Email Sender
Error Email TemplateEmail Template

Notes:
Make sure that cron is enabled for your Magento store.
Refer to the following article for more info:
http://www.magentocommerce.com/wiki/1_-_installation_and_configuration/how_to_setup_a_cron_job

2. Shell/Command line based price importing

There is no any backend settings for Shell based price importing.
You just have to open the terminal and run the following commands.
1. Go to Magento root directory:

cd /path/to/your/magento/root

2. Price Importer help command:

php -f shell/priceimporter.php --help

3. Run price import (without reindexing)

php -f shell/priceimporter.php -- -import "path/to/csv/file.csv"

4. Run price import (with reindexing)

php -f shell/priceimporter.php -- -import "path/to/csv/file.csv" -reindex
Command line price import (with reindexing)

Command line price import (with reindexing)

Command line price import (without reindexing)

Command line price import (without reindexing)

So far in Mass Importer Pro, Prices can be imported via Web Forms, Cron Jobs & Command Line. From wherever you run import, You can view the import history from menu ‘Mass Importer Pro’ > ‘Price Importer’ > ‘Import History’ Tab > Filter the results by ‘Imported Via’.

Price Importer: Import History

Price Importer: Import History

3. Price rounding feature

There are three types of options for price rounding:

Price Importer: Price Rounding

Price Importer: Price Rounding

Rounding TypeDescription
No RoundingNo action with the price
Round NormallyUses php’s round() function. Examples:
9.43 -> 9.00
9.63 -> 10.00
Round to NearestYou have to set Rounding value for this. Examples:
9.43 -> 9.00 + (If Rounding Value = 0.99) = 9.99
9.43 -> 9.00 + (If Rounding Value = 0.50) = 9.50
This option is useful if you want to update your prices ending with .99, .50 etc.

4. Relative(fixed & percentage) based price importing

You can self increment or decrement the prices by fixed value or percentage. Also you can set ‘special_price’, ‘tier_price’ & ‘group_price’ to be certain percentage of ‘price’.
Valid Examples:

ValueDescription
10Fixed Value
+10Increment current value by 10
-10Decrement current value by 10
10%10% of base price (note that there is no any sign in front)
Note: This applies only for price types: special_price, tier_price & group_price
+10%Increment current price by 10%
 -10% Decrement current price by 10%

5. Option to validate your CSV data

You can validate the CSV data before importing any prices.
Go to menu ‘Mass Importer Pro’ > ‘Price Importer’ > Select the import file > Click on the ‘Check Import’ button which will give you the check results.

Price Importer: Check Import Data

Price Importer: Check Import Data

6. Website wise price Import

Before Ver 1.1.0 You had to use combination of sku + store_id as unique identifier to update prices. Since Magento has either global or website level scope for prices, we have replaced ‘store_id’ by ‘website_id’.

In order to find the website_id, go to System > Manage Stores > Click on the related Website > Note the website_id from URL.
By default website_id = 0 (global)

7. Website wise price Export

You can export prices in format compatible with Mass Importer Pro from menu ‘Catalog’ > ‘Manage Products’.
How to export prices is clear from the below screenshot:

Price Importer: Export Prices

Price Importer: Export Prices

Also see: Importing regular, special, tier & group prices using ‘Mass Importer Pro: Price Importer’ Extension

Those were the major updates on newer version. Besides we have fixed some bugs & improved the functionality.
If you have any queries/issues regarding the extension, please do not hesitate to Contact Us.