I have been looking at how to stop checkout of maximum quantities in Magento for one product category e.g. one per transaction or one per customer only.
The short answer: it’s not possible (unless you want to pay for a commercial extension).
The workaround
Scenario: I am starting a penny sale – simply pay for postage and packaging and buy something for 1p, or add a penny item to your order.
Orders from the penny sale category will be restricted to one per transaction. As this is not possible I have decided on a surcharge instead.
This has the benefit of allowing customers to decide if they wish to purchase more than one and allows the retailer to use existing Magento tools.
We are going to use Shopping Cart Price Rules (under the promotions tab from the dashboard). If the customer purchases more than one item from the category then £1 will be added for the second and each additional item in the shoppping cart.
By default a surcharge (or negative discount) is not allowed in Magento so we need to change a few lines of code to enable it:
1. Go to app/code/core/Mage/Rule/Model/Rule.php
Find the line:
//check if discount amount > 0
if ((int)$this->getDiscountAmount() < 0) {
Mage::throwException(Mage::helper(‘rule’)->__(‘Invalid discount amount.’));
}
Just add some bars // to comment the code:
//check if discount amount > 0
//if ((int)$this->getDiscountAmount() < 0) {
//Mage::throwException(Mage::helper(‘rule’)->__(‘Invalid discount amount.’));
//}
2. Now go to: app/code/core/Mage/Adminhtml/Block/Promo/Quote/Edit/Tab/Actions.php
Look for:
$fieldset->addField(‘discount_amount’, ‘text’,
array(
‘name’ => ‘discount_amount’,
‘required’ => true,
‘class’ => ‘validate-not-negative-number’,
‘label’ => Mage::helper(’salesrule’)->__(‘Discount amount’),
And again comment like this:
$fieldset->addField(‘discount_amount’, ‘text’,
array(
‘name’ => ‘discount_amount’,
‘required’ => true,
// ‘class’ => ‘validate-not-negative-number’,
‘label’ => Mage::helper(’salesrule’)->__(‘Discount amount’),
3. Translate the discount word.
Go to app/locale/en_US
Find a file called Mage_Sales.csv. Inside look for the word discount you will find something like:
“Discount (%s)”,”Discount (%s)”
Change the text to what ever you like, e.g.:
“Discount (%s)”,”Extra Fee (%s)”
The above section comes from: http://php.quicoto.com/extra-fee-shopping-cart-price-rules-magento/
Next we need to create the shopping cart rule by setting the conditions:
My sale catgegory is 55 so I want to activate the rule when more than one item from category 55 is added to the cart.

Then we need to create the actions rule to tell Magento what to do when the above conditions are met:

You will notice I have added an extra condition in the actions section. This is to ensure only items in category 55 attract the £1 surcharge – without it £1 would be added for every item in the shopping cart.
It is not quite how I wanted it to work as it add £1 for every item in the basket if rather than only for the second one, third one etc, but it will have to do.
Whilest it is unfortunate that the rules in Magento are not more flexible or easier to customise, after hours or research this is the simplest and quickest workaround I have found.